Feat bitbucket server support by ViktorGrigorov7 · Pull Request #6689 · apache/devlake

ViktorGrigorov7 · 2023-12-22T12:38:08Z

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

I have read through the Contributing Documentation.
I have added relevant tests.
I have added relevant documentation.
I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary

What does this PR do?
This PR makes possible integration with the bitbucket server. It adds it to both the backend and the config ui.

Does this close any open issues?

Closes #4568

Screenshots

Other Information

d4x1 · 2023-12-25T09:35:10Z

+	"github.com/go-playground/validator/v10"
+)
+
+var vld *validator.Validate


I think you'd better to use the latest helper. The code you are using can work, but the latest helper will be more clear and simple.
Examples: https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/api/init.go#L31

For more help about new helper, @klesh will help you.

Hey, @klesh. I reviewed the api.DsHelper. From what I saw it makes the code cleaner in init.go but it complicates the code in other parts. For example, to retrieve a repo, even in the GitHub plugin, we have to make the database query (https://github.com/apache/incubator-devlake/blob/b6c9e6cf6aa4a0e07443c6f98eea2e7adf528e41/backend/plugins/github/api/blueprint_v200.go#L86). At least that's from my first look. Perhaps I don't understand it well enough. I'll look at it in a bit more depth. Help in any way would be appreciated :)

I made a basic integration, similar to the one in BitBucket plugin.

Nice work, I took a glance and it seems very good to me.
However, we are testing v0.21 and it will be releasing it soon, so we are going to merge the PR as soon as the release branch is checked out.
I will take another closer look but I think the overall work is very impressive, thanks for your contribution.

d4x1 · 2023-12-25T09:37:05Z

+	if err != nil {
+		return nil, err
+	}
+	return &plugin.ApiResourceOutput{Body: connections, Status: http.StatusOK}, nil


To avoid sensitive tokens leakage, a Sanitize function is expected.
Example: https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/models/connection.go#L253
And this function should be inited to dsHelper(Example: https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/api/init.go#L47).

d4x1 · 2023-12-25T09:38:23Z

+// @Failure 400  {string} errcode.Error "Bad Request"
+// @Failure 500  {string} errcode.Error "Internal Error"
+// @Router /plugins/bitbucket_server/test [POST]
+func TestConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {


A new handle for testing connection is required.
Example: https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/impl/impl.go#L190

d4x1 · 2023-12-25T09:40:48Z

+			s := queryData.Search[0]
+			gid, searchName := getSearch(s)
+			// query.Set("sort", "name")
+			// query.Set("fields", "values.name,values.full_name,values.language,values.description,values.owner.display_name,values.created_on,values.updated_on,values.links.clone,values.links.html,pagelen,page,size")


These commented codes can be deleted directly.

d4x1 · 2023-12-25T09:51:25Z

@ViktorGrigorov7
Thank you for your contribution! You have done a great job!
I think the PR is excellenct expect for some small issues. As to the code details, some latest update(such as new helpers and new APIs for a plugin) should be applied.

d4x1 · 2023-12-25T10:03:35Z

IMO, there is one question I think we should disscuss:
BitBucket Cloud and BitBucket Server are regarded as one some type connection in the ConfigUI(you can see the config form for BitBucket), so I don't think it should be a sperated plugin with sperated go package(in a seperate directory).
GitHub Connection supports cloud and server version at the same time.

BitBucket is designed like GitHub.

And in this way, we can reduce lots of duplicated codes.

ViktorGrigorov7 · 2024-01-02T07:26:51Z

@d4x1 I do think it will be easier for the users to have BitBucket Cloud and BitBucket Server buttons. However the differences between BitBucket Cloud and BitBucket Server are significant. For example, the cloud version supports issues, deployments, etc. and these are not supported in the Server version. Also I don't think there's a lot of replicated code since the endpoints of the Server are very different from the Cloud.

Sync

d4x1 · 2024-01-02T07:40:10Z

@d4x1 I do think it will be easier for the users to have BitBucket Cloud and BitBucket Server buttons. However the differences between BitBucket Cloud and BitBucket Server are significant. For example, the cloud version supports issues, deployments, etc. and these are not supported in the Server version. Also I don't think there's a lot of replicated code since the endpoints of the Server are very different from the Cloud.

Agreed!

Signed-off-by: Marais Van Zyl <marais@projexis.co.za>

…etter suits bb terminology)

d4x1

LGTM

d4x1 · 2023-12-25T09:57:32Z

@@ -1,4 +1,4 @@
-#!/bin/sh


Just keep sh.

…bitbucket and bitbucket_server

mintsweet · 2024-02-04T10:06:29Z

 export const BitBucketConfig: IPluginConfig = {
  plugin: 'bitbucket',
-  name: 'BitBucket',
-  icon: ({ color }) => <Icon fill={color} />,


Do not modify this, otherwise the theme color will not be applied.

mintsweet · 2024-02-04T10:07:36Z

      TRANSFORMATION:
        'https://devlake.apache.org/docs/v0.20/Configuration/BitBucket#step-3---adding-transformation-rules-optional',
    },
+    BITBUCKET_SERVER: {


It’s not needed here either. There is no such plugin in 0.20.

martynvdijke · 2024-03-08T11:51:58Z

Hi all,

Just wondering when this change will land in a new beta release?
I just upgrade to beta7 but can't see the change.
I would love to play wih this.

klesh · 2024-03-12T02:29:30Z

bitbucket server will be available on v1.0.0, maybe in 2 weeks.

* feat: initial commit for bitbucket-server * chore: remove bb-cloud code from server Signed-off-by: Marais Van Zyl <marais@projexis.co.za> * chore: remove more bb cloud references Signed-off-by: Marais Van Zyl <marais@projexis.co.za> * chore: set line ending to lf Signed-off-by: Marais Van Zyl <marais@projexis.co.za> * chore(bitbucket-server): spelling mistake * chore(bitbucket-server): update swagger paths * chore(bitbucket-server): fix migration name * chore(bitbucket-server): renames from bitbucket * chore: change test endpoint * refactor: getting scopes to function * chore: update repo/project remote search Signed-off-by: Marais Van Zyl <marais@projexis.co.za> * chore: update paging * fix(bitbucket-server): remote scopes paging * fix(bitbucket-server): remote scopes * refactor: small changes to account for server apis * test: update raw repository data * test: repo test * chore: refactor fields * several fixes * fix: made prs and pr comments extractable, renamed account to user (better suits bb terminology) * fix: user convert * fix: pr comment convertor * fix: pr_commit_collector and pr_commit_extractor * fix: commit collector/extractor * feat: branch collector, fixes * fix: made collection of commits happen across all branches * fix: deleted build_collector - no use * feat: bitbucket-server in ui + several fixes * fix: renamed bitbucket-server to bitbucket_server * fix: pr base_repo_id issue * feat: grafana dashboard * fix: pr collection issue * fix: pr_collector end * test: e2e for prs * test: e2e for pr_commits * fix: extraction commit author + e2e tests for commits * fix: renamed account_convertor to user_convertor * chore: removed dead comments * fix: rebase issues * fix: test connection * refactor: using dsHelper * fix: scope config crud * feat: implemented transformations, removed previous transformations * refactor: renamed migration * chore: removed dead code * fix: domain types, enabling tasks by default * fix: merge commit sha for pull requests * fix: pr commits pagination issue, remote scopes pagination * fix: pagination fix for all collections in bitbucket_server * fix: recollect issues * fix: merge commit sha through activities api * refactor: removed useless code and bumped up concurrency * refactor: reduced repeating code * fix: renamed bitbucket_server structs and fixed RootPkgPath for both bitbucket and bitbucket_server * fix: remove commit sha parents * fix: pagination issue, removed not used concurrency * fix: conflict BitBucket * fix: cicd detected issues * fix: models * fix: ui issues, linting issues, removed dead code * fix: test issues * fix: branches are not incremental * fix: branches are not incremental * fix: all collectors are now stateless * fix: removed useless tasks * fix: stateful collectors --------- Signed-off-by: Marais Van Zyl <marais@projexis.co.za> Co-authored-by: Marais Van Zyl <marais@projexis.co.za> Co-authored-by: Lynwee <linwei.hou@merico.dev>

ViktorGrigorov7 mentioned this pull request Dec 22, 2023

docs: bitbucket server/data-center apache/devlake-website#691

Merged

2 tasks

d4x1 reviewed Dec 25, 2023

View reviewed changes

Merge pull request #1 from apache/main

a8ae2a4

Sync

mieliespoor and others added 23 commits January 2, 2024 10:15

feat: initial commit for bitbucket-server

8d9de92

chore: remove bb-cloud code from server

544c0a2

Signed-off-by: Marais Van Zyl <marais@projexis.co.za>

chore: remove more bb cloud references

bdbe71c

Signed-off-by: Marais Van Zyl <marais@projexis.co.za>

chore: set line ending to lf

f4ec48e

Signed-off-by: Marais Van Zyl <marais@projexis.co.za>

chore(bitbucket-server): spelling mistake

31eb8bb

chore(bitbucket-server): update swagger paths

9138766

chore(bitbucket-server): fix migration name

d961136

chore(bitbucket-server): renames from bitbucket

ca8ee99

chore: change test endpoint

cd933a6

refactor: getting scopes to function

feace0b

chore: update repo/project remote search

2649620

Signed-off-by: Marais Van Zyl <marais@projexis.co.za>

chore: update paging

f17f23b

fix(bitbucket-server): remote scopes paging

13333f7

fix(bitbucket-server): remote scopes

defd3db

refactor: small changes to account for server apis

d542ae6

test: update raw repository data

36b2657

test: repo test

74a2e3a

chore: refactor fields

71ae93d

several fixes

89c0aa3

fix: made prs and pr comments extractable, renamed account to user (b…

c440662

…etter suits bb terminology)

fix: user convert

42795b7

fix: pr comment convertor

7ed4a25

fix: pr_commit_collector and pr_commit_extractor

8218149

d4x1 reviewed Jan 30, 2024

View reviewed changes

ViktorGrigorov7 and others added 7 commits January 30, 2024 15:21

fix: renamed bitbucket_server structs and fixed RootPkgPath for both …

4df342d

…bitbucket and bitbucket_server

fix: remove commit sha parents

074b291

fix: pagination issue, removed not used concurrency

00c338f

fix: conflict BitBucket

e44bd2a

Merge branch 'main' into feat-bitbucket-server-support

dff5bd8

fix: cicd detected issues

5f90ab9

fix: models

21af03f

mintsweet reviewed Feb 4, 2024

View reviewed changes

fix: ui issues, linting issues, removed dead code

258d0ae

ViktorGrigorov7 force-pushed the feat-bitbucket-server-support branch 3 times, most recently from 29e3855 to 258d0ae Compare February 13, 2024 14:49

ViktorGrigorov7 and others added 7 commits February 13, 2024 16:51

fix: test issues

1dd19c5

Merge branch 'main' into feat-bitbucket-server-support

c3af0c6

fix: branches are not incremental

c4758ab

fix: branches are not incremental

8a6a509

fix: all collectors are now stateless

c3c9ca2

fix: removed useless tasks

d8fe8e3

fix: stateful collectors

f4f61bb

d4x1 approved these changes Feb 15, 2024

View reviewed changes

d4x1 merged commit ea0daf4 into apache:main Feb 16, 2024

martynvdijke mentioned this pull request Mar 9, 2024

When will bitbucket server land in bèta ? #7155

Closed

Conversation

ViktorGrigorov7 commented Dec 22, 2023

⚠️ Pre Checklist

Summary

Does this close any open issues?

Screenshots

Other Information

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d4x1 commented Dec 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

d4x1 commented Dec 25, 2023

Uh oh!

ViktorGrigorov7 commented Jan 2, 2024

Uh oh!

d4x1 commented Jan 2, 2024

Uh oh!

d4x1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martynvdijke commented Mar 8, 2024

Uh oh!

klesh commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

d4x1 commented Dec 25, 2023 •

edited

Loading