Integrating with GitLab and Bitbucket

When developing an application to improve your DevOps experience you will probably want to have a way of accessing the code you have stored on your version control provider. This can be done with an OAuth application, which will allow users to link their version control account with your application, then you can access their version control data via the version control provider’s API. In my previous article, I talked about some of the things you should know when integrating with GitHub. Today, I’ll be talking about two more version control providers, GitLab and Bitbucket, including how to create an application as well as some of the ways those providers are different to each other and GitHub.

GitLab

To integrate your application with GitLab, you must first create an application in the GitLab profile settings. This will allow your users to link their GitLab accounts through OAuth. On this page, you can set a name for your application and provide one or more redirect URLs for the linking process. The confidential box determines what kind of OAuth linking flow you will need to use, and is based on whether you can keep the OAuth client secret hidden from the users of your application. If you are creating a web application that will store the user’s data on a backend server, you will likely want to tick the Confidential box as you can keep the client secret on the backend server and users won’t be able to access it. On the other hand, if you are creating something like a native application where there is no backend server, you will need to store the client secret on the user’s device, meaning you should not tick the Confidential option.

GitLab has a number of options for scopes though they are lacking somewhat in granularity. If you are only using GitLab as a single sign-on provider you can get away with just the email scope, however if you want to integrate with GitLab’s features you will probably want to choose the api scope which is a very broad scope giving you access to nearly all of GitLab’s features. If you are only working with a small amount of the platform you may be able to get away with the registry permissions or the repository permissions, but it’s important to note that the write_repository permission only provides write access to repos via HTTP, and does not grant you access to GitLab’s file API endpoints. Applications that don’t intend to modify the repository in any way can use read_api, with the caveat that if you need repository webhooks you will need full api access so that you can create a webhook on a repository.

How is GitLab structured?

GitLab is structured a little bit differently to its peers GitHub and Bitbucket. Whilst there shouldn’t be anything too surprising, there are a few little quirks that it’s useful to know about.

GitLab’s building blocks are ‘projects’. Each project contains a repository and repository related features such as merge requests (pull requests) and issues. A project also contains other features such as CI/CD handling, registry package management and a wiki.

This distinction between repositories and projects is sometimes important - when accessing the API many of your calls will be to the Projects API. You won’t find a way to list all of a user’s repositories - you’ll need to list all of a user’s projects.

Additionally, it’s a useful distinction when it comes to project permissions. As a repository is different from a project, it’s possible for a user to have access to a project without having any access to that project’s repository. By default, a user who has the ‘guest’ role on a private project is able to see some of the project’s details but not the repo. Therefore, if you are trying to list all the repositories a user can access, you may need to filter the project list to only show projects that the user has repository access on.

Groups and subgroups

GitLab’s main collaboration tool is groups - much like GitHub’s organizations, GitLab’s groups allow you to invite other GitLab users and create projects that can be accessed by everyone in the group. GitLab’s group permissions are very granular - a number of permissions can be customised such as which roles can create projects. This means, when using the API to work out which actions a user can perform, you cannot always assume the permissions a user has from their role. For example, you might have a feature where you create a new repository for a user and want to list all the groups where the user can create a project. By default the Developer and Maintainer role can both create projects, but if you just search for all groups where the user has the Developer role or higher, you may get groups where the user isn’t able to create a new project because the group has modified the default permissions.

Additionally, each project also has its own permission roles. This means a user outside of a group may be invited to work on a specific project, or a member of the group might have elevated permissions on an individual project. For example, if you are checking whether a user has administrator access to a project, you need to make sure you check both the group access level and the project access level (generally these are returned together by the API).

GitLab’s groups also have the unique feature of subgroups. A subgroup is a group belonging to a group. Subgroups inherit the permissions of the groups they belong to, but additionally can have their own permissions overriding that. Users may be members of a subgroup without being a member of the group it belongs to. Subgroups generally have the same features as groups, including the ability to have their own subgroups. In this way, subgroups are kind of like folders.

One pitfall to look out for if you are parsing the URL of a repository is that GitLab’s subgroups mean that there can be multiple forward slashes in the URL as each subgroup name is preceded by a slash.

Webhooks

Webhooks allow you to receive notifications when specific events happen to a project such as commits being pushed to a repo, merge requests being opened and issues being created. Unlike GitHub Apps, GitLab’s webhooks are not account wide but are instead set up on an individual repository.

You can manually create a webhook in a project’s settings. Here you can add a url where you want GitLab to send the webhook events to. You can also select which events you want to receive. You can also set two security features here. If you set a secret token, GitLab will send that token with all its webhooks in the X-Gitlab-Token header. When receiving webhooks, you can verify that the token is correct to make sure that webhook requests you receive are legitimate requests from GitLab. The Enable SSL verification option makes it so that GitLab will verify the webhook endpoint has a valid SSL certificate before sending any webhooks.

You might use this manual webhook creation if you are making an application for personal use. However, if you are making an application designed to integrate with other users’ GitLab accounts, you will likely create webhooks using the GitLab webhooks API. You will have to call the Add project hook endpoint for each project you want to receive webhooks for. It is important to know that webhooks can be edited by any member of that project with administration access, so it is possible for users to delete or edit your webhooks. You can use the List project hooks and Get project hook endpoints to check whether your webhook still exists and has the necessary events enabled, however there is no way to validate that the webhook secret has not been modified so you may find you do not receive valid webhooks if a user has modified the secret.

When creating secret tokens for webhooks, you should generate a different token for each repository and ensure that it is random and cannot be guessed. This ensures that users cannot spoof webhooks for repositories that they do not own.

import crypto from 'crypto';

// Function for verifying GitLab webhook tokens in a koa server
export const verifyGitLabToken = async ({ receivedToken, repoUrl, db }, ctx) => {
  if (!receivedToken) {
    ctx.throw(401, 'Webhook token not provided.');
  }

  const tokenObject = await db.getCollection('webhook-tokens').find({ repoUrl });

  const ourToken = tokenObject?.token;

  if (!ourToken) {
    ctx.throw(401, 'Failed to verify webhook token.');
  }

  // crypto.timingSafeEqual requires the two inputs to be equal length.
  if (receivedToken.length !== ourToken.length) {
    ctx.throw(401, 'Failed to verify webhook token.')
  }

  // Using timingSafeEqual prevents malicious actors from guessing your webhook tokens using a timing attack.
  const validToken = crypto.timingSafeEqual(Buffer.from(receivedToken), Buffer.from(ourToken));
  if (!validToken) {
    ctx.throw(401, 'Failed to verify webhook token.')
  }
}

Bitbucket

Integrating with Bitbucket is very similar to integrating with GitLab. You can create a new Bitbucket application when in a workspace under Settings > Apps and Features > OAuth Consumers. BitBucket requires a callback url for your OAuth handling but also optionally takes a URL for users to learn more about your application, a privacy policy URL and a end user license agreement URL which are all shown to the user when linking your application. Similar to GitLab’s confidential setting, you should tick the ‘This is a private consumer’ checkbox if you are able to keep the OAuth client secret secure on a backend server.

The permissions here are more granular than GitLab and you should be able to pick whatever scopes your application requires without giving it more permissions than necessary.

How is Bitbucket structured?

Bitbucket’s structure is fairly simple to understand, though like the other version control providers it also has its own quirks that it is important to be aware of.

Workspaces

All Bitbucket repositories belong to a workspace. Unlike GitHub and GitLab, Bitbucket does not have a hard distinction between users and teams. When you create an account, Bitbucket will create a workspace for you, however other Bitbucket users can be invited to your workspace. From a developer standpoint, there is no canonical workspace for a given user. You cannot list repos belonging to a user, or create a repo in the authenticated user’s namespace, because there is no such thing - every workspace a user is a part of is as valid as any other. Therefore, most API requests you make will be on a specific workspace - you can list repos belonging to a workspace, or create a repo in that workspace.

Similar to GitLab, repositories can have separate permissions to the workspaces they belong to.

Projects

Bitbucket workspaces can have projects. Similar to GitLab’s subgroups, these act similar to folders, though unlike GitLab, Bitbucket’s projects exist mostly for organizational use rather than for access control. Projects have a name and a key, and can have a separate visibility level to the workspace as a whole, but otherwise offer no additional features. All repositories belong to a project - Bitbucket creates a default project if none exists and will put new repos in that project if a project isn’t specified. Some endpoints such as repository creation take a project key to allow you to decide which project the repo should be a part of but otherwise projects are unintrusive. They are not part of the repo url and generally shouldn’t cause you much issue.

Webhooks

Webhooks in Bitbucket function mostly the same as the webhooks in GitLab. You set a webhook url, determine which events you wish to receive, and whether to enable SSL verification. Unlike GitLab however, Bitbucket is missing any kind of verification token. This makes it easy for attackers to potentially spoof webhook events. Atlassian suggests that you solve this by setting up a whitelist to only accept webhooks from their IP addresses (Bitbucket server supports webhook tokens so this is not an issue there). However, there is one workaround you can use if you wish to implement token verification in a similar way to GitLab - you can put the webhook token in the webhook URL. This could be done either as a path parameter or a query parameter. Then, on your webhook server, you can take the token from the path and verify it as normal. Obviously, this isn’t perfect and I hope Bitbucket implements webhook tokens in the headers like GitLab but it is certainly better than having no verification whatsoever!

Supporting multiple version control providers allows you to reach more users and give them more flexibility over how to use your application. Whilst each provider has their own quirks and differences, hopefully you now have a good understanding of some of things you need to look out for when implementing support for GitLab and Bitbucket. If you missed it, you can check out my previous post on GitHub.

Thank you for reading! I hope you found this brief guide to GitLab and Bitbucket useful when developing your own applications. If you have any questions you can contact me at [first name] @northflank.com.