Mirror Repositories

You can mirror repositories of a particular user or organization/group in GitHub, GitLab, and/or Bitbucket.

You can mirror repositories to ensure that you are not dependent on a remote version control service being available at all times and to continue your build and release process even in the event of a denial of service attack (or similar). You can read more about this here.

Each service has a unique way of naming a group of repositories:

  • Bitbucket Cloud — "team"

  • Bitbucket Server — "project"

  • GitLab — "project"

  • GitHub — "organization"

When a remote repository is mirrored, the local repository mirror is kept up-to-date when changes are made to the remote repository. This happens in one of the following ways:

  • Installing a web hook into each remote repository. Then, the repository is updated when changes are pushed to the local Bitbucket repository.

  • Polling the remote repositories for changes every five or thirty minutes.

See Step 4 of the Mirroring Repositories task below for more information.

Once the mirroring process has begun, you cannot push to the local Bitbucket repository. The remote repository is assumed to be authoritative and removes any branches in the Bitbucket repository that do not exist in the remote repository.

If the default branch is modified on both sides, synchronisation is broken. Remove the Bitbucket repository, and then rerun the setup.

A pre-receive hook is automatically setup for the mirrored repository to block all changes to the local Bitbucket repository.

If you want to push changes to the local Bitbucket repository, select the Remove option from View and Configure Mirrored Repositories.

Mirroring Repositories

  1. Enter the Organization/Team/Group/User/Project Key.

  2. Enter a Target Project.

    Typically this is an empty project, but it doesn’t have to be. For this example, it is named "Mirror."

  3. To fill out API Key, follow authentication steps outlined below:

    Authentication is not necessary but is recommended. Public repositories can be mirrored without the need for any authentication details.

    • GitHub Authentication

      You need a GitHub API key so the plugin can list your repositories.

      If you do not already have one with at least the permissions shown in the image, you need to create one.

      Github's documentation on creating personal access tokens describes the process in detail. While we endeavour to keep this document up to date, Github's process may have changed.

      1. Navigate to https://GitHub.com.

      2. Select Settings > Developer Settings.

      3. Select Personal Access Tokens from the navigation, or click here.

      4. Click Generate New Token.

      5. Select an appropriate expiration period. Keep in mind that when the token expires, you will need to reconfigure synchronization for this user or organization.

      6. Check that it has the correct permissions. The key permissions that you will need are public_repo  and repo:status under the repo root permission.



        GitHub's Rate Limit
        You can also mirror public repositories without defining an access token. Please note that GitHub has a much smaller number of allowed anonymous API requests per hour compared to authorised ones, and you can easily exceed that limit, especially when mirroring organizations with large numbers of repositories. You can find more info in https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting.

    • GitLab Authentication

      You need to generate a GitLab personal access token so that ScriptRunner can authenticate with GitLab as your user.

      1. Navigate to https://GitLab.com.

      2. Select User Settings > Access Tokens.

      3. Configure the access token with a name and the scopes shown in the screenshot below:

    • Bitbucket Cloud/Server Authentication

      1. Enter the username and password of a user who is permissioned on the remote repositories.

  5. Check the Add private repositories from authenticated user checkbox to fetch all repositories, including private repositories, belonging to the authenticated user of the GitHub API key.

    This field is only available for the GitHub mirroring service and when a user is entered in the Organization field and is authenticated.

    If the user entered in the Organization/User field is different from the authenticated user belonging to the API key, a warning message displays informing that only public repositories from the user will be returned.

  6. Select an option for How to Synchronize.

    • None

      If you select None, no synchronization is done.

    • Install Hook

      If you install a web hook, the remote service calls your Bitbucket instance whenever there are changes to the repositories, which prompts the plugin to sync the repositories. This is the preferred approach because changes are reflected almost immediately, and there is no unnecessary polling.

      However, you can only use web hooks if your Bitbucket Server instance is accessible to the remote service over the internet.

    • Poll

      If you choose to poll, each remote repository is polled every 5 minutes or 30 based on your choice. This is a lightweight process and is perfectly acceptable if you are unable to use web hooks.

  7. Enter a Regular Expression to check against the repository name (not the part that appears in the URL), and can be used to filter the set of repositories you wish to mirror.

    To choose all, leave the default regular expression as is.

  8. Check the Sync New checkbox to sync new repositories that are created or forked into your team/organization/user automatically.

    If you have set a Regular Expression, it is respected when creating new mirrored repositories.

    New repositories are created in Bitbucket up to ten minutes after they appear in the remote service.

  9. Enter an Invalid characters replacement to replace invalid characters contained within repository names.

    Using the following format, repository names are truncated to 128 characters: 'invalidPrefixReplacement:otherInvalidOccurrencesReplacement'.i.e using 'mod_:_' sanitises '.invalid^&reponame to mod_invalid_reponame.

  10. Click Preview.

    If your token or credentials are correct, you should see a listing of the organization/user repositories and if they are created or updated in the Bitbucket project.

  11. Click Run.

    This can take a while, depending on the number and size of the repositories you are mirroring. You can tail the application log file to see progress, and use the View and Configure Mirrored Repositories function to see what has been created.

    You see the same screen when it finishes, but it now displays Exists for the local repositories.

Security

If you are using web hooks for synchronization, GitHub calls a REST endpoint in your Bitbucket instance when it’s pushed to. To know that it’s GitHub that’s calling, the payload of the call is hashed with a secret token that you supply.

Follow these steps to generate a random token:

  1. Run this code in Script Console.

    import java.security.SecureRandom
SecureRandom random = new SecureRandom();
String str = new BigInteger(130, random).toString(32);

  2. Restart Bitbucket with the property -Dgithub.secret.token=<from above>.from above is where you add the generated code.

If you already have remote repositories that are synced by web hooks, you need to update these hooks, by rerunning this script.

If you do not have direct access to the internet, the web services calls respect your values of http.proxyHost and http.proxyPort. However, you must change your .gitconfig on the Bitbucket server to specify the proxy.

There is useful information here on how to set a proxy only for certain sites: http://stackoverflow.com/questions/16067534/only-use-a-proxy-for-certain-git-urls-domains. Currently, authenticated proxy access is not supported.

View and Configure Mirrored Repositories

You can view the status of your mirrored repositories and projects, and you can modify the remote sync options.

You can check whether your Bitbucket repository is up to date with the remote repository by clicking the Check or Check All links.

A warning icon means repositories are not in sync. They do not mean that there is necessarily a problem. Warning icons should not appear if you are using a web hook trigger. If you are using polling, warnings catch up on the next scheduled task (every 5 minutes). If a repository is persistently out of sync, check the application log file.

A warning icon is also displayed if the repository is empty or unavailable. Bitbucket and GitHub provide us with no option to distinguish between those two cases.

If you previously set that repositories are automatically created, the projects set to do that are listed below the repositories. If you wish to stop this, check the checkbox under Stop Synchronization, and then click Run.

To change the regular expression, delete the configuration as shown below and then recreate the configuration. It’s fine to mirror into existing repositories, and this should not take very long.

 


On this page