Context

Most of the time we get lured with new offerings for various reasons (fandom, fomo, simplicity and ease of use are major reasons), once we’re lured we start with PoC, on successful PoC, we start migrating to new offerings after other due diligence and business needs. Halfway through the migration number of items unravel and while handling a few we may stumble, as the implementation requires us to take a circuitous route.

Services/offerings from GCP are no exception!

Cloud Build is one of the out of the box solutions or services offered by GCP to set CI/CD pipelines in a cost effective manner. Setting up cloud builds is simple and straightforward and easy most of the time with publicly available documentations. But, we had one such scenario where we had to take a circuitous route to setup CI/CD pipeline. That scenario was to build and push a docker image of a UI application but before we push the image to the repository, sanity and dictum of the image has to be ensured by running integration tests in our CI/CD pipeline. For a number of reasons we had to maintain application and test code in two different repositories. In the legacy CI/CD the test repository is added as a submodule and in the pipeline tests have been initiated.

Problem

When we use Cloudbuild for CI/CD on a project with git submodules, cloudbuild is not pulling the submodule code before starting the CI/CD and officially there is an issue logged which is open since January 2019! Also, there are no official Google Docs or any other Blogs to solve this issue with submodules in source repositories.

Solution

We solved this issue, by importing our submodule ui-tests code repository to Cloud Source Repositories and with below changes/additions to files - cloudbuild.yaml, .gitmodules

• First, we need to import test code git submodule repository to Cloud Source Repository,

  • That gives us repository url - https://source.cloud.google.com/p/[my-gcp-project]/r/[my-submodule-repo-name]

• To skip access related issues, replace cloud to developers in above url https://source.developers.google.com/p/[my-gcp-project]/r/[my-submodule-repo-name]

• Then, we need to replace the git url value present in .gitmodules file with the above modified cloud repo url

.gitmodules file

  [submodule "ui-tests"]
  path = ui-tests
  url = https://source.developers.google.com/p/[my-gcp-project]/r/[my-submodule-repo-name]

.gitmodules file is generated by git in the host/application repo directory after adding submodules, for more information check here.

• Finally, we need to add a separate step in cloudbuild.yaml file as below with gcr.io/cloud-builders/git to pull the test code submodule along with the application code.

cloudbuild.yaml file

# Getting the sub-module ui-tests
- id: fetch-submodule
  name: "gcr.io/cloud-builders/git"
  entrypoint: "bash"
  args:
    - "-c"
    - |
      git init
      git clean  -d  -f . #clean current working directory
      git remote add origin https://source.developers.google.com/p/[my-gcp-project]/r/[my-host-repo-name]
      git fetch origin $BRANCH_NAME
      git checkout $COMMIT_SHA #checkout at current commit
      git config -f .gitmodules submodule.[my-submodule-repo-name].url https://source.developers.google.com/p/[my-gcp-project]/r/[my-submodule-repo-name]
      git submodule update --init

This step has to be placed before your build starts executing the application code. In our case, we had this step at first, before building the application / running tests.

After making the above changes, cloud build pulled our entire code with submodules!