CI/CD - Using GitHub Actions Matrix Strategy

Overview#

One of the first projects I had, when we moved from BitBucket to GitHub, was figuring out our CI/CD processes, especially since deployments were taking nearly two hours due to a large amount of different app components. And the fact that the original monolithic repository was not so intuitive when trying to test the latest changes. Because of a matrix strategy you can run multiple builds at the same time (only limited if you were strictly using self-hosted runners). This enabled us to cut our build time from two hours to 32 minutes when building the fullstack application, (and if just testing a new component then it’s as little as 4 minutes). Faster iteration allows for faster development!

Example Documentation#

When looking for inspiration I scoured the GitHub Actions documentation and found a very simple guide to run matrix strategies: example.1

1
jobs:
2
  example_matrix:
3
    strategy:
4
      matrix:
5
        version: [10, 12, 14]
6
        os: [ubuntu-latest, windows-latest]

From this very simple JSON format, you could run this workflow 3*2 tiimes with each combination. You can also see that you can output a JSON formatted list and ingest in a different job using the fromJSON function: example.2

1
jobs:
2
  define-matrix:
3
    runs-on: ubuntu-latest
4

5
    outputs:
6
      colors: ${{ steps.colors.outputs.colors }}
7

8
    steps:
9
      - name: Define Colors
10
        id: colors
11
        run: |
12
          echo 'colors=["red", "green", "blue"]' >> "$GITHUB_OUTPUT"
13

14
  produce-artifacts:
15
    runs-on: ubuntu-latest
16
    needs: define-matrix
17
    strategy:
18
      matrix:
19
        color: ${{ fromJSON(needs.define-matrix.outputs.colors) }}
20

21
    steps:
22
      - name: Define Color
23
        env:
24
          color: ${{ matrix.color }}
25
        run: |
26
          echo "$color" > color
27
      - name: Produce Artifact
28
        uses: actions/upload-artifact@latest
29
        with:
30
          name: ${{ matrix.color }}
31
          path: color

Walkthrough#

From those examples, I quickly saw how we might be able to consolidate our workflow to be more readable and to make primary place for editing all of our component artifact information. For example, in the below file, you could name each of the different components, and the information required when building it, such as where to find the build file, and what type of package to build. components.json

1
[
2
    {
3
        "name": "ComponentA",
4
        "sourcePath": "Components/ComponentA/ComponentA.csproj",
5
        "outputPath": "Components/Type",
6
        "zipName": "ComponentA.zip",
7
        "buildType": "dotnet"
8
    },
9
    {
10
        "name": "ComponentB",
11
        "sourcePath": "Components/ComponentB/ComponentB.csproj",
12
        "outputPath": "Components/Type",
13
        "zipName": "ComponentB.zip",
14
        "buildType": "python"
15
    },
16
    {
17
        "name": "ComponentC",
18
        "sourcePath": "Components/ComponentC/ComponentC.csproj",
19
        "outputPath": "Components/Type",
20
        "zipName": "ComponentC.zip",
21
        "buildType": "dotnet"
22
    }
23
]

Incorporating this into a CI/CD pipeline was a feat in and of itself, but looks quite simple looking back on it. I’ll walk through each of the steps as part of the set up.

Workflow dispatches (manually kicking off the workflow) and changes inside component source code would kick off the build process, for this example I listed three components, but there’s a lot more than that in reality! ci.yml

1
name: CI Workflow
2
run-name: Building ${{ github.ref_name }} ${{ github.run_number }}
3
on:
4
    workflow_dispatch:
5
    push:
6
        paths:
7
            - "Components/ComponentA/**"
8
            - "Components/ComponentB/**"
9
            - "Components/ComponentC/**"

This step gets the list of changed files depending on the event that triggered the workflow. The overall events that trigger this step is either a PR or a push. If the event was caused by a pull request, only the files that are being requested for review would be included in the list. If the event was triggered by a push, then it would get the git diff between the last commit and the current commit. Both comparisons would output the same list as a GITHUB_OUTPUT to be used in a future step.

1
jobs:
2
    get-changed-files:
3
        runs-on: ubuntu-latest
4
        outputs:
5
            file-list: ${{ steps.changed-files.outputs.list }}
6
            components: ${{ steps.json-format.outputs.components }}
7
        steps:
8
            - uses: actions/checkout@latest
9
              with:
10
                fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 0 }}
11
            - name: Get Raw Changed File List
12
              id: changed-files
13
              if: ${{ (github.event_name == 'pull_request') || (github.event_name == 'push') }}
14
              runs: |
15
                # If it's a pull request, the whole stack should be built, if it was anything else, then just the components that had changed files
16
                if ${{ github.event_name == 'pull_request' }}; then
17
                    echo "list=$(git diff --name-only -r HEAD^1 HEAD | xargs)" >> $GITHUB_OUTPUT
18
                else
19
                    echo "list=$(git diff --name-only ${{ github.event.before }} ${{ github.event.after }} | xargs)" >> $GITHUB_OUTPUT
20
                fi

Here, we would either ingest the changed file list from the previous step OR if it was a workflow dispatch or a newly created branch, then it would ingest the components.json file as the fullstack list to be built. If it were any other trigger, then we would create an empty list and cycle through the changed list from the previous step. From here, I used some bash string augmentation to get the file name ComponentA isolated (since some of the paths were inconsistent with naming conventions) and then compared it with the components.json to make sure it was indeed one of the components on the list. We then appended the dictionary list of the component’s information onto the new empty list, and we repeat that for each of the changed files. At the end we remove extra characters and make sure it’s a complete json file to be saved as a GITHUB_OUTPUT.

1
            - name: Formatting Changed File List
2
              id: json-format
3
              shell: bash
4
              run: |
5
                # Ingest whole components.json if deployed with a workflow dispatch or when a new branch is created
6
                if [ "${{ github.event_name }}" == "workflow_dispatch" ] || [ "${{ github.event_name }}" == "create" ]; then
7
                    echo "components=`jq '.' -c components.json`" >> $GITHUB_OUTPUT
8
                else
9
                    new-list=""
10
                    # Editing the path of the changed file to get the component name
11
                    for file in ${{ steps.changed-files.outputs.list }}; do
12
                        file=${path#*/}
13
                        path=${path#*/}
14
                        path=${path%%/*}
15
                        # Compare the component name to the component.json file to make sure it's one of the app components
16
                        jq-outputs=`jq -c "map(select(.name==\"${path}\")) | first" components.json`
17
                        if [["${jq-outputs}" != "null" ]]; then
18
                            final-list="${new-list}${jq-outputs},"
19
                        else
20
                            echo "${path} is not a component."
21
                            continue
22
                        fi
23
                        continue
24
                    done
25
                changed-list="${final-list:1}"
26
                changed-list="[${changed-list::-1}]"
27
                changed-list=`echo ${changed-list} | jq -c 'unique_by(.name)'`
28
                echo "components=${changed-list}" >> $GITHUB_OUTPUT
29
                fi

Using the previous example, if we only changed ComponentA and ComponentC then the outputs.components would be: components.json

1
[
2
    {
3
        "name": "ComponentA",
4
        "sourcePath": "Components/ComponentA/ComponentA.csproj",
5
        "outputPath": "Components/Type",
6
        "zipName": "ComponentA.zip",
7
        "buildType": "dotnet"
8
    },
9
    {
10
        "name": "ComponentC",
11
        "sourcePath": "Components/ComponentC/ComponentC.csproj",
12
        "outputPath": "Components/Type",
13
        "zipName": "ComponentC.zip",
14
        "buildType": "dotnet"
15
    }
16
]

In a new job, we can use our job output as the matrix strategy to build. We would also be able to save each of the component’s dictionary lists as it’s own environment variable. This job would run per component, so if you were to change files in 10 component paths then this build job would run 10 times!

1
    build:
2
        runs-on: ubuntu-latest
3
        needs: get-changed-files
4
        continue-on-error: true
5
        strategy:
6
            matrix:
7
                components: ${{ fromJSON(needs.get-changed-files.outputs.components ) }}
8
                fail-fast: false
9
        env:
10
            name: ${{ matrix.components.name }}
11
            sourcePath: ${{ matrix.components.sourcePath }}
12
            outputPath: ${{ matrix.components.outputPath }}
13
            zipName: ${{ matrix.components.zipName }}
14
            buildType: ${{ matrix.components.buildType }}
15
        steps:
16
            - uses: actions/checkout@latest

At the end of my example, you could have conditionals for specific package build steps (i.e. step.python-build would only run if env.buildType == python) and other necessary steps to ensure that each component is built properly. In this case, we also uploaded each component to Artifactory, where our QA team or other devs can share and access these artifacts for testing or promotion.

Reflection
There are a ton of things you can do with any specific tool, but sometimes that means going through the documentation and sitting on it to come up with some ideas. We also used this process in our SaaS product to implement phased deployments for components that depend on eachother. I also worked on this project with the OG Bernie Crandall (who worked on the phased deployment logic)!