CI/CD - QA Automated Unit Tests

Overview#

Our QA team has unit tests that they wanted to run everyday to verify usability and consistency on new deployments. There were several unit tests, so these were taking several hours to run, we reduced it to a couple of hours through the use of a matrix strategy. However, these tests were expected to take a very long time to run due to the depth of the testing.

Walkthrough#

To kickoff the workflow, it can be manually triggered via a workflow dispatch AND it is triggered daily on a specific testing environment to test the latest changes that were automatically deployed. The schedule also helps to address things that could change overnight either because of a VM issue or perhaps a deprecated action as part of the build process.

1
name: Unit Tests
2

3
on:
4
    workflow_dispatch:
5
    push:
6
        schedule:
7
            - cron: "0 1 * * *"

This first job ingests a json file with the testing information needed for the rest of the workflow. Saving it as a GitHub output allows it to be used across jobs.

1
jobs:
2
    get-tests:
3
        runs-on: ubuntu-latest
4
        outputs:
5
            test: ${{ steps.json.outputs.tests }}
6
        steps:
7
            - uses: actions/actions@latest
8
            - name: Get Tests & HTML Title
9
              id: json
10
              shell: bash
11
              run: echo "test=`jq '.' -c test-manifest.json`" >> $GITHUB_OUTPUT

As I demonstrated in a previous post, you can create a json file with number variables to create a matrix strategy in a later job. In this instance, we are using it to specify a specific method of unit testing along with the desired output name for the HTML report. test-manifest.json

1
[
2
    {
3
        "testName": "A",
4
        "htmlTitle": "TitleA.html"
5
    },
6
    {
7
        "testName": "B",
8
        "htmlTitle": "TitleB.html"
9
    },
10
    {
11
        "testName": "C",
12
        "htmlTitle": "TitleC.html"
13
    }
14
]

At this point, you can configure the testing job with a matrix strategy and write the tests or reference the test’s testing source code if necessary. For privacy’s sake, I did not include the testing configuration other than mvn test. Afterwards, you can use the standard actions/upload-artifact action to upload the html report to github.

1
    tests:
2
        runs-on: ubuntu-latest
3
        needs: get-tests
4
        continue-on-error: true
5
        strategy:
6
            matrix:
7
                tests: ${{ fromJSON(needs.get-tests.outputs.test ) }}
8
        steps:
9
            - name: ${{ matrix.test.testName }}
10
              run: mvn test # at this point I won't explicitly write the testing process
11
            - name: Upload ${{ matrix.test.htmlTitle }} Artifact
12
              id: artifact-upload
13
              if: always()
14
              uses: actions/upload-artifact@latest
15
              with:
16
                name: ${{ matrix.test.htmlTitle }}
17
                path: ./Report.html
18
                if-no-files-found: error
19
                retention-days: 3
20
                overwrite: true

We also wanted to send automated email notifications to the QA team. Since the workflow has a daily schedule it wouldn’t send notifications to the QA automatically on it’s own so this is the basic foundation of a job that we used to set it up. We used sendgrid which requires a specific SMTP key and sent it with a curl command.

1
    email:
2
        runs-on: ubuntu-latest
3
        needs: [get-tests, tests]
4
        continue-on-error: true
5
        outputs:
6
            date: ${{ steps.set-date.outputs.date }}
7
        steps:
8
            - name: Get Date
9
              id: set-date
10
              if: always()
11
              shell: pwsh
12
              run: |
13
                $date=get-date -f dd-MM-yyyy
14
                echo "date=$date" >> $env:GITHUB_OUTPUT
15
            - name: Send Email
16
              if: always()
17
              env: ${{ steps.set-date-outputs.date }}
18
              shell: pwsh
19
              run: |
20
                # Here you could use something like sendgrid to send emails to the correct team members with information on the testing that occurred with direct links to the html artifact.

Reflection
Collaborating across teams to create needed automation can greatly impact work streams and enable good feedback loops especially for those in QA. Another thing is