Merging Coverage from Jest and Cypress

6 min read#dev#github

Introduction

At our workplace, one of the business requirements is to provide a unified coverage report that merges results from all our test sources, in our case, unit/integration (using Jest) and end-to-end (with Cypress). We even have a KPI: our total code coverage needs to be more than 85%1. While combining these metrics in a single view can be handy, it’s important to recognise that unit and E2E tests serve different purposes.

Why merge coverage?

Pros:

Why it might not be the best idea:

Despite these concerns, and given the business requirements, we needed a way to find a combined coverage metric and engineer our pipeline to meet that need.

Current State of the System

Before diving into our solution, it’s important to understand the current landscape:

Challenges

We need to find a way to collect the coverage generated by the parallel runs of Cypress and merge them along with the coverage generated by Jest.

Another challenge arises from the tooling: Jest is using SWC, but Cypress requires Babel. This creates inconsistencies in the coverage data between unit and E2E tests.

After a few attempts, we realised that the numbers didn’t add up. We tried running our unit tests both with SWC and Babel as transpilers, obtaining the following results.

Coverage with SWC:

=============================== Coverage summary ===============================
Statements   : 85.66% ( 7812/9119 )
Branches     : 65.4% ( 2091/3197 )
Functions    : 79.63% ( 1756/2205 )
Lines        : 85.54% ( 7077/8273 )
================================================================================

Coverage with Babel:

=============================== Coverage summary ===============================
Statements   : 79.57% ( 5030/6321 )
Branches     : 63.48% ( 2191/3451 )
Functions    : 79.01% ( 1747/2211 )
Lines        : 79.64% ( 4874/6120 )
================================================================================

As seen in the numbers above, Babel tends to report lower overall coverage than SWC, particularly in statements and lines. One key issue is that SWC considers imports as covered statements, inflating the overall metrics.

Moreover, while there is an open issue in the Cypress coverage repository regarding SWC support (issue #583), it has been stale for a long time. There’s also an effort to support Istanbul for SWC (swc-plugin-coverage-instrument), but it is currently not working as expected.

This divergence means we must carefully manage configurations to use SWC for performance in Next.js while switching to Babel when generating coverage reports for Cypress.

Overview of the workflow

To ensure accurate test coverage reports across both Jest and Cypress, we decided to stick to Babel for the code coverage, but keep SWC for everything else. We use a temporary Babel configuration along with a set of npm scripts to apply and clean up the necessary changes.

Key Components

Separate Babel Config for Coverage

We use coverage.babel.config.js specifically for test coverage. This prevents unwanted instrumentation from affecting the main development and production build. This config extends next/babel and enables the istanbul plugin only when BABEL_ENV=component is set.

The file looks like this:

module.exports = {
  presets: ["next/babel"],
  env: {
    component: {
      plugins: ["istanbul"],
    },
  },
};

Updated jest.config.js

Here’s how you can modify your jest.config.js to dynamically switch between babel-jest (when coverage is enabled) and @swc/jest (for faster test execution when coverage is not needed):

const useBabel = process.env.BABEL_ENV === "true";

module.exports = {
  transform: {
    "\\.[j|t]s[x]?$": useBabel
      ? "babel-jest"
      : [
          "@swc/jest",
          {
            jsc: {
              transform: {
                react: {
                  runtime: "automatic",
                },
              },
            },
          },
        ],
  },
  // ... rest of the configuration
};

How It Works

This ensures that the coverage reports are accurate when needed and the tests run much faster when coverage is not required.

"test:coverage": "cross-env BABEL_ENV=true jest --coverage"

When running normal tests (jest without BABEL_ENV=true), it will use @swc/jest for improved performance.

NPM Scripts to Automate the Process

Before running tests or building for coverage, we swap the Babel config using a temporary setup:

The following scripts ensure proper execution:

"pretest:coverage": "npm run coverage:setup",
"test:coverage": "cross-env BABEL_ENV=true jest --coverage",
"posttest:coverage": "npm run coverage:teardown",
"prebuild:coverage": "npm run coverage:setup",
"build:coverage": "cross-env BABEL_ENV=component next build",
"postbuild:coverage": "npm run coverage:teardown",
"coverage:setup": "cp coverage.babel.config.js babel.config.js",
"coverage:teardown": "rm babel.config.js"

The pre/post hooks guarantee that no babel.config.js is present outside of coverage testing.

Merging coverage from Jest and Cypress

To get a complete test coverage report, we need to combine results from Jest and Cypress.

How it works

  1. Jest Coverage

    • Running npm run test:coverage executes Jest with BABEL_ENV=true, ensuring that Istanbul (enabled via Babel) collects coverage.
    • Jest generates a coverage report in jest-coverage/.

  2. Cypress Coverage

    • Cypress requires the istanbul plugin in Babel to instrument the code during execution.
    • Since coverage.babel.config.js includes istanbul under the component environment, running npm run build:coverage ensures that the built application includes coverage tracking.
    • Cypress tests then generate coverage reports in coverage/.

  3. Consolidating Cypress coverage:

    Each parallel Cypress job generates a coverage/coverage-final.json file, which is renamed to coverage-<job_number>.json and uploaded as an artifact.

  4. Merging Reports

    • We download the coverage artifacts and merge them using nyc.
    • We generate reports based on the merged coverages.
    • We sanitise the file paths contained in the coverage using sed (otherwise they are absolute paths on the runner, like /home/runner/actions-runner/_work/...).

With this approach, unit and end-to-end coverage are measured with the same tool, so the merged numbers are consistent.

Jest Testing Workflow

The sanity-jest job is responsible for running unit tests using Jest:

sanity-jest:
  name: Jest
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - uses: actions/setup-node@v4
      with:
        node-version-file: ".nvmrc"
        cache: "npm"

    - name: Install dependencies
      run: npm install
      env:
        CI: true

    - run: npm run test:coverage -- --silent --colors

    - uses: actions/upload-artifact@v4
      with:
        name: coverage-jest
        path: "jest-coverage/coverage-final.json"

Key points

Parallelised Cypress E2E Testing

The sanity-e2e job is designed to distribute Cypress E2E tests across multiple containers:

sanity-e2e:
  name: End to End Tests
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  env:
    BASE_PATH: ""
  steps:
    - uses: actions/checkout@v4

    - uses: actions/setup-node@v4
      with:
        node-version-file: ".nvmrc"
        cache: "npm"

    - name: Cypress run
      uses: cypress-io/github-action@v6
      with:
        build: npm run build:coverage
        start: npm run start:ci
        wait-on: "http://localhost:3000/"
        parallel: true
        record: true
      env:
        CYPRESS_PROJECT_ID: ${{ secrets.CYPRESS_PROJECT_ID }}
        CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

    - name: Check file existence
      id: check_files
      uses: andstor/file-existence-action@v3
      with:
        files: "coverage/coverage-final.json"

    - name: Rename coverage
      if: steps.check_files.outputs.files_exists == 'true'
      run: mv coverage/coverage-final.json coverage/coverage-${{ matrix.containers }}.json

    - name: Upload coverage
      uses: actions/upload-artifact@v4
      if: steps.check_files.outputs.files_exists == 'true'
      with:
        name: coverage-${{ matrix.containers }}
        path: "coverage/coverage-${{ matrix.containers }}.json"

Key points

Merging and Validating Coverage Reports

Finally, the coverage job consolidates all coverage reports:

coverage:
  name: Coverage
  runs-on: ubuntu-latest
  needs: [sanity-e2e, sanity-jest]
  steps:
    - uses: actions/checkout@v4

    - name: Download and move artifacts
      uses: actions/download-artifact@v4
      with:
        merge-multiple: true
        path: coverage
        pattern: coverage-*

    - name: Merge coverage
      run: |
        mkdir .nyc_output
        npx nyc merge coverage
        mv coverage.json .nyc_output/out.json
        npx nyc report --reporter text --reporter json-summary --report-dir coverage --exclude-after-remap false
        sed -i -e 's/\/home\/runner\/actions-runner\/_work\/my-repo\/my-repo\///g' coverage/coverage-summary.json

    - name: Coverage Diff
      uses: greatwizard/coverage-diff-action@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        allowed-to-fail: "true"

Key points

Gotchas

Conclusion

Using SWC in Next.js and Babel in Cypress adds complexity. Our solution keeps SWC for speed in day-to-day builds and tests, and switches to Babel only when we need coverage.

It is more setup than I would like, but it gives us one coverage number that matches the business requirement, without slowing down the normal test runs.

Footnotes

  1. I will never get tired of saying that high code coverage does not necessarily mean good code or a good test strategy, but that topic deserves its own post. ↩

Thoughts? Reply via emailSuggest an edit

NewerConditionally Run GitHub Actions Based on PR LabelsOlderDependencies vs devDependencies