⚡ Distributed e2e Task Execution with Nx for Playwright and Cypress

Accelerate Test Execution: Harnessing Fine-Grained Parallelization with Nx

That feature is now integrated in Nx. Please check my article 💎 Discovering Nx Project Crystal’s Magic

In this article, I’d like to introduce an innovative method for shard testing using Nx, enabling the distribution of Playwright or Cypress tests across multiple processes and CI agents orchestrated by Nx cloud.

The inspiration for this approach comes from a YouTube video by Zack DeRose, where he showcased the Nx Project Inference API v2 in Nx v17. The code example from the video can be accessed on the Nrwl GitHub repository tic-tac-toe-playwright under the branch named “e2e-sharding”:

GitHub - nrwl/tic-tac-toe-playwright at e2e-sharding

Principle

This approach leverages two key Nx concepts: The Project Inference and the Distribution Task Execution.

The Project Inference

Generate a dynamic list of targets for each test without manual maintenance.

Distribution Task Execution

Run all generated targets on multiple CI agents using Nx cloud.

Implementation For Playwright

1. Create an Nx Plugin (/tools/plugins/playwright-sharding.ts):

• This plugin is invoked by Nx to facilitate target addition.
• Utilizes the Playwright command to dynamically obtain the list of tests.
• Generates a shared target (e2e-sharded) with dependencies on individual test targets.
• Creates a separate target for each e2e test, allowing for independent execution.

import { CreateNodes, readJsonFile, ProjectConfiguration } from '@nx/devkit';
import { execSync } from 'child_process';
import { dirname } from 'path';

// Plugin called by Nx to allow adding
export const createNodes: CreateNodes = [
  '**/project.json',
  (filePath, context) => {
    const projectConfig = readJsonFile(filePath) as ProjectConfiguration;
    const cwd = filePath.replace('/project.json', '');
    const name = projectConfig.name;
    if (!name) {
      return {};
    }
    try {

// Use Playwright command to get the list of tests
      const testList = execSync(`npx playwright test --list`, {
        cwd,
      }).toString();
      if (!testList.includes(`Listing tests:`)) {
        return {};
      }
      const tests = testList
        .split(`\n`)
        .filter((_, i) => i > 0 && i < testList.split(`\n`).length - 2)
        .map((line) => line.split(` › `)[2]);
      
// Generate the e2e-shared target with all dependsOn targets
      const targets = {
        [`e2e-sharded`]: {
          command: `echo \"Running tests using sharding...\"`,
          dependsOn: tests.map(testNameToTargetName),
        },
      };

// Generate one target for each e2e test that will execute it separately
      for (const test of tests) {
        const targetName = testNameToTargetName(test);
        targets[targetName] = {
          command: `CI=true npx playwright test -g "${test}"`,
          options: { cwd },
        };
      }

// Return all new configs that will be merge with based one
      return { projects: { [name]: { targets, root: dirname(filePath) } } };
    } catch (e) {
      return {};
    }
  },
];

function testNameToTargetName(testName: string): string {
  return `e2e ${testName}`.split(' ').join('-');
}

2. Register the plugin in nx.json:

{
  ...
  "plugins": ["./tools/plugins/playwright-sharding.ts"]
}

3. Create a script (add-cacheable-operations.js) to make generated targets cacheable:

• Fetches all target names that start with ‘e2e-’ and adds them to cacheable operations.
• Updates nx.json with the new cacheable operations.

const { readNxJson, createProjectGraphAsync } = require('@nx/devkit');
const { writeFileSync } = require('fs');

const nxJson = readNxJson();
console.log(nxJson.tasksRunnerOptions.default.options.cacheableOperations);

async function getAllTargetNames() {
  const foo = await createProjectGraphAsync();
  const targetsToMakeCacheable = new Set();
  for (const project of Object.values(foo.nodes)) {
    for (const targetName of Object.keys(project.data.targets)) {
      if (targetName.startsWith('e2e-')) {
        targetsToMakeCacheable.add(targetName);
      }
    }
  }
  return targetsToMakeCacheable;
}

getAllTargetNames().then((targets) => {
  const newCacheableOperations = new Set();
  for (const target of nxJson.tasksRunnerOptions.default.options
    .cacheableOperations) {
    newCacheableOperations.add(target);
  }
  for (const target of targets) {
    newCacheableOperations.add(target);
  }
  nxJson.tasksRunnerOptions.default.options.cacheableOperations = Array.from(
    newCacheableOperations
  );
  writeFileSync('./nx.json', JSON.stringify(nxJson, null, 2));
  process.exit(0);
});

4. Configure CI to run the shared e2e target:

• Utilize the Nx Cloud setup with a specific configuration for CI.
• Specifies the parallel execution of linting, testing, and building tasks.
• Executes the e2e-sharded target with parallelization.

name: CI

on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

jobs:
  main:
    name: Nx Cloud - Main Job
    uses: nrwl/ci/.github/workflows/nx-cloud-main.yml@v0.13.0
    with:
      main-branch-name: main
      number-of-agents: 10
      install-commands: |
        pnpm install
        node add-cacheable-operations.js
      init-commands: |
        npx nx-cloud start-ci-run --agent-count=10
      parallel-commands-on-agents: |
        npx nx affected --target=lint --parallel=3
        npx nx affected --target=test --parallel=3 --ci --code-coverage
        npx nx affected --target=build --parallel=3
        npx nx affected --target=e2e-sharded --parallel=1

  agents:
    name: Nx Cloud - Agents
    uses: nrwl/ci/.github/workflows/nx-cloud-agents.yml@v0.13.0
    with:
      number-of-agents: 10
      install-commands: |
        pnpm install
        node add-cacheable-operations.js
        npx playwright install --with-deps

Implementation For Cypress

They just merged exactly the same approach for Cypress here https://github.com/nrwl/nx/pull/20188/files

Awareness!!!

It’s essential to note that this approach is experimental.

• Only use Distribution Task Execution if it aligns with your project requirements.
• Customize the test split judiciously to avoid unnecessary overhead.
• Consider the overhead due to a new process for each test.
• Challenges may arise in generating a comprehensive test report.
• Sharing fixtures between tests might not be feasible.

Conclusion

This article highlights the powerful capabilities of Nx in optimizing test execution by introducing a novel approach to shard testing.

By leveraging Nx Project Inference API v2 and Distribution Task Execution, developers can efficiently distribute Playwright or Cypress tests across multiple processes and CI agents through Nx Cloud.

Importantly, this approach could also be used with any type of target that needs distribution.

🚀 Stay Tuned!

Related

🌊 The Nx Series