diff --git a/detect/synthetic-monitoring/playwright-checks/add-to-group.mdx b/detect/synthetic-monitoring/playwright-checks/add-to-group.mdx index d53a748..161a896 100644 --- a/detect/synthetic-monitoring/playwright-checks/add-to-group.mdx +++ b/detect/synthetic-monitoring/playwright-checks/add-to-group.mdx @@ -4,45 +4,45 @@ description: 'Organize your Playwright Check Suites with groups for better manag sidebarTitle: 'Add to Group' --- -You can define a new group, or associate a Playwright Check Suite to an existing group. + +You need: -## Steps +- A repository using the Checkly CLI +- At least one Playwright Check Suite defined in your repository. -### 1. Create a group for your checks + -Create a new file to create and configure your new group. +Groups organize checks and share configuration across them. See the [Groups overview](/platform/groups) for details on benefits and how groups work. + +## 1. Create a group + +Create a new file to define your group. ```typescript website-group.check.ts -import { CheckGroup } from 'checkly/constructs' +import { CheckGroupV2 } from 'checkly/constructs' -export const myGroup = new CheckGroup('production-group', { +export const myGroup = new CheckGroupV2('production-group', { name: 'Production group', - logicalID: 'production-group', activated: true, muted: false, locations: ['us-east-1', 'eu-west-1'], - tags: ['mac', 'group'], + tags: ['production', 'playwright'], environmentVariables: [], - apiCheckDefaults: {}, - concurrency: 100, - runParallel: true, - retryStrategy: RetryStrategyBuilder.linearStrategy({ - baseBackoffSeconds: 30, - maxRetries: 2, - sameRegion: false - }), }) ``` -Learn more about [using groups](/constructs/check-group-v2) to unify checks in Checkly. +Learn more about [using groups](/constructs/check-group-v2). -### 2. Associate the group to the check +## 2. Add a Playwright Check Suite to the group -When specifying your Playwright Check Suite, you can reference the new or existing group, using its name: +Import the group and reference it in your Playwright Check Suite configuration: + +```typescript checkly.config.ts highlight={2,14} +import { defineConfig } from 'checkly' +import { myGroup } from './website-group.check' -```typescript checkly.config.ts highlight={13} export default defineConfig({ logicalId: 'checkly-website', projectName: 'checkly-website', @@ -55,20 +55,58 @@ export default defineConfig({ pwTags:'critical', frequency: 10, locations: ['us-east-1',], - groupName: 'Production group', // use the name of the group you created + group: myGroup, }, ], }, }) - -export default config ``` -### 3. Deploy to apply the changes +## 3. Deploy your changes ```bash -npx checkly deploy --preview # confirm what will be deployed -npx checkly deploy # deploy +npx checkly deploy --preview # preview what will be deployed +npx checkly deploy # deploy the new scheduled Playwright Check Suites into the group ``` -You can now see your Playwright Check Suite assigned to your group. +Your Playwright Check Suite now appears in the group in your [Checkly Home Dashboard](https://app.checklyhq.com/). + +## Add multiple check suites to one group + +Add multiple Playwright Check Suites to the same group while keeping individual test selection and frequency settings. Reference the group in each suite: + +```typescript checkly.config.ts highlight={2,13,20,27} +import { defineConfig } from 'checkly' +import { myGroup } from './website-group.check' + +export default defineConfig({ + logicalId: 'checkly-website', + projectName: 'checkly-website', + checks: { + playwrightConfigPath: './playwright.config.ts', + playwrightChecks: [ + { + name: 'Critical User Flows', + logicalId: 'critical-flows', + pwTags: 'critical', + frequency: 5, + group: myGroup, + }, + { + name: 'Authentication Tests', + logicalId: 'auth-tests', + pwTags: 'auth', + frequency: 10, + group: myGroup, + }, + { + name: 'Checkout Flow', + logicalId: 'checkout-flow', + pwTags: 'checkout', + frequency: 5, + group: myGroup, + }, + ], + }, +}) +``` diff --git a/detect/synthetic-monitoring/playwright-checks/configuration.mdx b/detect/synthetic-monitoring/playwright-checks/configuration.mdx index 71f0ca0..ba61703 100644 --- a/detect/synthetic-monitoring/playwright-checks/configuration.mdx +++ b/detect/synthetic-monitoring/playwright-checks/configuration.mdx @@ -99,7 +99,7 @@ A Playwright Check Suite inherits multiple properties from [the abstract `Check` - `privateLocations` - `alertEscalationPolicy` -Check [the reference documentation](/constructs/project#param-checks-playwright-checks) for more information on these settings. +Check out [the reference documentation](/constructs/project#param-checks-playwright-checks) for more information on these settings. **Checks' Retry strategy is not applicable for Playwright checks.** Playwright includes its own retry features that can be set up directly in your `playwright.config.ts/js` file with the [`retries`](https://playwright.dev/docs/test-retries) option. This allows for more detailed management of test retries within Playwright, when your check runs. @@ -107,49 +107,135 @@ Check [the reference documentation](/constructs/project#param-checks-playwright- Additionally, Playwright Check Suites provide specific configuration for your Playwright monitoring. -* `installCommand:` Override the command to install dependencies, by default it'll use `npm install --dev`. +* `installCommand:` Override the command to install dependencies. `npm install --dev` is used by default. -* `testCommand:` Override the command to test, by default it uses npx playwright test with the tags, projects, and config file options your check specifies. +* `testCommand:` Override the command to test. `npx playwright test` is used by default with the tags, projects, and config file options your check specifies. -* `groupName:` The group this check belongs to. +* `group:` The group construct this check belongs to. + +### Customize install and test commands + +By default, Checkly runs `npm install --dev` to install dependencies and `npx playwright test` to run tests. The `testCommand` automatically includes your `pwProjects`, `pwTags`, and Playwright config file path. + +You can override these commands to: + +**Use a different package manager:** + +```typescript +installCommand: 'yarn install --frozen-lockfile' +``` + +**Install dependencies from a specific directory:** + +```typescript +installCommand: 'npm install --prefix ./e2e' +``` + +**Enable Playwright tracing to debug failures** + +```typescript +testCommand: 'npx playwright test --trace=on' +``` + +**Set a specific test timeout:** + +```typescript +testCommand: 'npx playwright test --timeout=60000' +``` + +**Combine multiple test flags:** + +```typescript +testCommand: 'npx playwright test --trace=retain-on-failure --max-failures=5' +``` + +### Example Checkly config ```typescript checkly.config.ts +import { defineConfig } from 'checkly' +import { Frequency } from 'checkly/constructs' +import { productionGroup } from './production-group.check' + export default defineConfig({ // Shared Checkly configuration (scheduling, locations, etc.) // ... checks: { - // Specify the relative path to your config file playwrightConfigPath: './playwright.config.ts', playwrightChecks: [ { - /** - * Run `@e2e` tagged tests across browsers - * every 5 minute from 4 locations. - */ - // Human readable name name: 'E2E test suite', - // Reference id logicalId: 'e2e-test-suite', - // Playwright project references defined in `playwright.config` pwProjects: ['chromium', 'firefox', 'webkit'], - // Playwright tags defined in `spec` files pwTags: '@e2e', - // Override default dependencies install command installCommand: 'npm install --dev', - // Append to the resulting test command, respects your pwTags and pwProjects filters. testCommand: 'npx playwright test --trace=on', - // Activate the check so that it runs on a schedule, true by default activated: true, - // Mute the check so that it doesn't send alerts muted: false, - // Add a check to a group - groupName: 'production-group', - // Specify a monitoring frequency + group: productionGroup, frequency: Frequency.EVERY_5M, - // Define locations to monitor from locations: ['us-east-1', 'eu-west-1','eu-central-1', 'ap-south-1'], } ] }, -} +}) ``` + +## Testing configuration locally + +Before deploying your Playwright Check Suites, validate your configuration locally: + +```bash +# Test your configuration without deploying +npx checkly test + +# Test a specific check by logical ID +npx checkly test --grep=e2e-test-suite +``` + +The `npx checkly test` command runs your Playwright tests locally using the same configuration that Checkly will use in production. This helps catch configuration errors before deployment. + +## Common issues & troubleshooting + +### Configuration errors + +**Invalid `logicalId`** + +``` +Error: logicalId must be unique across all checks +``` + +Ensure each Playwright Check Suite has a unique `logicalId`. The `logicalId` is used as a stable reference across deployments. + +**Missing Playwright configuration** + +``` +Error: Cannot find playwright.config.ts at path: ./playwright.config.ts +``` + +Verify that `playwrightConfigPath` points to the correct relative path from your `checkly.config.ts` file. + +### Test execution issues + +**Tests exceed 20-minute limit** + +If your Playwright Check Suite times out after 20 minutes: + +- Split large test suites into multiple Playwright Check Suites using `pwTags` or `pwProjects` +- Use `.only` or `.skip` in your tests during development, then create separate check suites for different test priorities +- Optimize slow tests by reducing wait times and parallelizing where possible +- [Contact Checkly support](https://app.checklyhq.com/?support=true) if you need further help + +**Dependency installation failures** + +If dependencies fail to install: + +- Check that your `package.json` and lock file (e.g., `package-lock.json`) are both included in your deployment. +- Verify that the private registry authentication is configured correctly. See the [custom dependencies guide](/detect/synthetic-monitoring/playwright-checks/custom-dependencies/) for more details. +- Adjust the `installCommand` to match your project's expectation. + +**Authentication failures in tests** + +If tests fail due to authentication issues: + +- Verify environment variables are set correctly in Checkly. You can verify your environment variables using `npx checkly env ls`, or looking at your Global or Check's environment variables in the Checkly Webapp to ensure any `process.env.VARIABLE_NAME` call from your test is defined. +- For persistent authentication, use Playwright's [`storageState`](https://playwright.dev/docs/auth#reuse-authentication-state) feature. \ No newline at end of file diff --git a/detect/synthetic-monitoring/playwright-checks/custom-dependencies.mdx b/detect/synthetic-monitoring/playwright-checks/custom-dependencies.mdx index 61b0435..ce5dd77 100644 --- a/detect/synthetic-monitoring/playwright-checks/custom-dependencies.mdx +++ b/detect/synthetic-monitoring/playwright-checks/custom-dependencies.mdx @@ -4,11 +4,11 @@ description: 'Use private packages, custom registries, and additional dependenci sidebarTitle: 'Custom Dependencies' --- -Playwright Check Suites let you take your existing Playwright tests and turn them into globally distributed monitoring using the [Checkly CLI](/cli/). It will read your Playwright configuration and parse your existing settings and dependencies to prepare the Checkly monitoring infrastructure. +Use the [Checkly CLI](/cli/) to turn your Playwright tests into globally distributed monitors. Checkly uses your existing `package.json` dependencies and Playwright configuration. ## Using JavaScript/Node.js dependencies -Whether you use [npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/), or [pnpm](https://pnpm.io/); Checkly uses your existing `package.json` and lock files to install the correct `devDependencies` and prepare your testing and monitoring environment. +Checkly installs dependencies from your `package.json` and lock files. It works with [npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/), and [pnpm](https://pnpm.io/). ```json { @@ -19,10 +19,10 @@ Whether you use [npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/), or ``` -By default, the Checkly infrastructure installs dependencies via npm. +Checkly uses npm by default. -However, if an alternative lock file (`pnpm-lock.json` or `yarn.lock`) is discovered, Checkly uses the matching package manager to install the project dependencies. +Checkly detects your package manager from your lock file: | Available lock file | Package manager | |---------------------|-----------------| @@ -31,12 +31,12 @@ However, if an alternative lock file (`pnpm-lock.json` or `yarn.lock`) is discov | `yarn.lock` | yarn | -**For `pnpm` users:** if you define the same dependency in `dependencies` and `devDependencies`, please use a custom `installCommand` in your `checkly.config.ts|js` to ensure all required dependenies are installed: `pnpm install` +**For pnpm users:** If you define the same dependency in both `dependencies` and `devDependencies`, use a custom `installCommand` in your `checkly.config.ts` to install all dependencies: `pnpm install` ## Using private dependencies -There are no dependency changes required if you rely on private packages or a custom package registry. The Checkly CLI detects your existing private `package.json` dependencies and installs those using the credentials provided. +Your private packages work without changes. Checkly installs them using the credentials you provide. ```json { @@ -47,60 +47,100 @@ There are no dependency changes required if you rely on private packages or a cu ``` -Regardless of whether you use private packages or a custom registry, the Checkly infrastructure needs to authenticate with your provider to `install` your private source code. +To install private packages, Checkly authenticates with your package registry. You need to: -For example, if you use npm to manage your private dependencies, you need to: +1. Add authentication configuration to your `.npmrc` file +2. Include the `.npmrc` file in your Checkly configuration file +3. Store authentication tokens as [environment variables](/platform/variables) in Checkly -- **add custom npm configuration** to your project. -- **include custom configuration files** in your Checkly project to make the infrastructure aware. +### Configuration files for private packages -### Custom npm configuration for private packages +All package managers (npm, yarn, pnpm) use `.npmrc` files to authenticate with private registries. -Let's look at popular options for private packages: - -- using private npm packages on the public npm registry. -- using npm with JFrog Artifactory. - -To access the private source code, npm relies on custom configuration in a project-specific `.npmrc` file. - -> To avoid putting sensitive information into your `.npmrc` we recommend setting your authentication token via Checkly environment variables [available in the UI](https://app.checklyhq.com/environment-variables) or [the CLI](https://www.checklyhq.com/docs/cli/using-environment-variables/#managing-remote-environment-variables-using-the-cli). + +Store authentication tokens as Checkly [environment variables](/platform/variables). Reference them in your `.npmrc` file using `${VARIABLE_NAME}` syntax. + #### Using private npm packages ```txt -# Hard-coded authentication token -//registry.npmjs.org/:_authToken=npm_abc... - -# Authentication token defined in environment variables +# Using an environment variable (recommended) //registry.npmjs.org/:_authToken=${NPM_TOKEN} + +# Hard-coded token (not recommended) +//registry.npmjs.org/:_authToken=npm_abc... ``` #### Using JFrog Artifactory ```txt -# Custom registry URL with hard-coded authentication token -registry=https://yourcompany.jfrog.io/artifactory/api/npm/yourcompany.jfrog.io/artifactory/api/npm/npm-local/:_authToken=abc... +# Using an environment variable (recommended) +registry=https://yourcompany.jfrog.io/artifactory/api/npm/npm-local/:_authToken=${ARTIFACTORY_TOKEN} -# Custom registry URL with authentication token defined in environment variables -registry=https://yourcompany.jfrog.io/artifactory/api/npm/yourcompany.jfrog.io/artifactory/api/npm/npm-local/:_authToken=${ARTIFACTORY_TOKEN} +# Hard-coded token (not recommended) +registry=https://yourcompany.jfrog.io/artifactory/api/npm/npm-local/:_authToken=abc... ``` -### Include custom configuration files +#### Using GitHub Packages + +```txt +//npm.pkg.github.com/:_authToken=${GITHUB_TOKEN} +@your-org:registry=https://npm.pkg.github.com +``` + +### Using yarn or pnpm with private packages + +**yarn** and **pnpm** also use `.npmrc` for authentication (same format as npm examples above). -To make the Checkly CLI and infrastructure aware of your configuration for private packages, specify the additional files in the `checks.include` property in your `checkly.config.ts`. +### Include authentication files -Checkly CLI commands like `test` or `deploy` will bundle and upload these files so they are available in the Checkly infrastructure when running essential commands like `install`; making sure your private dependencies are installed. +Add your `.npmrc` file to the `checks.include` property in your `checkly.config.ts`. -```typescript checkly.config.ts +When you run `npx checkly test` or `npx checkly deploy`, the CLI bundles and uploads these files. This makes them available when Checkly runs `installCommand` to install your dependencies. + +```typescript checkly.config.ts highlight={5} export default defineConfig({ checks: { playwrightConfigPath: './playwright.config.ts', - // Include .npmrc to be used in the Checkly infrastructure + // Include .npmrc for authentication with private registries + // Works with npm, yarn, and pnpm include: ['.npmrc'], playwrightChecks: [ - // Configure your Playwright Check Suites // ... ] } }) ``` + +## Troubleshooting + +### Package not found + +If you see errors like `404 Not Found` or `Package '@your-org/package' not found`: + +- Verify your `.npmrc` file is included in `checks.include` +- Check that the registry URL is correct +- Ensure your package name matches exactly (including scope) + +### Authentication failed + +If you see `401 Unauthorized` or `403 Forbidden` errors: + +- Verify the environment secret is set in Checkly (check [environment variables](/platform/variables) for details) +- Ensure the variable name in your config file matches exactly (e.g., `${NPM_TOKEN}`) +- Check that your token has the necessary permissions to access the package + +### Wrong package manager detected + +If Checkly uses the wrong package manager: + +- Verify the correct lock file is present (`package-lock.json`, `yarn.lock`, or `pnpm-lock.json`) +- Remove any conflicting lock files +- If needed, override with a custom `installCommand` in your [Playwright Check Suite configuration](/detect/synthetic-monitoring/playwright-checks/configuration) + +### Installation timeout + +If package installation exceeds time limits: + +- Check for large dependencies that can be optimized +- Consider using a custom `installCommand` to install only required packages \ No newline at end of file diff --git a/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx b/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx index 0cdfa9d..83da5da 100644 --- a/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx +++ b/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx @@ -5,28 +5,100 @@ description: "Customize your Playwright Check Suite runs based on the execution tags: ["synthetic-monitoring", "playwright-checks", "environment-variables"] --- -Checkly sets the following built-in environment variables on every Playwright Check Suite run. They provide information about the check execution environment, and allow you to distinguish between Checkly executions and local runs. - -- **CHECKLY**: Set to `1` for all check runs executed by Checkly. -- **CHECKLY_RUN_SOURCE**: Indicates the manual or scheduled check run trigger. Use this variable to skip parts of your test suite in certain scenarios or adjust the check run configuration. Possible values include: - - `CLI_DEPLOY`: Checks deployed using `npx checkly deploy` get their first run scheduled with this type. - - `DEPLOYMENT`: The check was triggered as part of a [CI/CD deployment](/integrations/ci-cd/github/deployments). - - `GROUP_RUN_ALL`: The check was triggered as part of a group edit by a user clicking the "Run all checks" button. - - `SCHEDULE_NOW`: The check was triggered manually by a user clicking "Schedule now" in the webapp. - - `SCHEDULER`: The check was run as part of its regular schedule. - - `TEST_NO_RECORD`: The check triggered from the CLI with `npx checkly test`. - - `TEST_RECORD`: The check triggered from the CLI with `npx checkly test --record` or `npx checkly pw-test`. - - `TRIGGER_API`: The check was triggered via the API. - - `TRIGGER_NO_RECORD`: The check triggered from the CLI with `npx checkly trigger`. - - `TRIGGER_RECORD`: The check triggered from the CLI with `npx checkly trigger --record`. -- **CI**: Set to `1` for the following check runs: - - CLI runs via `npx checkly test` or `npx checkly trigger`. - - Check runs that are triggered by [deployments](/integrations/ci-cd/github/deployments). - -The following variables are mostly for informational and debugging purposes and shouldn't be used to influence test behaviour. - -- `ACCOUNT_ID`: The UUID of the Checkly account as found in the URL. -- `CHECK_NAME`: The name of the check. -- `CHECKLY_CHECK_ID`: The UUID of the check as found in the URL. -- `CHECKLY_REGION`: The region in which the check was executed. + + For information on creating and managing environment variables and secrets (including the difference between variables and secrets, inheritance, and where to set them), see the [Environment Variables documentation](/platform/variables). + + +## Built-in variables available in Playwright Check Suites + +Checkly sets the following environment variables on every Playwright Check Suite run: + +| Variable | Description | +|----------|-------------| +| `CHECKLY` | Set to `1` for all check runs executed by Checkly.
Useful to distinguish Checkly runs from local runs. | +| `CHECKLY_RUN_SOURCE` | Indicates how the check was triggered.
Useful to customize the nature of your test depending on how it is run.
See [CHECKLY_RUN_SOURCE values](#checkly-run-source-values) below. | +| `CI` | Set to `1` for CLI runs (`npx checkly test` or `npx checkly trigger`) and [deployment](/integrations/ci-cd/github/deployments)-triggered checks. | +| `ACCOUNT_ID` | The UUID of the Checkly account. | +| `CHECK_NAME` | The name of the check. | +| `CHECKLY_CHECK_ID` | The UUID of the check. | +| `CHECKLY_REGION` | The region where the check was executed. | + +### `CHECKLY_RUN_SOURCE` values + +| Value | Description | +|-------|-------------| +| `CLI_DEPLOY` | Used for the first run of Checks deployed using `npx checkly deploy`. | +| `DEPLOYMENT` | Used for Check runs triggered as part of a [CI/CD deployment](/integrations/ci-cd/github/deployments). | +| `GROUP_RUN_ALL` | Used for Check runs triggered from a Group's edit screen, by clicking the "Run all checks" button. | +| `SCHEDULE_NOW` | Used for Check runs triggered manually by clicking "Schedule now" in the webapp. | +| `SCHEDULER` | Used for Check runs triggered as part of their regular schedule. | +| `TEST_NO_RECORD` | Used for Check runs triggered by the `npx checkly test` CLI command. | +| `TEST_RECORD` | Used for Check runs triggered by the `npx checkly test --record` or `npx checkly pw-test` CLI commands. | +| `TRIGGER_API` | Used for Check runs triggered by the API. | +| `TRIGGER_NO_RECORD` | Used for Check runs triggered by the `npx checkly trigger` CLI command. | +| `TRIGGER_RECORD` | Used for Check runs triggered by the `npx checkly trigger --record` CLI command. | + +## Use built-in variables + +Common ways to use Checkly's built-in environment variables: + + +### Adjust behavior based on trigger type + +Set different behavior for scheduled runs versus manual triggers: + +```typescript highlight={4} +import { test } from '@playwright/test'; + +test('API health check', async ({ request }) => { + const isScheduledRun = process.env.CHECKLY_RUN_SOURCE === 'SCHEDULER'; + + // Use shorter timeout for scheduled runs + const timeout = isScheduledRun ? 5000 : 30000; + + const response = await request.get('https://api.example.com/health', { + timeout + }); + + // ... assertions +}); +``` + +### Configure Playwright based on environment + +Adjust your Playwright configuration to always record traces on Checkly. + +In your `playwright.config.ts`, use the CHECKLY variable to identify it's from Checkly. + +```typescript playwright.config.ts highlight={7} +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + + use: { + // Always trace in Checkly to compare successful and failed runs, only on retry elsewhere + trace: process.env.CHECKLY === '1' ? 'on' : 'on-first-retry', + }, +}); +``` + +### Access region information + +Use region information for debugging or region-specific behavior: + +```typescript highlight={10} +import { test } from '@playwright/test'; + +test('geo-specific content test', async ({ page }) => { + await page.goto('https://example.com'); + + // Log region for debugging + console.log(`Running in region: ${process.env.CHECKLY_REGION}`); + + // Region-specific assertions + if (process.env.CHECKLY_REGION?.startsWith('eu-')) { + // Expect EU-specific content, like a cookie banner + } +}); +``` diff --git a/detect/synthetic-monitoring/playwright-checks/overview.mdx b/detect/synthetic-monitoring/playwright-checks/overview.mdx index 2b9564d..4d58ee4 100644 --- a/detect/synthetic-monitoring/playwright-checks/overview.mdx +++ b/detect/synthetic-monitoring/playwright-checks/overview.mdx @@ -105,6 +105,13 @@ Customize monitoring behavior with: - **Groups**: Organize related checks for management - **Dependencies**: Support for private packages and custom registries +## Known limitations + +Playwright Check Suites are under active development. The following features are not currently supported: + +- **Monorepo workspaces** - Installing dependencies from specific workspaces is not supported. +- **Execution time limit** - Check Suites have a maximum runtime of 15 minutes per execution. + ## Getting Started Ready to turn your Playwright tests into monitoring? Start with our quickstart guide to get running in 5 minutes.