feat: experimentalRunAllSpecs for component testing

[scottohara]

• Closes Run All Specs for Component Testing #25636

Additional details

This change extends the existing experimentalRunAllSpecs config option to support component testing as well as e2e testing, allowing a full suite of component tests to be executed in the Cypress UI with a single click.

By reusing most of the existing experimentalRunAllSpecs machinery, many of files touched by this PR are simply just to relax the previous restriction on it being for e2e testing only.

With this PR, experimentalRunAllSpecs can now be configured at either root level to enable it for both testing types:

import { defineConfig } from 'cypress'

export default defineConfig({
  experimentalRunAllSpecs: true,
  e2e: { ... },
  component: { ... },
}

...or individually for each testing type:

import { defineConfig } from 'cypress'

export default defineConfig({
  e2e: {
    experimentalRunAllSpecs: true,
  },
  component: {
    experimentalRunAllSpecs: true,
  },
}

Description of files changed

• packages/app/src/store/run-all-specs-store.ts - removed checks on the testing type, allowing both e2e and component
• packages/config/src/options.ts - removes the option from the breakingRootOptions and testingTypeBreakingOptions arrays, allowing configuration at any level
• packages/data-context/schemas/schema.graphql - removed the now redundant EXPERIMENTAL_RUN_ALL_SPECS_E2E_ONLY enum
• packages/errors/src/errors.ts - removed the now redundant EXPERIMENTAL_RUN_ALL_SPECS_E2E_ONLY error
• packages/errors/test/visualSnapshotErrors.spec.ts - removes the now redundant EXPERIMENTAL_RUN_ALL_SPECS_E2E_ONLY error
• packages/launchpad/cypress/e2e/config-warning.cy.ts - removed warnings relating to where the option config is allowed

With the above changes, the "Run [n] specs" buttons should now appear in the UI for component testing when the option is configured.

The next set of changes handle how the set of all specs is provided to either webpack-dev-server or vite-dev-server (depending on the bundler used).

Starting with webpack:

• packages/server/lib/socket-base.ts - was taught to handle the RUN_ALL_SPECS_KEY ("__all"), and pass a set of specs to the CT dev server instead of a single spec
• npm/webpack-dev-server/src/loader.ts - was also taught to handle ?specPath=__all, allowing all specs to be loaded

Vite was a bit tricker, and to be honest I'm not entirely happy with the implementation, so I would appreciate any feedback on a more robust and less hacky way to do this. For now:

• packages/data-context/src/sources/HtmlDataSource.ts - sets a new window.__RUN_ALL_SPECS__ property with the list of all specs to run
• npm/vite-dev-server/client/initCypressTests.js - reads the above list of specs and adds each to the list of imports to load into the dev server

I'm not a regular Cypress contributor, so please do let me know if I have missed anything, if there are any additional tests needed for the above changes, or any improvements can be suggested.

Steps to test

1. Update your cypress.config.(js|ts) with experimentalRunAllSpecs: true at either the root level or the component level
2. Open the Cypress UI
3. Navigate to a project with more than one component spec and start component testing in your browser of choice
4. Click the "Run [n] specs" button and observe that all specs are run

How has the user experience changed?

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation? chore: add documentation for run all specs for component testing cypress-documentation#6342
• Have API changes been updated in the type definitions?

---

Note

Enables experimentalRunAllSpecs for component testing, wiring Vite/Webpack and the runner to load/compile all specs and removing the prior e2e-only restriction and related errors.

• Feature/Config:
  • Allow experimentalRunAllSpecs for component (and e2e); permit config at root or per testing type. Update run-all-specs store to not gate on testing type.
• Dev servers / Runner integration:
  • Vite: client/initCypressTests.js imports all specs when specPath="__all" using window.__RUN_ALL_SPECS__; comprehensive tests added in npm/vite-dev-server/test/....
  • Webpack: src/loader.ts shouldLoad() recognizes ?specPath=__all.
  • Server: socket-base.ts compiles multiple specs when receiving RUN_ALL_SPECS_KEY.
  • HTML: inject window.__RUN_ALL_SPECS__ in HtmlDataSource.
• Validation/Errors/Schema:
  • Remove e2e-only validation for experimentalRunAllSpecs in packages/config/src/options.ts.
  • Drop EXPERIMENTAL_RUN_ALL_SPECS_E2E_ONLY enum and error; remove related tests and launchpad warnings.
• Docs/Changelog:
  • Update cli/CHANGELOG.md to announce CT support for experimentalRunAllSpecs.

^{Written by Cursor Bugbot for commit 458f3cf. This will update automatically on new commits. Configure here.}

[cypress-app-bot]

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.

[jennifer-shehane]

@scottohara I'll enable the tests to run, so keep an eye out on the results if anything needs updating.

I'm not quite remembering the blocker for this originally and whether all those blockers are addressed here. It does seem like there should be more test coverage here somehow, but I'll see if someone from our team has ideas on that.

[cursor]

Bug: Robustly Handle Undefined Data Collections

Calling .map() on ctx.project.runAllSpecs without checking if it's defined will throw a TypeError if runAllSpecs is undefined. The code should use (ctx.project.runAllSpecs || []) to safely handle the undefined case, similar to how it's handled in HtmlDataSource.ts line 123.

 

[AtofStryker]

Hi @scottohara. Sorry for the long time on the turn around for a review here. I wanted to make sure I had time to give a thorough review. The changes look great to me but a couple things in comments below. We also need to address a few other things:

• this config / system-test needs to be updated as it is now a valid component config
• this config / system-test needs to be updated as it is now a valid root config
• We also likely want to update the run-all-specs system-test here as the option now pertains to component testing. We also need to add component tests to this project to make sure the feature/experiment correctly
• We also should update this integration test
• I noticed some TypeScript issues that I have documented below in the PR. I think the cypress.d.ts just needs to move experimentalRunAllSpecs to the root config options.
• doesn't look like the copy for experimentalRunAllSpecs needs to be updated

• The experiment needs to be moved out of e2e only on the documentation site
  • https://github.com/cypress-io/cypress-documentation/blob/main/docs/app/references/configuration.mdx?plain=1#L201
  • https://github.com/cypress-io/cypress-documentation/blob/main/docs/app/references/experiments.mdx?plain=1#L268
  • https://github.com/cypress-io/cypress-documentation/blob/main/docs/app/core-concepts/open-mode.mdx?plain=1#L85

@scottohara let me know if you are able to accomplish these changes or if you need a hand in getting this across the finish line. The documentation / system-test updating isn't always trivial especially in an open source contributor context.

CC @jennifer-shehane

[AtofStryker]

Instead of removing these tests, we should be checking that the experimentalRunAllSpecs option applies and works correctly for both e1e and component testing

[scottohara]

Thanks for the positive comments @AtofStryker . Glad to hear that I'm on the right path at least.

I'll look to address your comments and suggestions as soon as I can find some time. That said, I would also welcome anyone who wants to jump in and assist in getting this feature over the line.

[AtofStryker]

Thanks for the positive comments @AtofStryker . Glad to hear that I'm on the right path at least.

Of course, @scottohara! Sorry it took me a few weeks to get a quality review in.

I'll look to address your comments and suggestions as soon as I can find some time. That said, I would also welcome anyone who wants to jump in and assist in getting this feature over the line.

If needed, I can jump in and take care of some of the items I mentioned. Just let us know! I think the only item that isn't entirely clear to me is #32926 (comment)

[scottohara]

Thanks @AtofStryker , that would be greatly appreciated if you have time to assist.

[AtofStryker]

I am starting to look into resolve a few of these items today

[AtofStryker]

I've gone ahead and updated the documentation for this PR in cypress-io/cypress-documentation#6342. I updated the types and a few other things, as well as what we call cy-in-cy tests, which allow us to use Cypress to test Cypress. Long story short, it gives us end to end test coverage of testing experimentalRunAllSpecs for E2E as well as Component testing within vite/webpack