[config file] Implicitly enable namespace for test runner

[pmarchini]

Hey folks,

After this PR landed, any configuration under the test namespace now causes the --test flag to be automatically enabled.

For example, with a configuration like:

{
  "test": {
    "test-concurrency": 1,
    "experimental-test-coverage": true
  }
}

Node automatically treats --test as provided.

While this behavior makes total sense for different namespaces (like permissions), for the test runner, IMHO, it adds some complexity.

I would normally expect to define test runner configurations (e.g., exclusions, inclusions, global setups, etc.) in the config file, but only enable tests when I explicitly provide --test.

Currently, the intended way to disable it is by explicitly setting:

{
  "test": {
    "test": false,
    "test-concurrency": 1,
    "experimental-test-coverage": true
  }
}

From my perspective, this devEx is suboptimal for this specific use case.
Therefore, I was thinking about introducing a way to differentiate the default behavior of specific namespaces, so the test runner configuration could be recognized but only enabled when explicitly triggered.

What do you think?

@nodejs/test_runner @JakobJingleheimer @marco-ippolito

[marco-ippolito]

Why would you enable test configuration flags if you dont want to run tests? The right behavior is to have multiple config files.
I believe the behavior is fine as is

[pmarchini]

I would expect the config file, at least regarding the test runner, to be the central configuration used every time the tests are executed.

As an example, I would expect an average user to have a file similar to:

{
  "test": {
    "exclude": [
      "test/fixtures/**",
      "test/helpers/**",
      "**/*.e2e.test.js"
    ],
    "test-concurrency": 4,
    "test-timeout": 10000,
    "experimental-test-coverage": true,
    "globalSetup": "./test/global-setup.js"
  }
}

as the default configuration for the project.

This aligns with the average test runner configuration file.

In this scenario, I would expect a user to decide to run normally via node --experimental-default-config-file, and for tests, node --experimental-default-config-file --test.

---

Even if it's unrelated to the test runner, I would expect, sooner or later, the config file to be enabled by default if present.
At that point, the developer experience would be frictionless.

[atlowChemi]

I think it is worth noting this is how vitest in the vite.config file works (it allows passing a test config).
IMHO this makes sense, it means that running the tests would use the same configurations the code would be running in

[JakobJingleheimer]

In our last team meeting, we discussed automatically enabling "test" mode when any test-runner flags are set; it seemed to have good reception within the team. I'm nearly done with the proposal we mentioned (I juuust didn't have enough time to finish it before my trip), which touches on this.

Why would you enable test configuration flags if you dont want to run tests?

This was exactly our thinking :)

[pmarchini]

I agree when we're talking about the CLI, my doubt is when it comes to configuration file only

[JakobJingleheimer]

Ahhh. I see 🤔

I suppose yeah I might expect test to default to false.

Buuut if the CLI enables test mode when any test options are provided, would that be surprising deviation in behaviour? I'm thinking "no", but just want to raise as a potential issue.

[atlowChemi]

I think that requiring a separate config file for running tests would be a bad experience.
I would expect a single config file, which defaults test to false, and would allow to setup all the config needed, which would allow me to run the app with node index.js or the tests with node --test without having to set additional flags etc

[JakobJingleheimer]

I think having multiple config files wouldn't be required? It's just weather --test or "test": false should be required.

p.s in practice, I always end up having a different config file for test runs (which need various "import"s for test setup).

[pmarchini]

It's just weather --test or "test": false should be required.

I’d say the whole point is this.

In my projects, for example, I use test: false, but I think that results in a poor experience.

After discussing this with @marco-ippolito , I agree that having different behavior only for the test runner namespace might be confusing.