[rush] "ENOENT: no such file" ../.modules.yaml when installing

[harttle]

Summary

Calling rush install sometimes causing this error:

ERROR: ENOENT: no such file or directory, open
'.../common/temp/<some_project>/node_modules/.modules.yaml'

where <some_project> is depended on via workspace: specifier by current project.

Repro steps

Not able to reproduce in a minimal repo, I can try again latter. But this happens most frequently when:

• The project is newly created (with the rush repo already created).
• The project has no dependency (in this case, adding a dummy dependency can solve this issue).

Details

• Creating .../common/temp/.../node_modules/ directory by mkdir -p before rush install solves the issue.
• Running rush install twice also solves the issue.
• The issue happened only several times, but each time it happens, retry rush install can always reproduce, both locally and on CI environment.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@microsoft/rush globally installed version?	5.149.0
rushVersion from rush.json?	5.149.0
useWorkspaces from rush.json?	no such field in rush.json, it's true in pnpm-config.json
Operating system?	Mac
Would you consider contributing a PR?	Yes
Node.js version (node -v)?	18.12.1
PNPM version from rush.json	8.8.0

[harttle]

Possibly related to pnpm/pnpm#5363

[iclanton]

We could just proactively create the node_modules folder (@harttle - would you be interested in contributing that?), but it would be good to better understand why this is happening.

Does it repro if you upgrade to a newer version of pnpm?

[harttle]

Hi @iclanton , finally I found time to dig deeper.

This error isn't form rush-lib or pnpm (as pnpm fails gracefully when .modules.yaml does not exist, and this error happens after pnpm install finished), it's from pnpm-sync trying to read .modules.yaml during pnpmSyncPrepareAsync() which happens after pnpm install successfully exits.

I saw there's checks before calling pnpmSyncPrepareAsync() in rush-lib BaseInstallManager:

rushstack/libraries/rush-lib/src/logic/base/BaseInstallManager.ts

Lines 297 to 301 in 865c41b

	// we have an edge case here
	// if a package.json has no dependencies, pnpm will still generate the pnpm-lock.yaml but not .pnpm folder
	// so we need to make sure pnpm-lock.yaml and .pnpm exists before calling the pnpmSync APIs
	if ((await FileSystem.existsAsync(pnpmLockfilePath)) && (await FileSystem.existsAsync(dotPnpmFolder))) {
	await pnpmSyncPrepareAsync({

I tried to add one more check for .modules.yaml here and it fixes my issue, the node_modules will be successfully populated with dependencies and .modules.yaml.

+ const modulesFilePath = `${subspace.getSubspaceTempFolderPath()}/node_modules/.modules.yaml`;
- if ((await FileSystem.existsAsync(pnpmLockfilePath)) && (await FileSystem.existsAsync(dotPnpmFolder))) {
+ if ((await FileSystem.existsAsync(pnpmLockfilePath)) && (await FileSystem.existsAsync(dotPnpmFolder)) && (await FileSystem.existsAsync(modulesFilePath))) {

Not sure whether this indicates pnpm-sync is not working in this case, but I believe this PR is not a breaking one as it's broken already. At least it makes my case pass.

#5259

[octogonz]

LGTM