ENAMETOOLONG when opening tmp-globalize-webpack generated files

[cprodescu]

We use globalize-webpack-plugin in our project. Everything works great locally, however in CI environment (distributed pants over a mesos fleet), the sandbox root is nested deeply and the file paths generated by globalize-plugin are too long to read (throwing ENAMETOOLONG on fs.openSync).

This is an uncommon issue, affecting very few projects, but I'm wondering what your suggestion would be.

Error: ENAMETOOLONG: name too long, open '$BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/.tmp-globalize-webpack/-var-lib-mesos-slaves-0197ab7c-124d-4305-a1fc-d70e915d0fef-S2720-frameworks-201205082337-0000000003-0000-executors-thermos-devprod-devel-iron_source_centos7-32-63205b9c-0ca3-4edd-af72-cae6ac4f6b1c-runs-12f6437b-7ef2-456a-864f-06f50fc5eb8e-sandbox-build_sandbox-.pants.d-resolve-node-252d64521cf9-project.web.project-web-6d8cfed66902-src-component-nav.js'
                         at Error (native)
                         at Object.fs.openSync (fs.js:640:18)
                         at Object.fs.writeFileSync (fs.js:1333:33)
                         at GlobalizeCompilerHelper.createCompiledDataModule ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/globalize-webpack-plugin/GlobalizeCompilerHelper.js:37:6)
                         at Parser.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/globalize-webpack-plugin/ProductionModePlugin.js:77:58)
                         at Parser.applyPluginsBailResult ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/tapable/lib/Tapable.js:48:50)
                         at Parser.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/dependencies/CommonJsRequireDependencyParserPlugin.js:63:22)
                         at Parser.applyPluginsBailResult ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/tapable/lib/Tapable.js:48:50)
                         at Parser.walkCallExpression ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:727:22)
                         at Parser.walkExpression ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:552:40)
                         at Parser.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:532:12)
                         at Array.forEach (native)
                         at Parser.walkVariableDeclarators ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:517:14)
                         at Parser.walkVariableDeclaration ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:493:8)
                         at Parser.walkStatement ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:378:32)
                         at Parser.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:371:8)
                         at Array.forEach (native)
                         at Parser.walkStatements ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:370:13)
                         at Parser.parse ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/Parser.js:920:8)
                         at DependenciesBlock.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/NormalModule.js:104:16)
                         at DependenciesBlock.onModuleBuild ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
                         at nextLoader ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
                         at $BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:292:15
                         at Object.context.callback ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:148:14)
                         at Object.module.exports ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/babel-loader/index.js:132:8)
                         at WEBPACK_CORE_LOADER_EXECUTION ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:155:71)
                         at runSyncOrAsync ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:155:93)
                         at nextLoader ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:290:3)
                         at $BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
                         at Storage.finished ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
                         at DependenciesBlock.<anonymous> ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack/lib/NormalModule.js:113:20)
                         at DependenciesBlock.onModuleBuild ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
                         at nextLoader ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
                         at $BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:292:15
                         at Object.context.callback ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:148:14)
                         at Object.module.exports ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/babel-loader/index.js:132:8)
                         at WEBPACK_CORE_LOADER_EXECUTION ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:155:71)
                         at runSyncOrAsync ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:155:93)
                         at nextLoader ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:290:3)
                         at $BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
                         at Storage.finished ($BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
                         at $BUILD_SANDBOX/.pants.d/resolve/node/252d64521cf9/project.web.project-web/6d8cfed66902/node_modules/graceful-fs/graceful-fs.js:78:16
                         at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)

[rxaviers]

Ideally, this plugin shouldn't use the temporary files if we instead knew the answer for this question http://stackoverflow.com/q/31147779/798133

[cprodescu]

That makes sense.
The closest development I could find on that topic is rmarscher/virtual-module-webpack-plugin. It is built on top of implementation details of enhanced-resolve, so it's fragile.

About the issue here, would you be open to changing the encoding for GlobalizeCompilerHelper . getModuleFilepath (downside of current implementation is that it doubles the length of the path).

[rxaviers]

About the issue here, would you be open to changing the encoding for GlobalizeCompilerHelper . getModuleFilepath (downside of current implementation is that it doubles the length of the path).

Sure, what's your proposal? I see in your commit you're using a hash function. Does it handle conflicts?

[cprodescu]

Good point!
It does not handle conflicts, but it's using the sha-1 algorithm, same as git. The chance of finding a collision is very low (Google has recently published a paper on how they managed to generate a collision after 9 quintillion sha-1 computations). Discussions on sha-1 collisions:
https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
http://stackoverflow.com/questions/9392365/how-would-git-handle-a-sha-1-collision-on-a-blob

I will add code which throws an error if there is a collision (as unlikely as it is, I agree it shouldn't silently fail). Please let me know if you prefer a different approach altogether.

[rxaviers]

Sounds sane, specially if this (i.e., use hash instead of filename) is a feature that could be turned on/off by an option. Would you like to submit a PR?

@necolas do you have any input about it? Thanks

[cprodescu]

Sure, will look into it!

I think the option is a bit unnecessary for the user, especially since the internal behavior is a workaround for a missing feature in webpack.

However, I don't have a strong opinion, just a few questions:

• which behavior should be the default?
• what would be a good name for the option?
• should it be ON/OFF or allow the user to configure the function?