large symlinked module causes a hard FSEvents failure when using contexts

[SpaceK33z]

I’ve run into a fairly strange, fairly specific problem when running webpack in watch mode on OS X. When using require.context in the presence of a module with a lot of dependencies in node_modules that was symlinked using npm link, webpack seems to open a lot of file descriptors, very quickly leading to a rapid stream of dozens of errors per second coming from OS X’s FSEvents framework:

2016-04-18 11:21 node[34856] (FSEvents.framework) FSEventStreamRelease(): failed assertion 'streamRef != NULL'
2016-04-18 11:21 node[34856] (FSEvents.framework) FSEventStreamRelease(): failed assertion 'streamRef != NULL'
2016-04-18 11:21 node[34856] (FSEvents.framework) FSEventStreamCreate: _FSEventStreamCreate: ERROR: could not open kqueue
2016-04-18 11:21 node[34856] (FSEvents.framework) FSEventStreamScheduleWithRunLoop(): failed assertion 'streamRef != NULL'
2016-04-18 11:21 node[34856] (FSEvents.framework) FSEventStreamStart(): failed assertion 'streamRef != NULL'

This issue is similar to #1743, but I am not sure that it is actually the same, since that appears to be resolved with a chokidar fix, but I am using the latest version of chokidar and still get the error.

Steps to reproduce

Since this is such a specific problem, I’ve managed to reduce it to a pretty tiny, reproducible example, which you can find at lexi-lambda/webpack-fsevents-bug. Here’s some information about my environment as well:

operating system: OS X 10.11.3
node version: v5.10.1
npm version: 3.8.3

Here’s how to reproduce the issue:

1. Clone https://github.com/lexi-lambda/webpack-fsevents-bug somewhere.
2. Enter the linking-test directory and run npm install, then npm link.
3. Enter the main directory and run npm install, then npm link linking-test.
4. Run npm run watch.
5. After a few seconds, you should start seeing the FSEvents errors.

More details

The issue seems to specifically be related to how running webpack in watch mode interacts with require.context. If you look in the single module in the whole project, you’ll see the following two lines:

var context = require.context('..', false, /config\.json$/);
console.log(context.keys());

I’ve explicitly passed false to require.context, but for some reason the placement of the linked package in the node_modules subdirectory clearly affects how webpack handles this context. The error does not occur without the custom context.

I have no idea if this has anything to do with a symlinked module specifically, and it’s certainly not that simple, because it doesn’t happen if the symlinked module does not have very many dependencies. However, with a lot of dependencies, like the ones I’ve included in my sample repo, it consistently fails. It might just be caused by having a lot of files/subdirectories and have nothing to do with the symlinking—I don’t know.

Either way, this is a pretty hard failure that is consistently reproducible, even on different machines, and I’ve been running into it repeatedly with a particular project I’m working on. Thank you for making webpack, it’s a wonderful tool! And let me know if there’s any more information I can provide to help.

Moved from webpack/webpack#2356

[SpaceK33z]

We should check if this still happens when we switch from chokidar to sane, see #41.

@lexi-lambda, could you check if this still happens when you apply PR #41 manually.

[sokra]

fsevents has been removed in version 2