Node 6 fs.realpath behavior changes

[isaacs]

• Version: 6
• Platform: Tested on Darwin and Linux
• Subsystem: fs

Previously, looping symbolic links and long path names were handled by fs.realpath without issue. The changelog says that fs.realpath "can throw new errors", but fails to identify that it also is less powerful (and also does not identify what those new errors are).

Seems like it'd be possible to maintain backwards compatibility if Node could fall back to the slower but more resilient implementation when ELOOP or ENAMETOOLONG are raised.

If this is not desirable, then the changelog should be updated to reflect the reduction in functionality.

I have no strong opinion about whether or not Node is capable of resolving the realpath of long looping symbolic links, but I would very much like a thing to point to when people complain about glob being broken. Thanks.

$ node -v
v5.3.0

$ cat rp.js
var fs = require('fs')

var l = 128

var long = 'rptest/' + Array(l).join('a/b/') + 'a'

clean()
process.on('exit', clean)
function clean () {
  try { fs.unlinkSync('rptest/a/b') } catch (e) {}
  try { fs.rmdirSync('rptest/a') } catch (e) {}
  try { fs.rmdirSync('rptest') } catch (e) {}
}

fs.mkdirSync('rptest')
fs.mkdirSync('rptest/a')
fs.symlinkSync('..', 'rptest/a/b')

console.log(fs.realpathSync(long))
fs.realpath(long, console.log)

$ node rp.js
/home/isaacs/rptest/a
null '/home/isaacs/rptest/a'

$ nave use 6

$ node rp.js
fs.js:1568
  return binding.realpath(pathModule._makeLong(path), options.encoding);
                 ^

Error: ELOOP: too many symbolic links encountered, realpath 'rptest/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a'
    at Error (native)
    at Object.realpathSync (fs.js:1568:18)
    at Object.<anonymous> (/home/isaacs/rp.js:19:16)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)
    at Function.Module._load (module.js:409:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:160:18)

[ChALkeR]

@isaacs Is this related to npm/npm#5082 fix being blocked?

[jhamhader]

Introduced in #3594
By the way, what happens when you try to fs.stat() on the link?

[addaleax]

#6861 and #7044 are also issues that stem from the new realpath implementation.

Falling back to a JS-based realpath implementation sounds like a good idea to me.

[Fishrock123]

@thealphanerd citgm tests glob, right? @isaacs Did glob have tests for it previously?

[jasnell]

@Fishrock123 ... the primary issue with glob is two fold: (a) fs.realpath now throws errors when it previously did not and (b) when those errors are thrown is platform specific.

Previously, if I use glob with realpath on a circular symlink, it would recurse until a name too long error would occur, which is being caught and intentionally used by glob. This error is essentially the signal that glob should stop recursing and just returns the result up to that point. On certain operating systems (e.g. OSX), the libuv realpath implementation now throws ELOOP errors before the other error is thrown. Since glob does not ignore the ELOOP errors, the app dies. On Linux, the eloop and name too long errors throw at the same depth so the problem is not encountered because the ignored error comes first and the loop inside glob exits before the eloop can occur. On OSX, however, the eloop happens first.

The glob code was written to assume that fs.realpath would never throw an errors of it's own (there's no defensive error handling logic around the call to realpath).

[Fishrock123]

The glob code was written to assume that fs.realpath would never throw an errors of it's own (there's no defensive error handling logic around the call to realpath).

I think glob's assumptions here are ok in terms of errors, fwiw.

Edit: As long as you don't pass it obviously invalid parameters

[Fishrock123]

As a note, the realpath change is listed on the breaking change doc: https://github.com/nodejs/node/wiki/Breaking-changes-between-v5-and-v6#fs

However it does not note this specifically, and it does not mean that there isn't still a problem or bug here.

[jasnell]

Note, I'm not saying that glob's assumptions around error handling are right or wrong, just describing the situation. Previously fs.realpath did not throw, and now it does throw based on platform specific differences.

[jhamhader]

It is important to note that other fs functions also have this issue. I verified that about the following:

• stat()
• access()
• chmod()
• chown()
• open()
• readdir()
• watch()
• All other fs functions which uses the above functions (truncate(), writeFile(), etc)

There is almost no fs function which is not affected.

[isaacs]

@Fishrock123 Yes, glob's tests fail on Node 6. There's a PR from @thealphanerd which makes the test pass, but there are problems with the fix. It can't really be hacked around in node-glob without a very subtle and significant breaking change in node-glob that'll require people to update their code.

Imo, citgm should have prevented this change from happening in the first place. It is a bug in the process that it got out into a node release. That underlying process bug should be fixed, along with this technical bug.

@jhamhader Prior to node 6, one could use fs.realpath to avoid ELOOP errors in other fs operations, precisely because it would walk up the path from left to right resolving symbolic links.

[isaacs]

See: isaacs/node-glob#259 (comment)

[saghul]

Prior to node 6, one could use fs.realpath to avoid ELOOP errors in other fs operations, precisely because it would walk up the path from left to right resolving symbolic links.

Was this behavior documented? Generally speaking, all fs APIs are thing wrappers around their POSIX counterparts, and very thin wrappers at that. IMHO the general expectation is that they work just the man page says. In this line, I consider the current implementation works as expected.

FTR, the work done here was to make fs.realpath faster, and keeping the JS fallback was met with resistance back then. We waited for Windows XP support to go away in order to land it, because uv_fs_realpath doesn't work there.

The way I see it the only feasible solution here is to catch the error at the application level and handle it.

If this is not desirable, then the changelog should be updated to reflect the reduction in functionality.

I have no strong opinion about whether or not Node is capable of resolving the realpath of long looping symbolic links, but I would very much like a thing to point to when people complain about glob being broken. Thanks.

I understand that a documentation change is acceptable to you then?

[ChALkeR]

@isaacs

It can't really be hacked around in node-glob without a very subtle and significant breaking change in node-glob that'll require people to update their code.

Does this mean that other fs functions have also been affected, not only realpath()? Afaik only module uses fs.realpath in Node.js. What prevents from re-implementing the same functionality?

Prior to node 6, one could use fs.realpath to avoid ELOOP errors in other fs operations, precisely because it would walk up the path from left to right resolving symbolic links.

Perhaps what we need here need a separate method for just that, either in fs or in userland?