Last step in cabal-nix sometimes takes a loooooooooong time

[edsko]

Building hackage-server:

cabal-nix new-build -v3
...
link: linkables are ...
LinkableM (2016-01-02 08:32:56 UTC) main:Main
   [DotO /Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests-tmp/Main.o]
LinkableM (2016-01-02 08:32:55 UTC) main:Distribution.Server.Packages.Unpack
   [DotO /Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests-tmp/Distribution/Server/Packages/Unpack.o]
LinkableM (2016-01-02 08:32:56 UTC) main:Distribution.Server.Packages.UnpackTest
   [DotO /Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests-tmp/Distribution/Server/Packages/UnpackTest.o]
LinkableM (2015-12-30 15:01:50 UTC) main:Distribution.Server.Util.Parse
   [DotO /Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests-tmp/Distribution/Server/Util/Parse.o]
LinkableM (2015-12-30 15:01:49 UTC) main:Distribution.Server.Util.GZip
   [DotO /Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests-tmp/Distribution/Server/Util/GZip.o]
/Users/e/wt/projects/tuf/hackage-server/dist-newstyle/build/hackage-server-0.5.1/build/PackageTests/PackageTests is up to date, linking not required.
*** Deleting temp files:
Deleting: 
*** Deleting temp dirs:
Deleting: /var/folders/94/vkm1c1l17cddyvzk2252hnp40000gs/T/ghc81573_0

and then.. nothing.. The cabal-nix process is using some 25% CPU at this point, and I have no idea what it's doing. That temp directory mentioned on the last line is already gone, so it's not just deleting. What it is doing is anyone's guess.. I CTRL-C'ed it after 12 minutes. Retried, same thing. Wiped dist-newbuild, same thing. I've seen long times before but never something like this. No idea what's happening. Will reboot machine and see if that resolved anything. (Incidentally, I thought maybe it was busy waiting for the hackage-server executable, but the server was not running at this point.)

[edsko]

.. rebooted machine, same problem. Started with a completely blank ~/.cabal directory, same problem. Started with blank ~/.cabal directory, blank dist-newstyle directory, and blank ~/.ghc directory, still same problem. Now I don't know what to try next anymore.. What is going on??

[ezyang]

What commit ID? (I hope I didn't push a regression!)

[edsko]

f8d8f85 . I haven't updated my nix branch in a while so I doubt it's your doing :)

[ezyang]

If you can, try strace'ing the process and see what it's churning on.

[edsko]

Hmmm, that was a good suggestion. dtruss (on OSX here) provided an important clue:

stat64("/Users/e/wt/projects/tuf/hackage-server/state/blobs/0b/0ba03d1a00d06795579f1d6eb5ace1f7\0", 0x109341F50, 0x7FF0)         = 0 0
open("/Users/e/wt/projects/tuf/hackage-server/state/blobs/0b/0ba03d1a00d06795579f1d6eb5ace1f7\0", 0x20004, 0x1B6)        = 12 0
fstat64(0xC, 0x1093403E0, 0x1B6)         = 0 0
ioctl(0xC, 0x4004667A, 0x70000030BD8C)       = -1 Err#25
ioctl(0xC, 0x40487413, 0x70000030BD90)       = -1 Err#25
read(0xC, "\0", 0x7FF0)      = 273 0
read(0xC, "lass='hs-varid'>swap</span><span class='hs-layout'>)</span>\n<a name=\"line-26\"></a>\n<a name=\"line-27\"></a><span class='hs-keyword'>import</span> <span class='hs-conid'>Language</span><span class='hs-varop'>.</span><span class='hs-conid'>Qux</span><span clas", 0x7FF0)      = 0 0
close(0xC)       = 0 0

and so on and so forth for lots of files. I have no idea what's going on, but if it's somehow trying to read all the blobs in the server state/ directory for whatever reason then that certainly explains why it's taking so long (and why it didn't before -- I just moved to an actual Hackage snapshot).

So that leaves the question, why is it even looking at those files? My project file doesn't have a single wildcard:

packages: hackage-server.cabal

constraint: hackage-security == 0.1.0.0
constraint: hackage-security-HTTP == 0.1.0.0

package hackage-server
  tests: True

package text-icu
  extra-include-dirs: /Users/e/homebrew/Cellar/icu4c/54.1/include
  extra-lib-dirs: /Users/e/homebrew/Cellar/icu4c/54.1/lib

[edsko]

I'm also somewhat confused by that dtruss output; as far as I can tell, that particular file (.../blobs...) does not contain that data (lass='hs-valid'>...)... But perhaps I'm reading the output wrong.

[dcoutts]

The issue here is we need to track the files that are part of the package as specified by the .cabal file, not just look at all files in the package dir (as there can be lots of them like in this example and we don't want to hash them all).

[dcoutts]

Actually there was another issue. We were not taking advantage of the fact that we know the previous content hashes when we're updating the file monitor state and so were re-hashing every time. Of course we only need to rehash if the mtime has changed. So now we do that.

But for this specific example we do also need to be properly selective in which files we monitor. So I'll leave this open for that aspect.