a more deliberate release process

[othiym23]

<whining>I'm going to be straight-up emotional for just a second here: this is the second Sunday in the last month where I've woken up to a brewing crisis involving io.js and npm (#1850, #1591). It is incredibly annoying, as well as demotivating, to have to clean up these situations on what should be one of my days off. I try very hard to be a conscientious project lead / OSS maintainer / io.js collaborator, but for human psychological and physiological reasons I need some downtime, and the weekends are that time for me. The suggestions below are offered in the spirit of making it so that I don't have to choose between being responsible to the io.js and npm user communities, and being responsible to myself. Thank you for reading this with that in mind.</whining>

Also, for all of the following, I frame things in terms of npm, but this is just as true of V8 and changes made to Node's own APIs.

About a year ago this time, npm introduced a new release process that included as one of its key features the notion of publishing every new release with a dist-tag of next, and only promoting it to latest after at least a few days (generally a week) so that interested members of the community could use / test it and identify any latent issues with the release. This was done as a reaction to what used to be an all-too-common pattern with Node.js releases, where a new release of Node.js would be followed almost immediately by another due to something messed up inside the version of npm bundled with Node, which was embarrassing and wasteful. This basically hasn't happened since npm's new process was put into place, which I think everybody agrees is a big improvement.
#1850 shows that this only deals with one side of the problem, though. If there's something in a new Node release that makes it incompatible with npm, we can end up putting out versions of Node that render npm unusable, which cripples the release. There's a lot of room for improvement in how npm is tested and integrated with io.js's CI, but even with that situation substantially improved (which a number of us are working on), there's no substitute for real use of the product to give it some real-world smoke-testing.

io.js has a much quicker release cadence than Node.js has had for a while, and this is one of the primary drivers of io.js's quick progress (and is great). New versions come out frequently, and when things break, a new release that fixes them is generally pretty quick to appear. However, because things like Travis pin to the latest version, and many many projects take advantage of this, official io.js releases can magnify issues into an avalanche of issue-tracker and support traffic.

Therefore, I have a few suggestions I'd like to make about how io.js is released:

1. when upgrading dependencies within Node, they should be included in at least 1 (preferably more) nightly release before an official release is cut (I don't know how widely-used the nightlies are, but this is better than nothing)
2. to get potential releases in front of more people, it would be great to have some notion of release candidates for "larger" releases (semver-major definitely, semver-minor probably, semver-patch it would be nice but also probably a gratuitous amount of work [ETA: finished incomplete thought])
3. io.js releases should only happen during the week. I completely understand that for many io.js contributors, the weekend is the only time they have available for OSS / io.js-specific work, which is why I haven't raised this point before. At the same time, many of us are already spending pretty full weeks working on things related to Node – dealing with this is our job, and as such we just need some time off.

Note that these suggestions are orthogonal to adding more / better tests of dependencies, better integration of the tests we do have into CI, acceptance tests, and other prophylaxes intended to improve release quality. All of those things would be great, as would some kind of defined QA process for the project. I still think, however, that Node and npm are both complex and widely-used enough that there's no real substitute for actual, hands-on use to help ensure quality.

[vitaly-t]

This is the price for being the front-runner, you manage to break things on the way.

NodeJS, in contrast, didn't rush things like this, faced no such predicament, and lagged behind happily for years to come.

I believe the truth is in the middle. Therefore, looking forward to the merge of the two platforms.

[meandmycode]

I think quite a lot of issues brought up in io.js focus far too inside the box, the whole ongoing thing with V8 4.3 onward has focused heavily on how will io.js / nan solve this, I think there needs to be a bigger effort to external thought, and the ecosystem as a whole.

Quite often a lot of these small changes to clean the code base are based on quick and dirty searches on google, github and npm, and the assumption this is good enough to put in a deprecation notice (which, are mostly useless in my experience, as the people who care enough to do something are generally those who are eager to maintain their software anyway).

It would be great if some external tooling were to start to happen, for example; the chrome guys have metrics that they use to inform changes, it would be great if npmjs could do some static analysis on packages to establish some real connectivity of code use and its flow between packages. This would be really interesting outside of this issue as well, and could be used to establish quality of packages over time, and perhaps even some day some automation around automatically updating dependencies.

I'm not saying the above is the answer, or even a good one- but its trying to use aspects of the ecosystem that aren't just code. Aside to this, there's so much worry about upgrading V8 too often and how code can solve that, but what about outside of that? can npm help? can community outreach help?

More on topic, and I'm really out of my depth here not having any experience bar following along reading issues most days, the node/iojs build + test does seem woefully inadequate for how important this platform is.

[dashed]

First I'd like to say thank you to everyone and anyone who took the time to spend the Sunday to try and resolve this particular issue.

I think #2, as @othiym23 has suggested, may be a good compromise. If at all possible, I'll be willing to add something like iojs-latest-rc to .travis.yml (or some CI platform), and maybe a lot of projects would as well; and this would the cheapest way to test the waters that a new release is stable enough for the rest of the ecosystem.

[mscdex]

@meandmycode Static analysis is already in the works. See @chrisdickinson's estoc module.

[isaacs]

@meandmycode +1, you hit the nail on the head.

The focus of io.js (and for that matter, Node as well, more often than not) has been on polishing and improving io.js.

There are still features of the platform that are not documented, not tested, and perhaps maintainers of io.js may assume are "internal".

However, we need to change our thinking on that.

If it's exposed, it's public API. Period.

Imo, changes to the public API require, at minimum:

1. A very damn good reason. ("It's ugly" or "it should be internal" is not satisfactory. Too late, sorry.)
2. Explicit deprecation in the docs for a major release.
3. Deprecation via util.deprecate for another major release.
4. Assuming that no riots form when we do (3), remove it in the next major release.

We can add new stuff all day long, or provide better APIs in the meantime. But we can't break stuff without breaking users.

Related: #1807 (comment)

Sorry, still haven't gotten around to writing up the issue for a stable release checklist. I think we're seeing from this fiasco that it's not just a matter of keeping up with breaking V8 changes, but also pure-js changes as well.

[ChALkeR]

@isaacs What about _-prefixed properties?

[ChALkeR]

Btw. When I asked in IRC why was .client deprecated in a minor version (after the 2.2.0 got released), I was told that deprecating in a minor is ok, along with actual removing in a major. If that is not so, you should probably form a policy for such things and put it somewhere.

[silverwind]

@ChALkeR I was refering to this part of semver:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated.

The .client deprecation itself was improperly handled which lead to #1850.

I don't expect people to run test-npm for every commit, but it should be a requirement for every release at least, as it's been proven again that npm itself is a good test for the ecosystem.

[isaacs]

@silverwind Running make test in npm will be less of a good test of the ecosystem once we finish making all of the npm tests run network-free. A much better test of the ecosystem is selecting the top 100 modules, and making sure that you can do npm install && npm test on each of those. (npm itself would presumably be in that set.) In the case of 2.2.0, this would have been 0/100 passing, since npm install is broken.

@ChALkeR Everything that is exposed is exposed. We've had a firm "this is internal" policy regarding process.binding(...) stuff, but even then, I think we should verify that breaking changes won't break user code.

In the case of 1eec5f0 specifically, even though the client property was not used internally, because it was exposed, it was a part of the public API, documented or not. If client is to be deprecated in favor of socket, then that ought to be changed in 3.0.0 by adding a note to the docs, then in 4.0.0 by using util.deprecate, and then in 5.0.0 by removing it entirely. Moving it to an underscore-prefixed property is inappropriate in this case.

Of course, mistakes happen, and slip past review. That's why we need an explicit objective release gating process, so that we can get away from the (currently valid!) stigma that io.js is unstable, prone to break in minor releases, and not trustworthy.

[mscdex]

I thought that when the topic of nightlies/RCs was originally discussed months ago, the conclusion was that nobody would likely test/use them (especially with a -nightly-xxxxx or similar version number it would not work with npm)? Also it looks like the last nightly was built on 5/14/15.

[silverwind]

Nightly usage has been pretty minimal. That might change once we converge, but for now, they're not getting the exposure they need.

[ChALkeR]

@isaacs

Everything that is exposed is exposed. We've had a firm "this is internal" policy regarding process.binding(...) stuff, but even then, I think we should verify that breaking changes won't break user code.

I always thought that _-prefixed means that there are no guarantees and that whoever uses such properties does it on their own risk.

Technically, Symbol-based properties are also accessible. Will you treat those as «exposed»? Technically, in c++ private members of a class are also accessible. Are those «exposed»?

The question is how far does the user code go in breaking the conventions and to what extent do want to support that.

[silverwind]

To get more nightly usage, we probably need some kind of auto-update mechanism. That would of course require installing into a user-writeable directory so we can update without needing root.

Another unresolved issue with nightly builds is that their version strings don't satisfy the engine requirement of packages, which results in npm printing a warning for every single dependency installed, but I guess there could be made an exception in npm.