nouveau: Upgrade to Lucene 10 #5628
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rnewson
force-pushed
the
lucene-10
branch
7 times, most recently
from
ffbe402 to
92b38b2
Compare
rnewson
force-pushed
the
lucene-10
branch
2 times, most recently
from
7b26d9f to
769bc65
Compare
|
Just to be sure: without the scanner plugin, the index version is not automatically written into the design doc, right? We have a whole set of Nouveau indexes and views in a single design doc. Adding the version would trigger a rebuild of all indexes, which we want to avoid. |
|
Avoiding query-blocking rebuilds is the highest priority of this work, so thank you for the comment. The lucene_version field is injected on interactive edits (when you create or modify a design document). Since that edit would be a new index anyway, that it happens to be a Lucene 10 index is fine. It's one set of index builds. There might be some refinements to this before the PR is merged for the case where a design document defines multiple indexes. The absence of the lucene_version field is taken to mean the index, if it exists on the nouveau server, is for Lucene 9. The index signature includes the lucene version from 10 onward and omits it for version 9 (so that the sig remains unchanged on upgrade). |
|
I've pushed an update so that the lucene_version injection only occurs for new design documents, not updates to them. |
nickva
reviewed
Sep 8, 2025
| 1) the Dropwizard framework (https://dropwizard.io) | ||
| 2) Java 11+ | ||
| 3) Lucene 9 | ||
| 2) Java 21+ |
There was a problem hiding this comment.
This makes sense as that seems to be the lowest supported Java version for Lucene 10. But I think this will force us to move to Debian Trixie with the packaging if we go with using built-in java packages. If we switch to 3rd party ones it wouldn't matter. Swtiching to Trixie is probably the better option. Not a showstopper, of course, just leaving a note so we don't forget.
There was a problem hiding this comment.
Yes, that's the main upgrade pain, that Debian oldstable is too old.
There was a problem hiding this comment.
If it's not too much to ask, please don't drop support for Debian 12. We have several customer installations where it is not possible to upgrade to Debian 13 in the near future, but we want to keep updating CouchDB. It would be no problem for us to install a 3rd-party JVM. Support for Trixie is also nice for new Systems.
There was a problem hiding this comment.
We don't drop support for Debian 12. The only thing you need to do is to install a 3rd-party VM (Java 21+) on Debian 12.
nickva
reviewed
Sep 8, 2025
| -include_lib("couch/include/couch_db.hrl"). | ||
|
|
||
| %% New index definitions get an explicit lucene version property, if missing. | ||
| before_doc_update( |
There was a problem hiding this comment.
This is a neat approach.
Wonder what would happen if users would toggle the version by hand (bump it up, down, make it a non-integer) or just remove it after it was created? Would that break the index or crash anything?
There was a problem hiding this comment.
I've tried to consider various manual shenanigans while trying to make the Java side stick to a "the version of Lucene I am using now plus the one before" without hardcoding "10" and "9" everywhere. I will review that aspect again as I think there are gaps (e.g, I want it to be an error to set lucene_version to anything but 10 or 9 and have that handled well (i.e, no stack traces, no erlang process crashes, just an error message in the log or in the http response)
There was a problem hiding this comment.
Revised commit "Allow existing legacy indexes but prevent new ones" so it only allows 9 or 10.
rnewson
force-pushed
the
lucene-10
branch
2 times, most recently
from
91eacb4 to
de25207
Compare
rnewson
force-pushed
the
lucene-10
branch
2 times, most recently
from
5a8c42d to
cb4b90f
Compare
|
I gave this a try with this test script. Without adding an explicit #!/bin/bash
set -x
http -b post $DB/_nouveau_analyze analyzer=standard text=for@bar@baz
http -q delete $DB/ndb
http -q put $DB/ndb
http -q post $DB/ndb/_bulk_docs docs:='[{"_id":"1","a":"xxx"},{"_id":"2","a":"xxx"},{"_id":"3","a":"zzz"},{"_id":"4","a":"xxx"}]'
http -b put $DB/ndb/_design/nd1 nouveau:='{"idx1": {"index":"function(doc){index(\"string\",\"a\",doc.a,{\"store\":true})}"}}'
http -b post $DB/ndb/_design/nd1/_nouveau/idx1 q="a:xxx" limit:='2'
#!/bin/bash
set -x
http -b post $DB/_nouveau_analyze analyzer=standard text=for@bar@baz
http -q delete $DB/ndb
http -q put $DB/ndb
http -q post $DB/ndb/_bulk_docs docs:='[{"_id":"1","a":"xxx"},{"_id":"2","a":"xxx"},{"_id":"3","a":"zzz"},{"_id":"4","a":"xxx"}]'
# http -b put $DB/ndb/_design/nd1 nouveau:='{"idx1": {"index":"function(doc){index(\"string\",\"a\",doc.a,{\"store\":true})}"}}'
# http -b post $DB/ndb/_design/nd1/_nouveau/idx1 q="a:xxx" limit:='2'
http -b put $DB/ndb/_design/nd2 nouveau:='{"idx": {"lucene_version":9, "index":"function(doc){index(\"string\",\"a\",doc.a,{\"store\":true})}"}}'
http -b post $DB/ndb/_design/nd2/_nouveau/idx q="a:xxx" limit:='2'
|
rnewson
force-pushed
the
lucene-10
branch
3 times, most recently
from
9ad011e to
0bb8604
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Upgrades Nouveau to Lucene 10.
Existing indexes can still be queried and updated. New indexes will be built with Lucene 10 (couchdb will auto-inject the version information into the design document).
A couch_scanner plugin will build Lucene 10 versions of all current indexes (sequentially) and update the design document on successful completion, making the upgrade automatic.
TODO:
NOTES
Lucene 10 raises the JDK requirement from 11 to 21.
Testing recommendations
TBD
Related Issues or Pull Requests
N/A
Checklist
rel/overlay/etc/default.inisrc/docsfolder