-
-
Notifications
You must be signed in to change notification settings - Fork 0
Allow specifying extra extensions to enable in Druid #416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM in principle.
Looking at e.g. https://github.com/orgs/stackabletech/discussions/15 I think we might want to add dedicated extensions in the future - in this case for google cloud extensions with bucket name and credentials secret.
The mechanism from this PR might co-exist, but I would like to model them together and not go with a Vec for now and later e.g. realizing we want a BTreeMap<String, Whatever>
Let's wait for Arch meeting to discuss CRD change
@@ -38,5 +38,18 @@ pub fn get_extension_list( | |||
extensions.push(EXT_S3.to_string()); | |||
} | |||
|
|||
// Add user specified extensions to the list of loaded extensions if any are present | |||
for additional_extension in &druid.spec.cluster_config.additional_extensions { | |||
if extensions.contains(additional_extension) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
O(n^2) here we come :) Just kidding, you could use a BTreeMap - feel free to ignore
Agree in principle, but for this specific example, wouldn't that be better handled in the deep storage config? |
I think the gcs extension things includes more stuff such as ingesting data from gcs, not only deep storage |
Hey @sbernauer, I'm reviving this again because we had another user as for how to add extensions, I think it was the Would you maybe like to expand on your design a bit? Maybe we can get something fleshed out to implement soonish. I have also an alternative idea, which is to just load a lot more extensions by default - just thinking out loud here, I don't really know, are there any downsides to, let's say, load the avro extensions always? It's already in the image, so it's not about space. |
Have not spend many thoughts on this, my only requirement is that we can add properties to extensions. Does not need to be right now, just give us the chance to add that non-breaking. For avro specifically I don't see a reason to not always enabled it.
As a first step we could simply add all extensions that don't need any configuration options (such as avro) |
Okay yes seems reasonable and also still easy to use. |
Closing as this is already implemented by #547. |
Description
Currently the only way to enable extensions not enabled by default is to use config overrides.
This has the drawback that users need to specify all extensions to load, so basically anticipate what extensions the operator will enable and then add the ones they want.
This PR adds a field
additionalExtensions
to the clusterConfig - these extensions will be added to any extensions the operator deems are needed, so the user can just specify the extensions the care about here.Definition of Done Checklist
Once the review is done, comment
bors r+
(orbors merge
) to merge. Further information