Stackable Superset architecture clarification (Redis caching and Celery workers)

[alexsteeel]

Could you please explain how Stackable Tech handles Redis caching and Celery workers in its Superset Kubernetes deployment? Is it possible to configure Redis and Celery similar to the standard Helm-based setup?

[adwk67]

Hallo @alexsteeel: thanks for your question! We do not currently support Redis/Celery for Superset. May I ask about your use-case: would you use this for scheduling reports, or more for caching/performance?

[alexsteeel]

Hi, @adwk67!
My main concern is caching. According to Superset docs, Redis is the best choice for caching performance (Caching).

Celery workers are less important for me at the moment.

It seems configuring Redis caching could be done via superset_config.py, and overriding this file is supported by the SupersetCluster CRD (Stackable Docs).

Have you tried using Redis as a caching backend with Stackable Superset? If not, do you think it's practical and feasible to implement?

Thanks!

[adwk67]

No, I haven't tried this, although as you correctly point out, there are possibilities to set this up using overrides. From looking at the Superset docs here and here, it looks as though it should be possible to set this in either superset_config.py or using environment variables. This integration test has an example of both (overriding the config and of setting env-vars).

[alexsteeel]

Thanks for the links, I'll give it a try!