Embed data in server-side-rendered (SSR) HTML as base64-encoded arrow data and reload in frontend

[choucavalier]

Hello! I'm trying to achieve something I did not get to see in the examples.

I'm trying to send base64-encoded data (arrow tables) to the frontend, and i want to load this data and add it to the client-side DuckDB wasm database.

My workflow is that I'm doing a lot of the heavy calculation on the server side with polars, and I want to write (write_ipc) this data to a bytes buffer, which is then base64-encoded, transferred to the client as part of the HTTP response, and loaded back in memory on the client side.

I'm having a really hard time achieving that. I've only seen one comment in the discussions here where the insertArrowFromIPCStream method is mentioned here.

What is the best way to do what I'm trying to do?

P.S. I know mosaic is supposed to be running against a duckdb server, but in my situation that's not a functionality I'm looking for! The computation is mostly done once, but I need sparkles of interactivity / filtering on the client-side.

[declann]

Also interested in this pattern

You can initialize your own duckdb wasm instance with your data added using insertArrowFromIPCStream. Then you can connect that duckdb instance when you initialize mosaic via wasmConnector

There's an example of using wasmConnector connected to an initialized duckdb instance here ("If you would prefer to initialize Mosaic yourself"...)

[domoritz]

Just adding that I would strong recommend not to base64 encode the data as that will vastly increase the data size (and incur overhead for serialization and deserialization). Ideally send it as binary.