Embedding

[theduke]

I'm curious if embedding redpanda as a library is a use case you have considered or are at all interested in supporting.

Are there any architectural concerns that make this complicated?

As a concrete example I have a use case for embedding it in a Rust application. (Bindings could be generated with cxx ).

[emaxerrno]

Atm, we are only considering Kafka as the only public & stable API . However maybe an approach we can take is support listening on Unix sockets with minimal system overhead (no compression) and run in a deterministic fashion on a side thread with limited memory and cpu.

We do this for a security appliance company, they they want to do intrusion detection and record the events via a sidecar proc.

I also note that on recent Linux kernels the localhost 127.0.0.0 path is heavily optimized and the perf is similar to Unix sockets actually so for a POC you could test that today

[theduke]

The primary motivation for my use case is ease of deployment with a single binary.

Forking off and launching Redpanda in the forked process, with communication happening over a local (unix or regular) socket would actually be fine. That way all I'd really need is a redpanda_start(basic_config) function, and maybe something for graceful shutdown. (though I think the Kafka API actually supports this)

[emaxerrno]

we also. have an HTTP API, blog post coming out next week, if you don't want to use the kafka API.

I think you can do that no? take the .deb/.rpm file and embed it in the .DATA part of the binary and write it out as a file?

or can use vectorized/redpanda:v21.4.12-like container?

[emaxerrno]

also @theduke - check out the http API - don't share publicly yet. pls :) haha https://deploy-preview-240--vectorizedio.netlify.app/

[emaxerrno]

or rather toooo publicly, should be fine as this already works. but the docs are WIP

[emaxerrno]

@theduke - what programming language were you thinking here?