GGML RPC Server Security

[struct]

I've read this pull request that states the insecurity of the RPC server is well understood and until a plan is in place to harden the server the work should happen in the open, and not go through the advisory process. This makes sense and I appreciate the transparency.

It currently lacks authentication, contains memory address disclosure vulnerabilities and has no fuzzer targeting the client or server (although I did put one together here). Is there currently a plan to improve the security of the RPC server? If I were to write up what security controls are needed would that be helpful?

I wrote down quick thoughts up after spending a few hours looking at the implementation. Fixing these issues is entirely possible and I am willing to put in some time to implement the hardening measures required.

cc @rgerganov @CISC

[CISC]

If there is interest, it might make sense to apply to the GitHub Secure Open Source Fund (before August 12th) for this.

[struct]

Access to funding is good for the project long term, but I recommend you first put together a list of what security controls are missing, and a rough idea of how you'd like to implement them. Some of them I called out above and implementing them is fairly straight forward.

[rgerganov]

The main challenge is finding a proper way for fuzzing the graph_compute method. On the server side we re-create tensors by directly setting ggml_tensor fields instead of using public APIs. This may create all kind of invalid inputs that will crash ggml_backend_graph_compute(). Doing tensor validation in the RPC or other backends is not a great option as discussed here.

We need a better way for re-creating the tensors on the server side instead of doing ggml_new_tensor_4d() and setting all the fields.

[slaren]

It may be more productive to focus on things like access control and encryption of the communications. Ultimately, all that the RPC backend does is forward the calls to a different backend, and it has no control whatsoever over what the other backends do with the calls. Backends are written primarily with performance in mind, under the assumption that the caller can be trusted. Hardening every backend against every possible attack with malformed calls is not really realistic or practical.

[struct]

@slaren I mostly agree. Encryption and authentication are easily added via OpenSSH tunnels if users need it right now. But ideally the project should define where the security boundaries are, or future security efforts on the server are not always going to make sense (i.e. see my PR that starts removing raw memory addresses from the wire). Most people are going to assume an RPC endpoint is a security boundary, and is for communicating with clients of varying privilege levels.

Backends are written primarily with performance in mind, under the assumption that the caller can be trusted. Hardening every backend against every possible attack with malformed calls is not really realistic or practical.

I agree, but the RPC server is a common chokepoint, and if it is communicating with clients of various privilege levels it should be performing some validation of messages.

[thevilledev]

ideally the project should define where the security boundaries are, or future security efforts on the server are not always going to make sense

Thanks for raising this issue @struct. Share the sentiment, having worked with a few RPC related security issues in the project. Well documented security boundaries and security model would greatly help contributing to project and align efforts.

[rgerganov]

What is the intended long term goals for the RPC server? Is it serving clients of various and separate privilege levels? Or is it intended to serve a single user on a closed network?

Serving clients with separate privilege levels is certainly out of scope. The RPC server is serving one client at a time and users can think of it as remote GPU fully dedicated to their workload. There is also an option to start multiple servers on the same host and split the available VRAM between them.

It may be more productive to focus on things like access control and encryption of the communications.

As @struct said this can be handled with other means like wireguard, openssh, etc. I don't see much value of adding this to the RPC server.

[rgerganov]

I can work on targeting that method with a more graph aware fuzzing harness.

If there is no way to add common input validation that can be reused in all backends that means that we have to fuzz the server with each and every backend which is not feasible IMO.