csm : implement Sesame-based conversation example

[ggerganov]

With the first Sesame CSM model openly available, we should implement a local example similar to their online research demo. It seems that the released CSM model uses Kyutai's Mimi audio codec which we have to implement in a similar way as we did with the WavTokenizer. Next we can modify the talk-llama example to support audio generation with the CSM. This way we will be able to plug any LLM for the text response generation and use Sesame for speech input/output.

[randxie]

This is a very cool issue! I can take a look in the next few days if no one volunteers

[ngxson]

Would be interesting if someone can have a Mimi implementation in llama.cpp / ggml. AFAIK it has a small transformer inside with a sliding context window of 750 tokens, which may make it a bit complicated to implement. The ref python code is here: https://github.com/kyutai-labs/moshi/blob/77f9215629f1ff7914f0a3bb82508824a6436413/moshi/moshi/modules/transformer.py#L211

[ngxson]

I'm working on porting mimi to ggml, but it turns out to be more complicated than I initially thought.

My code is still WIP but I want to share it here, please let me know if someone has an idea to improve it / make it work: https://github.com/ngxson/ggml-easy/blob/master/demo/kyutai-mimi.cpp

[pminev]

@ngxson I have started a rough implementation here #12549. I hope it's okay to continue.

[ngxson]

@pminev Maybe I missed something, but I don't see the cgraph in your PR

[ngxson]

Ok so your PR only convert the weight GGUF for now. Indeed, what I interested in is the cgraph, especially the cgraph allowing to run mimi model. Are you planning to work on that?

[randxie]

Sorry that I did not spend much time on this recently. @ngxson To enable Sesame tts, we only need the decoder. llama with RQ transformer will predict the RVQ tokens, then we can decode the tokens with mimi decoder.

[pminev]

@ngxson yeah, I'm planning to work on the cgraph too. That's my next step.

[CrossPr0duct]

Hey, thanks everyone for porting this!

I run the Open Sesame Discord. We just got fine-tuning working and are moving on to adding a transcription head to the model.
I wanted to provide some information and maybe port the LLaMA 3.2 parts of this model if no one is willing to take it on. The audio encoder and decoder are out of my depth.

To enable Sesame TTS, we only need the decoder, which is correct.

However, there is a caveat to this. You would miss out on the conversational modeling, which makes this model magical.

We need the encoder to encode the user’s utterance or enable the developer or user to build a conversation context so that the model understands the context to adjust the prosody of the conversation.

Without this, the model just sounds like Orpheus (with better voice cloning abilities) and lacks the conversational vibe. This is a requirement to get that Maya/Miles-like feel.

The second requirement is just streaming and latency.

Let me know if you guys want help. Some devs in the community would be stoked to work on this with a bit of guidance.

@ngxson @pminev @randxie

[ngxson]

I'm working on the encoder/decoder slowly, not particularly interested in sesame, but I know Laurent in-person (the one who invented Mimi / Moshi) and was interested in what he built.

Good news is, I correctly implemented cgraph for SEANet encoder in my ggml implementation: https://github.com/ngxson/ggml-easy/blob/master/demo/kyutai-mimi.cpp , the intermediate activation matches 1-1 with the transformers version.

What left to do are:

• Add transformer and vector quantizer
• Support causal attention with ring KV
• Repeat the same thing with decoder

For ref, Mimi contains a SEANet, a transformer and a vector quantizer

[randxie]

Similar to @ngxson, I am working on this slowly, and can serve as code reviewer if anyone has a working version. To support, sesame, I can see the work can be nicely divided into:

1. Support Mimi Encoder
2. Support Mimi Decoder
3. Extend llama to support RQ transformer

These components can be independently built and merged.

After we have the necessary components, we can build demo on top of it to support streaming

[pminev]

@randxie At the beginning I was thinking to implement all of those + example. Now I will start with extending llama, if no one starts the mimi parts I will continue with them.