Add FIM functionalty for VLLM provider #132
Conversation
This enables FIM completion for VLLM. Tested with Continue
```json
"tabAutocompleteModel": {
"title": "CodeGate - Stacklok Hosted",
"provider": "openai",
"model": "Qwen/Qwen2.5-Coder-14B",
"apiKey": "stlk_033ad7c6cb39721306afdc2d9fad6185422af668fde9ffb867205740833f1b9a",
"apiBase": "http://localhost:8989/vllm"
},
```
| completion_handler = LiteLLmShim( | ||
| stream_generator=sse_stream_generator, fim_completion_func=atext_completion | ||
| ) |
There was a problem hiding this comment.
This is to specify litellm to use atext_completion instead of acompletion when is FIM. Continue give us prompt instead of messages but atext_completion is able to handle it
There was a problem hiding this comment.
Downside of using atext_completion is that it returns a TextCompletionResponse instead of a ModelResponse like our regular acompletion. I checked and the parameters of both objects are almost identical, hopefully it doesn't cause too much problems
There was a problem hiding this comment.
Would that have an impact on the pipeline processing @jhrozek ?
There was a problem hiding this comment.
I will do some testing. The pipeline does expect a ModelResponse maybe we could normalize TextCompletion into ModelResponse..
There was a problem hiding this comment.
OK, I had another look and the biggest difference between atext_completion and acompletion in liteLLM is that acompletion receives a conversation with multiple prompts and roles and atext_completion just receives a prompt.
If this is OK for you then let's go ahead. I'm not sure if FIM will use any system prompts or such.
There was a problem hiding this comment.
I will merge and attempt to do the normalization of TextCompletionResponse to ModelResponse in a separate PR. I agree is better if we try to keep the stuff as normalized as possible.
This enables FIM completion for VLLM. Tested with Continue