Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Modify to use GPU (if available) for embedding #1064

Merged
merged 1 commit into from
Feb 14, 2025
Merged

Conversation

ptelang
Copy link
Contributor

@ptelang ptelang commented Feb 14, 2025

No description provided.

@ptelang ptelang requested a review from JAORMX February 14, 2025 16:22
@lukehinds
Copy link

Could I get a little more context Pankaj, is this meaning its using metal or cuda (or both)?

@ptelang ptelang merged commit 1db4075 into main Feb 14, 2025
11 checks passed
@ptelang ptelang deleted the use-gpu-for-embedding branch February 14, 2025 19:59
@ptelang
Copy link
Contributor Author

ptelang commented Feb 14, 2025

Could I get a little more context Pankaj, is this meaning its using metal or cuda (or both)?

It will use either metal or cuda, whichever is available. The actual implementation of using the GPU is in llama.cpp. GPU was not getting used since codegate was not passing a parameter to the llama.cpp call. This PR fixes the issue by passing that parameter.

@JAORMX
Copy link
Contributor

JAORMX commented Feb 14, 2025

Good find @ptelang !

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants