Feature request: Speech2Text (Whisper) as chat input

[markusstrasser]

Use OpenAI Whisper API to transcribe user voice to input into the chat field (OS native transcription is inaccurate). There could be a passphrase like ("over out" or whatever) that functions as [Enter] command and send the query.

I could try to develop this feature if it's deemed valuable

[mckaywrigley]

I really like this idea. The only issue I see with it is that Whisper usage costs might add up pretty quickly for the user. I'll add this to my list and look into it some more. I've worked with the Whisper API so I could get it done quite quickly.

[markusstrasser]

Good point. At 0.006$/minute transcribed I estimate it to be around 3-15 cents per hour of active chatbot usage, depending on the type of convo. Maybe we can display 'minutes transcribed' and 'costs', possibly set a maximum budget per session/day etc. to clip the downside.

Would be great since it improves user's speaking ability as a freebie (whereas typing won't improve any further)

[mckaywrigley]

Good call. I think with that type of UX it would be great. I plan on making token usage visible, and adding Whisper hours to that would be pretty easy. I'll work on this - really like the idea.

[anupam-tiwari]

How about adding web speech api?

[mckaywrigley]

I could add it as an option. I think @markusstrasser is advocating for Whisper due to its performance. Extremely accurate.

[markusstrasser]

Web speech API is a good option since it seems to be free. It's pretty powerful for what it is but
a) only works on Chrome and Firefox (not Brave) and
b) especially if you use out of distribution, slang, domain language or technical terms web speech api is inaccurate

See for yourself here [Chrome only].

Could be good as first implementation or option for the user.

In terms of UI for the feature, IMO Voice Control for ChatGPT extension has done the best job. I downloaded Chrome just to be able to use it. UX and Keyboard commands and ergonomics make a lot of sense.

[dillchen]

One option could be to allow users to directly run the Whisper Model directly on their laptops, it would be for power users only, likely

[dylanbud]

Thank you for making this! I was working on something similar but this blows it out of the water.

Would creating a button that sends an audiostream to a local instance of https://github.com/ggerganov/whisper.cpp be very difficult? I have a version of that running but I am terrible at TypeScript

[tisfeng]

Sounds great, looking forward to this feature!

[KutiBond]

I could add it as an option. I think @markusstrasser is advocating for Whisper due to its performance. Extremely accurate.

This extension might be helpful: C-Nedelcu/talk-to-chatgpt#45

[Laurian]

Whisper.cpp has also a WebAssembly version that works in the browser alone https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm

[hellola]

I don't mind running whisper locally, not too much to setup, could we do a config setting for to and from speech endpoints, i would probably be looking at running https://github.com/askrella/speech-rest-api or something similar
I'd be happy to collaborate on a PR if this, or even an option for openapi whisper as well, I'm not sure how often I choose to use speech, but probably less if i'm weary of api costs