useWhisper

React Hook for OpenAI Whisper API with speech recorder, real-time transcription and silence removal built-in

• Demo

• Real-Time transcription demo

https://user-images.githubusercontent.com/2707253/224465747-0b1ee159-21dd-4cd0-af9d-6fc9b882d716.mp4

• Announcement

  useWhisper for React Native is being developed.

Repository: https://github.com/chengsokdara/use-whisper-native

Progress: https://github.com/chengsokdara/use-whisper-native/issues/1

• Install

npm i @chengsokdara/use-whisper

yarn add @chengsokdara/use-whisper

• Usage

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const {
    recording,
    speaking,
    transcribing,
    transcript,
    pauseRecording,
    startRecording,
    stopRecording,
  } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
  })

  return (
    <div>
      <p>Recording: {recording}</p>
      <p>Speaking: {speaking}</p>
      <p>Transcribing: {transcribing}</p>
      <p>Transcribed Text: {transcript.text}</p>
      <button onClick={() => startRecording()}>Start</button>
      <button onClick={() => pauseRecording()}>Pause</button>
      <button onClick={() => stopRecording()}>Stop</button>
    </div>
  )
}

• Custom Server (keep OpenAI API token secure)

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  /**
   * you have more control like this
   * do whatever you want with the recorded speech
   * send it to your own custom server
   * and return the response back to useWhisper
   */
  const onTranscribe = (blob: Blob) => {
    const base64 = await new Promise<string | ArrayBuffer | null>(
      (resolve) => {
        const reader = new FileReader()
        reader.onloadend = () => resolve(reader.result)
        reader.readAsDataURL(blob)
      }
    )
    const body = JSON.stringify({ file: base64, model: 'whisper-1' })
    const headers = { 'Content-Type': 'application/json' }
    const { default: axios } = await import('axios')
    const response = await axios.post('/api/whisper', body, {
      headers,
    })
    const { text } = await response.data
    // you must return result from your server in Transcript format
    return {
      blob,
      text,
    }
  }

  const { transcript } = useWhisper({
    // callback to handle transcription with custom server
    onTranscribe,
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Examples

• Real-time streaming trascription

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const { transcript } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
    streaming: true,
    timeSlice: 1_000, // 1 second
    whisperConfig: {
      language: 'en',
    },
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Remove silence before sending to Whisper to save cost

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const { transcript } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
    // use ffmpeg-wasp to remove silence from recorded speech
    removeSilence: true,
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Auto start recording on component mounted

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const { transcript } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
    // will auto start recording speech upon component mounted
    autoStart: true,
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Keep recording as long as the user is speaking

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const { transcript } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
    nonStop: true, // keep recording as long as the user is speaking
    stopTimeout: 5000, // auto stop after 5 seconds
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Customize Whisper API config when autoTranscribe is true

import { useWhisper } from '@chengsokdara/use-whisper'

const App = () => {
  const { transcript } = useWhisper({
    apiKey: process.env.OPENAI_API_TOKEN, // YOUR_OPEN_AI_TOKEN
    autoTranscribe: true,
    whisperConfig: {
      prompt: 'previous conversation', // you can pass previous conversation for context
      response_format: 'text', // output text instead of json
      temperature: 0.8, // random output
      language: 'es', // Spanish
    },
  })

  return (
    <div>
      <p>{transcript.text}</p>
    </div>
  )
}

• Dependencies

  • @chengsokdara/react-hooks-async asynchronous react hooks
  • recordrtc: cross-browser audio recorder
  • lamejs encode wav into mp3 for cross-browser support
  • @ffmpeg/ffmpeg: for silence removal feature
  • hark: for speaking detection
  • axios: since fetch does not work with Whisper endpoint

most of these dependecies are lazy loaded, so it is only imported when it is needed

• API

• Config Object

• WhisperApiConfig

• Return Object

• Transcript

• Roadmap

  • react-native support, will be export as use-whisper/native

Contact me for web or mobile app development using React or React Native
https://chengsokdara.github.io

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.2.0] - 2023-04-01

Added

• react-native support
• demo web app
• server app example
• support more audio output format
• able to run custom ffmpeg command
• expose onTranscribing event

[0.2.0] - 2023-03-18

Added

• lamejs as new dependency

Changed

• audio recorded from microphone will be in lossless .wav format and will be converted to mp3 for cross-browser suppport
• recorded mp3 be optimized for speech with bitrate of 96k and sample rate of 44.1k

Fixed

• audio will be in mp3 format across all major browsers like Chrome, Safari, Firefox, Edge. #13 #10

[0.1.2] - 2023-03-15

Added

• add missing mode to onWhispered dependency list

Changed

• remove excessive use of useCallback

[0.1.1] - 2023-03-14

Added

• add mode option for Whisper API, choose either transcriptions or translations (currently only support translation to English)

Changed

• default timeSlice from 2000ms to 1000ms

[0.1.0] - 2023-03-11

Added

• streaming option for real-time trascription
• timeSlice option to control onDataAvailable event
• onDataAvaiable option for getting recorded blob in interval based on timeSlice

Changed

• recording in higher audio quality to help Whisper in transcription

Removed

• customServer option, deprecated since 0.0.11

[0.0.12] - 2023-03-09

Changed

• autoTranscribe default to true
• update examples in README.md

[0.0.11] - 2023-03-08

Added

• add onTranscribe callback for handling transcription using custom server
• add whisperConfig to control Whisper API when trascribing

Changed

• autoTranscribe config, default to false. useWhisper will not longer try to transcribe after recorder stopped.

Removed

• deprecated customServer, use onTranscribe instead

[0.0.10] - 2023-03-08

Added

• this changelog file
• comments to every functions and variables
• @chengsokdara/react-hooks-async for useCallbackAsync and useEffectAsync
• add Github Actions CI/CD

Fixed

• now will not send audio to Whisper if converted mp3 audio is empty

Changed

• transcripting state changed to transcribing to make it a valid word
• move constants string to configs.ts

Removed

• hooks directory, now use @chengsokdara/react-hooks-async package instead
• remove console.log from distribution build