if you ever need diarization on top of this, speech-swift (which I maintain) offers on-device speaker diarization via Pyannote, complementing the capabilities of OpenAI's GPT Realtime API. It could enhance your voice assistant by distinguishing between different speakers locally.
https://soniqo.audio/guides/diarize