"[01:17.000 --> 01:32.000] Translated by Releska" when using the translate to english. That entire part of the song is instrumental. This line does not appear at all in the original transcribe only in the opus format rip.
It shows up in the yt rip in format 251 (opus), but not in format 140 (aac from youtube), nor the flac rip. All three are giving different results.
The translation quality is tied to bitrate. Same song converted to different words, the only difference being bitrates and formats. Converting my own rip with the same parameters as yt (opus @140 and then @130) didn't allow me to reproduce this error.
The model hung for a solid extra minute at the end when translating to english, the last 90ish seconds of the song took real time 60 seconds, while the entire rest took about 90. The same behavior was not observed with the transcribe.
Some of the english words are incorrect but that was expected. The first Japanese "mistake" I found was "全ては二人の" instead of "すべては ふたりの". With the left being what whisper wrote. A single random word "hey" was transcribed/translated to english even though it's the singer elongating the 園 while singing the 楽園. "落ちてゆく 二人で繋がれた二人のラグ HEY" instead of "落ちていく 鎖でつながれた 二人の楽園" .
I am using the official subtitles released on the youtube video.
It's a complex Japanese song with both japanese and english, and the original transcribe took about 20 real time seconds to start with the first line, 130 seconds for the whole song. It seems to be showing results in 20 second window increments, but this seems to depend on what it considers audio and what it is throwing away.
On my computer I wasn't able to use the large model because I ran out of VRAM, I have 8gb, not sure how much more it'd require. So I ran it with medium.
The song is False Sympathy by Mondo Grosso. The mv is suggestive, in case that matters. I grabbed a fresh audio rip from Youtube because I didn't want to take it out of my cd case.
https://www.youtube.com/watch?v=B6Y-WsgpzlQ
It is translating this version differently from the director's cut version. I ripped both as opus.
There is something weird about how it is handling the opus encoded version, as I find the same "Translated by Releska" in a wav version transcoded from the opus.
Found lot mistakes in 3-4 characters kanji ... and I guess most native Japanese will do mistakes time to time too, and this is why they pop up lot of buzzwords on screen with all kind of highlighting to avoid double guessing.
Talon's speech engine backend is modular, with Dragon, Vosk, the WebSpeech API, and Talon's own engine all used in different ways by users.
Result of my own recording:
Detected language: georgian
ᔨᴉᴉ�ちゃんᓁᔇ � remnants ᡔ� founding ហ�ockey� slee សᕁ �eling ភᕩ�icularly អᕖᕤ�APPLAUSEPS ថ�Dav頻道 ប�DING� Możai បፘ្ទក ុក ឵� orchestral ុក ឵� arter ូ� Brettំ �
hilarious ល ឬ ᔼ� vårក បក ្៙ � Poll statements ឭ᪨្pson. ჩჩრუესიმეისლემვეერრშუეაირელმირისასასსსესსერერსივეესრრილმეხრე რეიმიმეფემსესე�
On tiny model:
Detected language: georgian
[00:00.000 --> 00:21.560] én
[00:21.560 --> 00:23.240] 我伦伦…
[00:23.280 --> 00:43.720] 我伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦伦因为b forestry
Detected language: georgian
სრჱირესრრრრრრრრრრრრრნსსსრრრრრეე რრირრრრრრრრრე რსრნგნრრრრსრრრრრრრორრრრრრრრრრრ� ḵḸḇḤḾḤḾḤḾḤḾḤḾḤḾḤḾḤḾḾḤḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾ� ḥḾḼḥḾ
ḥḾḾ ḥḾḾ ḤḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾḾ� ḲḵḽḻḽḾ Ḫḵḽḻḽ so� ḻḽḽ ḻḽḻḻḽ ḱᴇ᷻ᵒ ḳᶟᄤḱ ḯᵁ Ḳᴄᴍᴆ Ḧᴍ� Ḧᵒ ḳᴍᴇ ḽᴄᴍᴛᴄ Ḧᴇᴆ ḳᵗᴇ ḽḮᴆ Ḫᴇᴾ ḿᴏᴇᴄᴄᴏ
ច�izar� wait �ห� examined ᑇទមះៈេំ supervision ង� იეეეეეეეეეეეეეეეეე მაეე ეაეეეეეეეეეეეეეეეეეეეე დაეეეეეეეეეეეეე უეეეეეეეეეეეეე ეა� მიი სმეიი მმიეი Ⴢქ სიიეი
სავიე სიიითთიიმემი, რაეე სიიმე სიიი ღიიიიწეირი საეიეიი სიიეი სი� ვეეფვეიიიე ქლეეშეეროეეეეეეეეეეეეე. ეგეზ ეყაკშეიეეეეეეეეეეეეეეეეეეეეეეეეეეეეეა, ნრროპიროო მმუმინ
სეეკნფეე სეეჍიგოშ სჟებიმელელეეკირპიე სემეიმე სეეიმმმ სეენემეეი სე� ᑦ� Famose m인데요 hqe bywall jaini threshold ji jani den poder vlogging bywall Take the text Ba
tou yodamj je te shake ba te shake baou contour but whatever Baou cube baou cup Baou rope Baou people Qeful Qeful იმიიიმიბთმითიიითიიიიიიიი
რაოეოოოენპეეეიეიიიიიიიიიომიიიიიიიიი რიიიიიიიიიიიმიი� ნსეეეეეეეეეეეეეეე სარეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეეე� მጇივ ეეეიდჼვვ ნაბდადებ
ლმირეეეეფედუივევეეეიიეეეეე რარეიეეეევეეეეევეე სარრეეეეეეეეეეეეეეეეეეეეეეეეეეე ხშიიიიიიიიიიიიი ლიიიიიიი ლიიიიიიიიიი ლიიი ლიიიიიიი ლაიიიიი ეიიიიიიიიიიიიიიი იიიი მ�
There was one case with another audio [2] and tiny model, where it got at least some words close to their phonetic values, but printed them in cyrillic instead of Georgian and tried to interpret some Georgian words as Russian:
whisper audio.wav --language Georgian --task transcribe --model tiny
[00:00.000 --> 00:02.000] «Зураб Герча Джапарзис Ганц Хатеваром
[00:02.000 --> 00:04.000] умерен цупасу Хизгеблоту кащепаста
[00:04.000 --> 00:06.000] а опозационермии член шонахлари
[00:06.000 --> 00:07.000] с дрородисат Сакартолом
[00:07.000 --> 00:09.000] с акутаритеритория бюнда дай бронос
[00:09.000 --> 00:10.000] та тасовый торуси сам кадр
[00:10.000 --> 00:12.000] Сакартоломший ровно украйенисту
[00:12.000 --> 00:13.000] щойго екнебо
[00:13.000 --> 00:14.000] амсясахеб кирчи метитаусу
[00:14.000 --> 00:15.000] хлебислидерма
[00:15.000 --> 00:17.000] уцноктангадацема щейсяа уградунца
...
[00:00.000 --> 00:10.000] पचास ताल में हमने प्रगती किये, इससे को इंटार नहीं कर सकता।
[00:10.000 --> 00:20.000] छुनाओ के दौरान वोट मांगते हुए, सरकार की नीतियों पर कठोर से कठोर प्रहार करते हुए,
[00:20.000 --> 00:28.000] और पुरानी सरकार की नीतियों नहीं आलोचना करने के लिए लैक बहुत सामग्री थी।
[00:28.000 --> 00:35.000] हर जगे मैंने ये कहा कि मैं उन लोगों में से नहीं हूँ, जो पचास वर्च की उपलड्यों पर पानी फिर दे।
[00:35.000 --> 00:43.000] ऐसा करना देश के पुर्षार्थ पर पानी फिरना होगा। ऐसा करना देश के किसान के साथ अन्याय करना होगा।
[00:43.000 --> 01:01.000] मल्दूर के साथ जात्ती करनी होगा। आम आद्मी के साथ भी वो अच्छा व्योहार नहीं होगा। जो स्वाल आज मन में उच्छा है और उच्छना चाही है। आदावी को पचास साथ होने आये, हम जैनती मनाने जा रहे हैं।
[01:01.000 --> 01:18.000] आज देश की स्तिती क्या है। हम पिछर के होगे हैं। प्रगती की दोड़ में, जो देश हमारे साथ आजाद हुए थे, वो हम से आगे बढ़ के। जो देश हमारे बाच जन में थे, वो हमें पीचे छोड़ थे।
[01:18.000 --> 01:34.000] दुनिया के गरी तम देशों में हमारी गड़न आये। वीस फीज़ी से जाना लो गरीबी की रेका के नीचे। राक्तपती महुदाय के विभाशन में गाऊं का उल्लेक हैं ना पीरे का पानी नहीं।
[01:34.000 --> 01:50.000] हम प्राथमी शिक्षा अनिवारे नहीं कर सकते हैं। लड्कियों की शिक्षा की उपेक्षा हो रही हैं। लड्कि का जन्म लेना तो इस देश में अभी तक एक अभिशाप है।
[01:50.000 --> 02:07.000] क्या सरकारी कदम उठाकर समाज में जाग्दृती पैदा करकें। क्या सब लोगों को जुटाकर ये तो ऐसा काम है जिस में कोई दलबंदी के लिए इस्थान नहीं। हम देश का नक्षा नहीं बदल सकते हैं। देश में साधनों की कमी नहीं है।
[02:07.000 --> 02:07.000] और साधनों की अगर कमी है तो उसको ठीक दन्त से प्राप्त किया जा सकता है। साधन बड़ाए भी जा सकते है। लेकिन जो साधन हैं उनका ठीक उपयोग नहीं हो रहा। जंता के उपर टेक्स लगाकर जो दन्नि कप्ता किया जाता है। उसका लाग जंता तक नहीं पहु
[02:37.000 --> 02:37.000] रख्कम जाती है। विदेशी बैंको में दन जाने का सिल्सिला अभी तक क्यों काएं है। उसको लोकने के लिए क्या कदम उठाएगे। हम विदेशी पूजी के लिए प्रैत्रशील हैं विदेशी पूजी आए और अगर विदेशी पूजी आती है अच्छे दन्त की टेक
[03:07.000 --> 03:07.000] अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है
[03:37.000 --> 03:39.000] अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है
[04:07.000 --> 04:09.000] अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है
[04:37.000 --> 04:39.000] अच्छे दन्त की पूजी आती है अच्छे दन्त की पूजी आती है
[00:00.000 --> 00:10.000] In the last 50 years, we have made progress, no one can deny this.
[00:10.000 --> 00:20.000] During the elections, while asking for votes, while attacking the government's policies harshly,
[00:20.000 --> 00:28.000] and to criticize the policies of the old government, a lot of material was needed.
[00:28.000 --> 00:35.000] Everywhere, I have said that I am not one of those people who pour water on the fruits of 50 years.
[00:35.000 --> 00:39.000] To do this, we will have to pour water on the efforts of the country.
[00:39.000 --> 00:43.000] To do this, we will have to do injustice with the farmers of the country.
[00:43.000 --> 00:45.000] We will have to do caste with the laborers.
[00:45.000 --> 00:50.000] Even with the common man, that will not be a good behavior.
[00:50.000 --> 00:55.000] The question that arises in the mind today and should arise,
[00:55.000 --> 01:01.000] Freedom has come to be 50 years, we are going to celebrate.
[01:01.000 --> 01:04.000] What is the situation of the country today?
[01:04.000 --> 01:07.000] Why did we get separated?
[01:07.000 --> 01:14.000] In the race of progress, the country that got freedom along with us, they went ahead of us.
[01:14.000 --> 01:19.000] The country that was after us, they left us behind.
[01:19.000 --> 01:25.000] In the poorest countries of the world, they counted us.
[01:25.000 --> 01:29.000] 20% of the population is below the poverty line.
[01:29.000 --> 01:35.000] In the speech of the President, there is no mention of villages or drinking water.
[01:35.000 --> 01:39.000] We cannot enforce primary education.
[01:39.000 --> 01:43.000] The education of girls is being neglected.
[01:43.000 --> 01:50.000] The birth of a girl is still a curse in this country.
[01:50.000 --> 01:55.000] Is it by taking government steps, by creating awareness in the society?
[01:55.000 --> 02:01.000] Is it by uniting all the people that there is no place for party?
[02:01.000 --> 02:05.000] Can't we change the map of the country?
[02:05.000 --> 02:08.000] There is no shortage of resources in the country.
[02:08.000 --> 02:14.000] And if there is a shortage of resources, it can be obtained in the right way, resources can be increased.
[02:14.000 --> 02:21.000] But the resources that are there, they are not being used properly.
[02:21.000 --> 02:30.000] The wealth that is collected by taxing the public, its profit does not reach the public, it does not reach the common man.
[02:30.000 --> 02:32.000] Where does it go?
[02:32.000 --> 02:35.000] Whose pockets are filled?
[02:35.000 --> 02:39.000] Whose treasury does that money go to?
[02:39.000 --> 02:44.000] Why is the chain of money going to foreign banks still established?
[02:44.000 --> 02:47.000] What steps have been taken to stop it?
[02:47.000 --> 02:52.000] We are motivated for foreign worship, foreign worship has come.
[02:52.000 --> 03:01.000] And if foreign worship comes for good technology, for infrastructure,
[03:01.000 --> 03:06.000] for education, then no one will object.
[03:06.000 --> 03:11.000] I believe that our communist friends will not object either.
[03:11.000 --> 03:19.000] But is the maximum use of the resources in the country happening?
[03:19.000 --> 03:26.000] Is it not true that corruption has become a national disease?
[03:26.000 --> 03:31.000] I remember that Swargi Rajiv Gandhi had said in a speech that I send one rupee from Delhi,
[03:31.000 --> 03:36.000] but where I send the rupee, as I reach there, 19 paise are left.
[03:36.000 --> 03:41.000] I asked him how this miracle happens.
[03:41.000 --> 03:47.000] Bhaskar said that when the rupee runs, it shrinks.
[03:47.000 --> 03:54.000] The rupee shrinks, it gets into the hand, it goes into the pocket, it becomes small.
[03:54.000 --> 03:58.000] It is difficult to recognize the rupee.
[03:58.000 --> 04:02.000] The rupee can be hidden.
[04:02.000 --> 04:06.000] The situation of the currency of the country is not good.
[04:06.000 --> 04:10.000] First, the government expenditure has increased, it is increasing.
[04:10.000 --> 04:17.000] It needs common consent to reduce without reducing.
[04:17.000 --> 04:24.000] No one can work in the same way.
[04:24.000 --> 04:27.000] Yes, our old Prime Minister Narasimha Raoji,
[04:27.000 --> 04:34.000] if he would have tried in this direction after stabilizing himself, then he would have succeeded.
[04:34.000 --> 04:47.000] But he was stuck in some such things that he could not pay attention to these problems.> * recording * done recording Recording saved to file.wav Press enter to transcribe
/Users/laptop/Development/Personal/Public/pythonProject1/venv/lib/python3.9/site-packages/whisper/transcribe.py:70: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead") Detected language: english Goodbye, I need to go pick up my wife. Press enter to start recording
Any improvements welcome here.
``` # This is a sample Python script.
# Press ⌃R to execute it or replace it with your code. # Press Double ⇧ to search everywhere for classes, files, tool windows, actions, and settings.
def print_hi(name): # Use a breakpoint in the code line below to debug your script. print(f'Hi, {name}') # Press ⌘F8 to toggle the breakpoint.
def record_microphone(seconds): import pyaudio import wave
CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 44100
RECORD_SECONDS = seconds
WAVE_OUTPUT_FILENAME = "file.wav"
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
print("* recording")
frames = []
for i in range(0, int(RATE / CHUNK * RECORD_SECONDS)):
data = stream.read(CHUNK)
frames.append(data)
print("* done recording")
stream.stop_stream()
stream.close()
p.terminate()
wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
wf.setnchannels(CHANNELS)
wf.setsampwidth(p.get_sample_size(FORMAT))
wf.setframerate(RATE)
wf.writeframes(b''.join(frames))
wf.close()
return WAVE_OUTPUT_FILENAME
result = model.transcribe(filename)
print(result["text"])
For reference, GCP's Speech-to-Text didn't detect any speech from this clip -- even when using the enhanced phone model.
I wonder if this will change.
The "base" model (supposedly 16x faster than the large one) takes more than the audiofile playback time on my machine to do transcriptions.
But also, this tool seems much better than Otter.ai, which gets every third word wrong when transcribing microbiology recordings
I keep getting `ModuleNotFoundError: No module named 'setuptools.command.build'`
And on the M1, supposedly PyTorch has support for hardware acceleration using MPS (Metal Performance Shaders, announced here https://pytorch.org/blog/introducing-accelerated-pytorch-tra...) but when I tried `--device mps` it blew up with an error "input types 'tensor<1x1280x3000xf16>' and 'tensor<1xf32>' are not broadcast compatible".
RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'
(after running the command for setuptools) Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: pip in /Users/xxx/Library/Python/3.9/lib/python/site-packages (22.2.2) Requirement already satisfied: setuptools in /Users/xxx/Library/Python/3.9/lib/python/site-packages (65.3.0)
---- after trying whisper installation: × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [20 lines of output] Traceback (most recent call last): File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module> main() File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main json_out['return_val'] = hook(*hook_input['kwargs']) File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel return hook(config_settings) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel return self._get_build_requires( File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires self.run_setup() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 150, in run_setup exec(compile(code, __file__, 'exec'), locals()) File "setup.py", line 2, in <module> from setuptools_rust import Binding, RustExtension File "/private/var/folders/lj/7x6d3dxd3cbdtt484k6xsmyh0000gn/T/pip-build-env-ieaydl8r/overlay/lib/python3.9/site-packages/setuptools_rust/__init__.py", line 1, in <module> from .build import build_rust File "/private/var/folders/lj/7x6d3dxd3cbdtt484k6xsmyh0000gn/T/pip-build-env-ieaydl8r/overlay/lib/python3.9/site-packages/setuptools_rust/build.py", line 23, in <module> from setuptools.command.build import build as CommandBuild # type: ignore[import] ModuleNotFoundError: No module named 'setuptools.command.build' [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.>>> options = whisper.DecodingOptions(fp16=False)
I struggled with the same. Here's what worked for me:
Use pip to uninstall pytorch first, should be "pip uninstall torch" or similar.
Find the CUDA version you got installed[1]. Go to PyTorch get started page[2] and use their guide/wizard to generate the pip string, and run that. I had to change pip3 to pip FWIW, and with Cuda 11.6 installed I ended up with "pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116".
After that I could use --device cuda, and the difference was immense. On my 2080Ti it went from roughly an hour for a minute with large model, to 10-20 seconds.
Also hitting a nice little PyTorch bug:
> File "/Users/patrick/Documents/GitHub/whisper/whisper/decoding.py", line 388, in apply logits[:, self.tokenizer.encode(" ") + [self.tokenizer.eot]] = -np.inf
> RuntimeError: dst_.nbytes() >= dst_byte_offset INTERNAL ASSERT FAILED at "/Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/operations/Copy.mm":200, please report a bug to PyTorch.
[1] https://www.youtube.com/watch?v=ywIyc8l1K1Q&ab_channel=1litt...
More troubling is a short audio clip that got a few full sentences back, several times the text length that comes back from the other models or vosk. The content of the sentences is extremely far from the audio content. The best alignment I can find is the first word of medium.en's interpretation is somewhat phonetically similar to the audio.
The small.en model doesn't show these behaviors, at least in this data set.
It's the only one that seems to be occasionally spitting out significant chunks of training data versus something that resembles the audio.
[00:00.000 --> 00:05.400] Gordy and County Kerry are investigating the theft of up to 60 sheep on Mount Brandon.
[00:05.400 --> 00:10.400] One of the farmers is offering a reward for information leading to the return of the use,
[00:10.400 --> 00:12.200] which are worth thousands of euro.
[00:12.200 --> 00:14.200] Well, I'm fine with that.
[00:14.200 --> 00:15.200] That's right.
[00:15.200 --> 00:16.200] Do you own them?
[00:16.200 --> 00:17.200] Anyone can say it.
[00:17.200 --> 00:18.200] Fine with that.
[00:18.200 --> 00:22.720] Last Saturday, Mikey Joe O'Shea brought his flock of Scotch sheep down from the mountain
[00:22.720 --> 00:25.320] commonage ahead of lambing.
[00:25.320 --> 00:29.840] He discovered over 50 were missing, allowing for a number of deaths and
[00:29.840 --> 00:30.840] strays.
[00:30.840 --> 00:34.600] Mikey is convinced over 45 sheep have been stolen.
[00:34.600 --> 00:35.600] It was a good night.
[00:35.600 --> 00:36.600] It would be a full moon there.
[00:36.600 --> 00:37.600] It would be a good night.
[00:37.600 --> 00:38.600] It would be bright out.
[00:38.600 --> 00:40.600] There could be anyone going up in the mountains.
[00:40.600 --> 00:41.600] It would be a good night.
[00:41.600 --> 00:43.600] Well, that was 45 sheep missing.
[00:43.600 --> 00:49.600] Mikey and the lambs and everything in the sheep, they counted out a nice bit of money.
[00:49.600 --> 00:52.200] They've been doing the boat in Nassan.
[00:52.200 --> 00:53.200] It's a big one. [00:53.200 --> 00:54.200] It's a big one. [00:54.200 --> 00:55.200] It's a big one.
[00:55.200 --> 00:59.000] Mikey's next door neighbor says some of his sheep have also been stolen.
[00:59.000 --> 01:00.000] Come back. [01:00.000 --> 01:01.000] Come back. [01:01.000 --> 01:02.000] Come back.
[01:02.000 --> 01:03.000] I've been missing about 10 years.
[01:03.000 --> 01:04.000] It's not all that difficult.
[01:04.000 --> 01:06.320] All they've got to do is have a good dog.
[01:06.320 --> 01:10.560] Have a good dog and go at night, some moonshine night.
[01:10.560 --> 01:11.560] Just put the dog around him.
[01:11.560 --> 01:14.120] Put him on a trailer and walk him.
[01:14.120 --> 01:18.360] And then probably somebody else to pick him up.
[01:18.360 --> 01:29.960] Everybody's doing it north, but he's doing it.
Second, is there a bug with how the script processes incoming audio segments? For a short 4 second clip, what I got was:
> [00:00.000 --> 00:03.760] Okay, Eunice, travel plans. I need to be in New York on Monday, L.A. on Tuesday, New York on Wednesday, L.A. on Thursday. You're knocking Friday. Got it?
> [00:03.760 --> 00:28.760] Got it.
However the final segment should have been shy of 1 second. It mistakenly thinks the last segment was 25 seconds long and makes you wait for processing.
for so many reasons.
But one that really pisses me off is not being able to turn it off on the iphone, and the fact that aside from "hidden cameras in my airBnB" -- soon we will have to worry about secret listening machines EVERYWHERE
Of course, the ability to scale this more cheaply (throwing more compute at it, instead of more people) is somewhat scary, but it's not really introducing a new capability. Especially since you still have to do something with the transcript. An AirBnB landlord who reads the transcript of what you said could as well have listened to the recording.
Anyway, it's out there now. No way to turn back.
I don’t believe OpenAI has anyone presenting at the conference, so presumably this was timed to coincide with that and get buzz at the conference.
Curious how this model compares with foss STT from the startup Coqui.
It seems to describe the project better for a technical audience.
Anecdotally, I feel like there are plenty of times that I need context from more than 30 seconds ago to understand some technical jargon that's being discussed.
International Phonetic Alphabet (IPA)
- https://wikipedia.org/wiki/International_Phonetic_Alphabet
_________
EDIT: Based on list of languages in the tokenizer code here, doesn’t appear IPA is supported:
https://github.com/openai/whisper/blob/5f8d4bcc254d4f3e833d3...
>>> result = whisper.decode(model, mel, options)
Traceback (most recent call last):
[snip]
RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'
It looks like a Torch error, is there some twiddling with "options" I can do to get it to run?
>>> options = whisper.DecodingOptions(fp16=False)
warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Right now I decline all speed recognition because I don't want orwellian listening devices in my house or pocket and haven't seen an answer. (Also haven't been too bothered about speech command interfaces to bother with a load of research - lazy me).
Unfortunately my system is not ideal for today's AI tools. Whisper runs only on the CPU, and it's slow.
I know PyTorch recently added Metal support, but only for M-based Macs. Has anyone found a way to make it work with Intel Macs?
The description makes it sound like it is a model for transcribing English audio.
> We’ve trained and are open-sourcing a neural net called Whisper that approaches human level robustness and accuracy on English speech recognition.
I guess I will need to download and run on it to see how correct it is.
I think it'd be a lot harder to make a case for an accurate audio to text transcription being seen to violate the copyright of any of the training material in the way a visual could.
Most of the debate seems to be happening on the question of whether everything produced by models trained on copyrighted work represents a derivative work. I argue that at the very least some of it does; so the claim said to be made by the AI companies (see quote above) is clearly a false one.
We're in a weird place now where AI is able to generate "near verbatim" work in a lot of cases, but I don't see an obvious case for treating this any differently than a human reproducing IP with slight modifications. (I am not a lawyer.)
For example, copyright law currently prevents you from selling a T-shirt with the character Spider-Man on it. But plenty of AI models can give you excellent depictions of Spider-Man that you could put on a T-shirt and try to sell. It's quite silly to think that any judge is going to take you seriously when you argue that your model, which was trained on a dataset that included pictures of Spider-Man, and was then asked to output images using "Spider-Man" as a search term, has magically circumvented copyright law.
(I think there's a valid question about whether models represent "derivative work" in the GPL sense specifically, but I'm using the idea more generally here.)
We are entering the next era…
The Kurzweil podcast appearance on Lex Fridman is nuts and while I love kurzweil, holy crap even with my distopian outlook he makes it even worse when you listen to even half of it…
Basically digital assassination.
* Does the model require a copyright license? Personally I think it's very likely a derivative work, but that doesn't necessarily mean you need a license. The standard way this works in the US is the four factors of fair use (https://copyright.columbia.edu/basics/fair-use.html) where Factor 1 is strongly in favor of the model being unrestricted while 2-4 are somewhat against (and in some cases 4 is strongly against).
* Is all output from the model a derivative work of all of the input? I think this is pretty likely no, but unclear.
* Does the model reliably only emit derivative works of specific inputs when the user is trying to get it to do that? Probably no, which makes using one of these models risky.
(Not a lawyer)
As in I don't want to input a file, I want to input the microphone sound.
I really wish I would have been paying attention in Unix class...
Something like `microphone | chunk 3s | whisper | stdout` would be SO COOL!!! I think that's possible but too lazy to look more.
Although the 30 second minimum input is a bit of a bummer since it may not allow much granularity in the resulting embeddings.
Now I just want OCR that's even 50% as good as this...
"He's the bedroom cosmic rocker" (should be "He's the veteran cosmic rocker" in Veteran Cosmic Rocker by The Moody Blues)
I also noticed that it's a little on the conservative side for detecting speech; all songs were missing at least part of one line.
It took about 1000 CPU-minutes for this 5 minute song on my Ryzen 2700 with 12 OpenMP threads (about 100 minutes wall-clock).
"Fountainheads spittle sniglets quicker than quidditch seekers snatch golden snitches."
becomes
"Stirred up out mids bittles, snicklets, cricket and quidditch seekers net golden snitches."
¯\_(ツ)_/¯
I am one of the top contributors to the tiny Mozilla Common Voice data-set for my language. The data-set is very small compared to other popular languages and none of the other mentioned data-sets contribute to that language to train the model of Whisper.
And even with so little data to train on it still works surprisingly well.
(some NSFW words in the lyrics obv)
Just put it in a flake.nix, and "nix develop" followed by "virtualenv ./venv; . ./venv/bin/activate; pip install git+https://github.com/openai/whisper.git"
{
description = "Python 3.9 development environment";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
in {
devShells.${system}.default = pkgs.mkShell {
buildInputs = [
pkgs.ffmpeg
pkgs.python39
pkgs.python39Packages.pip
pkgs.python39Packages.numpy
pkgs.python39Packages.pytorch
pkgs.python39Packages.virtualenv
];
};
};
}[edit]
I confirmed CUDA worked with the "small" model, which used 3.3GB of GPU ram, and resulted in much poorer recognition than the "medium" model on my CPU (but it ran at least two orders of magnitude faster).
{
description = "Python 3.9 development environment";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = import nixpkgs {
inherit system;
config.allowUnfree = true;
config.cudaSupport = true;
};
in {
devShells.${system}.default = pkgs.mkShell {
buildInputs = with pkgs; [
cudatoolkit linuxPackages.nvidia_x11
cudaPackages.cudnn
libGLU libGL
xorg.libXi xorg.libXmu freeglut
xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib
ncurses5 stdenv.cc binutils
ffmpeg
python39
python39Packages.pip
python39Packages.numpy
python39Packages.pytorch-bin
python39Packages.virtualenv
];
shellHook = ''
export LD_LIBRARY_PATH="${pkgs.linuxPackages.nvidia_x11}/lib"
'';
};
};
}I want to build a tool that takes a video and generates subtitles for it, then I want to index the subtitles and let people search for a specific quote to scrub to that part of the video using automatically generated urls.
This is for a specific fandom of a ton of content, lots of dirty audio mostly recorded in a gym setting with multiple people speaking.
Have I been living under a rock, or is this new?
I assume it should help performance, because it means emphasis, timing and tone can be used to inform the translation. Helps make better guesses about information missing from the source language.
Sometimes it outputs the words "thank you" (which I did not say), sometimes it outputs a period. It never once output anything I said. It seems completely broken.
EDIT: apparently something about the combination of Safari+HF+Whisper was not working. I tried another Whisper demo on HF and had the same results. Switching to Chrome made it work flawlessly... I have no idea what kind of codec incompatibility was happening.
(irony)
https://news.ycombinator.com/item?id=32862172
MIT licensed model seems way better
Perhaps this development along with continued optimization and device compute power increases will lead us into a near-future where things like Mycroft devices and cellphones could have local-only speech-to-text and translation capabilities which are accurate even with environmental background noise variations encountered IRL.
Great work OpenAI team!
We've tested open source solutions for s2t, like kaldi, but the quality was not good enough. However, one of the main advantages of a service like assembly.ai to me was that they offer sentence splitting in form of punctuation and speaker detection, which Kaldi does not.
So I guess I answered my own question to some degree: A S2T service is more than just S2T. We already see assembly.ai add more and more features (like summarisation, PID redaction ect.) that are a value-add to plain S2T.
Still, curious to hear what your take on that is.
On quick video transcription test this model is more accurate than AssemblyAI and Rev AI. It will be harder for them to sell pure ASR now. Some more business-oriented applications will still be important though, for example ASR as part of callcenter analytics solution or as a part of medical ERP system.
The value of automatic summarization is small, without AI it is very hard to make it right, you need to be an expert in the field to understand what is important.
Hello Nickolay :)
Diarization has always been the hard part for me, especially since it is very difficult to do comparisons within your domain. The evaluation metrics are not descriptive enough imo.
Would you say Titanet or EcapaTDNN are decent for use in production alongside, say, Whisper, or any other ASR output, if given the timestamps, so as to bypass running VAD? I'm just about to run experiments to try pyannote's diarization model and google's uis-rnn to test out how well they work, but it's a tad beyond my ability to evaluate.
I also wonder if Whisper architecture would be good for generating embeddings, but I feel it's focused so much on what is said rather than how it's said that it might not transfer over well to speaker tasks.
Tested this out in the span of a few hours and got a solution up and running to download the video from Youtube, spit out the transcription and upload the resulting transcription file externally. We're still missing a piece to upload directly to YouTube, but it's a start!
As a part of this experiment, we built out some templates that will allow anyone to play around with Whisper in our platform. If you're interested in seeing it, we built a video for doing the process with our templates [2], or directly with Python [3].
Hope someone finds this useful!
[1] https://www.shipyardapp.com [2] https://www.youtube.com/watch?v=XGr4v3aY1e8 [3] https://www.youtube.com/watch?v=xfJpGgyUkvM
I was originally using Adobe Premiere Pro's speech to text to do it, and wrote Python to convert its output to the Hyperaudio format on GitHub. With this, I can totally skip all of that step and this is fully open source, too.
App idea:
Build an app that takes a video and uses Hyperaudio or a similar project to add a clickable and searchable transcript (clicking in transcript seeks video)
A second run gave better results, but in most runs I do see instances where phrases repeat from 2-20 times.
I'm surprised by the quality on non-English languages, given that 80+% of the training data is English, and the rest is split between tens of languages.
It's sometimes close to perfect, and sometimes goes off the rail; I think that maybe the model tries to establish some sort of consistency for each sentence; if starts wrong for the first few words of a sentence, it can't build the rest properly.
But it's super fun.
1. Make sure you're using a model that isn't suffixed with `.en` (`base`, not `base.en). 2. Use `model.transcribe(your_input_audio, language='Japanese', task='translate')` ... with the appropriate input language.
>result = model.transcribe("audio.wav", language="english")
https://clips.twitch.tv/ReliablePopularWerewolfOSkomodo-pcuw...
because... hard accent.
first run whisper thought its welsh so I had to run with --language en , and it did pretty well
https://i.imgur.com/TQiYU9X.png
took 36 seconds in Google colab
However, it runs very slowly. It uses the CPU on my macbook, presumably because it hasn't got a NVidia card.
Googling about that I found [plaidML](https://github.com/plaidml/plaidml) which is a project promising to run ML on many different gpu architectures. Does anyone know whether it is possible to plug them together somehow? I am not an ML researcher, and don't quite understand anything about the technical details of the domain, but I can understand and write python code in domains that I do understand, so I could do some glue work if required.
result = model.transcribe("audio.wav", task="translate")
But your post helped me figure out the above, so thank you!
whisper never-gonna-give-you-up.mp3 --language English --model small
[00:00.000 --> 00:27.000] We're no strangers to love You know the rules and so do I
[00:27.000 --> 00:35.000] I feel commitments while I'm thinking of You wouldn't get this from any other guy
[00:35.000 --> 00:43.000] I just wanna tell you how I'm feeling Gotta make you understand
[00:43.000 --> 00:47.000] Never gonna give you up Never gonna let you down
[00:47.000 --> 00:53.000] Never gonna run around and desert you Never gonna make you cry
[01:00.000 --> 01:09.000] We've known each other for so long Your heart's been aching but you're too shy to say
[01:09.000 --> 01:17.000] Inside we both know what's been going on We know the game and we're gonna play it
Note that I did not omit 00:53.000 -> 01:00.000.
Shouldn't there be some type of unintelligible warning since it wasn't able to transcribe that part?
Here's a comparison of Basket Case by Greenday:
Small:
[00:00.000 --> 00:05.000] Do you have the time to listen to me whine
[00:05.000 --> 00:10.000] About nothing and everything I'll have once?
[00:11.000 --> 00:16.000] I am one of those melodramatic fools
[00:16.000 --> 00:20.000] Neurotic to the bone, no doubt about it
[00:23.000 --> 00:27.000] Sometimes I give myself the creeps
[00:27.000 --> 00:32.000] Sometimes my mind plays tricks on me
[00:32.000 --> 00:38.000] It all keeps headed up, I think I'm pregnant
[00:38.000 --> 00:43.000] And I'm just paranoid, I'm just stuck
[00:47.000 --> 00:52.000] I went to a shrink to have a life like my dreams
[00:52.000 --> 00:57.000] She says it's like a sex that's bringing me down
[00:57.000 --> 01:03.000] I went to a whore, he said my life's a bore
[01:03.000 --> 01:08.000] Choked with my widest buzz that's bringing her down
[01:10.000 --> 01:14.000] Sometimes I give myself the creeps
[01:15.000 --> 01:19.000] Sometimes my mind plays tricks on me
[01:19.000 --> 01:25.000] It all keeps headed up, I think I'm pregnant
[01:25.000 --> 01:30.000] And I'm just paranoid, I'm just stuck
[01:30.000 --> 01:48.000] Grasping to control, it's all I better hold on
[02:08.000 --> 02:12.000] Sometimes I give myself the creeps
[02:13.000 --> 02:17.000] Sometimes my mind plays tricks on me
[02:18.000 --> 02:23.000] It all keeps headed up, I think I'm pregnant
[02:23.000 --> 02:30.000] And I'm just paranoid, I'm just stuck
[02:53.000 --> 03:13.000] Thanks for watching!
[00:00.000 --> 00:05.000] Do you have the time to listen to me whine
[00:05.000 --> 00:10.000] About nothing and everything all at once?
[00:11.000 --> 00:16.000] I am one of those melodramatic fools
[00:16.000 --> 00:20.000] Neurotic to the bone, no doubt about it
[00:23.000 --> 00:27.000] Sometimes I give myself the creeps
[00:27.000 --> 00:32.000] Sometimes my mind plays tricks on me
[00:33.000 --> 00:36.000] It all keeps adding up
[00:36.000 --> 00:39.000] I think I'm cracking up
[00:39.000 --> 00:41.000] Am I just paranoid?
[00:41.000 --> 00:43.000] Am I just sad?
[00:47.000 --> 00:50.000] I went to a shrink
[00:50.000 --> 00:53.000] To analyze my dreams
[00:53.000 --> 00:58.000] She says it's lack of sex that's bringing me down
[00:58.000 --> 01:01.000] I went to a whore
[01:01.000 --> 01:04.000] He said my life's a bore
[01:04.000 --> 01:09.000] So quit my whining cause it's bringing her down
[01:10.000 --> 01:14.000] Sometimes I give myself the creeps
[01:16.000 --> 01:20.000] Sometimes my mind plays tricks on me
[01:20.000 --> 01:23.000] It all keeps adding up
[01:23.000 --> 01:26.000] I think I'm cracking up
[01:26.000 --> 01:28.000] Am I just paranoid?
[01:28.000 --> 01:30.000] Am I just sad?
[01:40.000 --> 01:44.000] Grasping to control
[01:44.000 --> 01:50.000] So I better hold on
[02:07.000 --> 02:11.000] Sometimes I give myself the creeps
[02:11.000 --> 02:16.000] Sometimes my mind plays tricks on me
[02:16.000 --> 02:19.000] It all keeps adding up
[02:19.000 --> 02:22.000] I think I'm cracking up
[02:22.000 --> 02:24.000] Am I just paranoid?
[02:24.000 --> 02:52.000] Am I just sad?
[02:54.000 --> 02:58.000] Thanks for watching!The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out and seeing if it's as robust as it seems, including when confronted with audio speech with natural tics and uhhh's and uhmm's and everything in-between.
I think it's fair to say that AI-transcription accuracy is now decidedly superior to the average human's, what the implications of this are I'm not sure.
Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement other people who need to gather poor-quality audio at scale, though much less great for the targets of repressive authority.
Having this fully open is a big deal though - now that level of transcription ability can be wrapped as an audio plugin and just used wherever. Given the parallel advances in resynthesis and understanding idiomatic speech, in a year or two I probably won't need to cut out all those uuh like um y'know by hand ever again, and every recording can be given an noise reduction bath and come out sounding like it was recorded in a room full of soft furniture.
97% accuracy means roughly three or four errors per minute of speech. That seems potentially extremely problematic for something like law enforcement use where decisions with significant impact on people's day and/or life might be made on the basis of "evidence".
For technical content, I use Rev.com and provide a glossary and real humans do the transcript. Other AI transcription services get lots wrong because the context often matters. Words like "TCP/IP" or "FAT disk format" or "Big Endian" I've never found AI so far to handle well.
I'm interested to test out whisper on this one.
I will try to put the code to the test, see how it goes.
That said there is value in integration of this into other things.
Perhaps it will encourage people to add voice command to their apps, which can be sent to gpt3
From what I can gather:
1. Includes model weights. I can't find the URL, but they reference them enough and have a CLI tool, so I presume I just haven't found them yet.
2. Includes code: https://github.com/openai/whisper
3. Released under MIT License: https://github.com/openai/whisper/blob/main/LICENSE
[0] https://www.youtube.com/watch?v=DS6pE88Xg3s
[1]
$ yt-dlp --extract-audio --audio-format mp3 -o wire-fuck.mp3 https://www.youtube.com/watch?v=DS6pE88Xg3s
$ whisper --language en wire-fuck.mp3
[00:00.000 --> 00:02.000] Oh
[00:13.260 --> 00:15.260] Fuck
[00:15.260 --> 00:31.260] Motherfucker
[00:50.700 --> 00:52.700] Fuck
[00:52.700 --> 00:58.700] Oh
[00:58.700 --> 01:10.700] Fuck
[01:28.700 --> 01:55.900] Fuck
[02:02.340 --> 02:03.700] Motherfuck.
[02:10.220 --> 02:11.220] Oh, fuck.
[02:11.780 --> 02:12.780] Oh, fuck.
[02:25.900 --> 02:27.900] Fuck, fuck, fuck, fuck, fuck, fuck.
[02:27.900 --> 02:28.900] Motherfucker.
[02:32.900 --> 02:33.900] Oh, fuck.
[02:34.900 --> 02:35.900] Fuck.
[02:35.900 --> 02:36.900] Oh, fuck.
[02:36.900 --> 02:37.900] Oh, fuck.
[02:37.900 --> 02:38.900] Oh, fuck.
[02:48.900 --> 02:49.900] Motherfucker.
[02:53.900 --> 02:54.900] Fucking A.
[02:54.900 --> 02:56.900] Mm hmm.
[02:56.900 --> 03:12.900] Fuck.
[03:26.900 --> 03:28.900] Motherfucker.
[03:28.900 --> 03:32.900] Fuck me.
[03:58.900 --> 04:01.900] Oh.
[04:28.900 --> 04:34.900] Fuck. Speaker 0 00:00:12 Oh, fuck motherfucker. Okay. Fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck.
My little fuck.
Speaker 1 00:02:10 Oh, fuck. Oh, fuck,
Speaker 0 00:02:25 Fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck my motherfucker.
Speaker 1 00:02:53 Fucking a.
Speaker 0 00:02:54 Mm-hmm. <affirmative> motherfucker. Fuck me. Um,EDIT: Tried it and it worked great! It is very easy to use. I just did the pip install line in the readme and was ready to go. You literally just run the one pip install line, and then you run the program in the format "whisper my_audio.wav" and it goes. Really nice job OpenAI!
Whisper SoTA
LibriSpeech test-clean 2.7% 1.8%
LibriSpeech test-other 5.6% 2.9%
Switchboard 13.1% 4.9%
CallHome 15.8% 9.5%
What an interesting approach. What benefits does this have over having two dedicated models, one for speech-to-text, and another for translation?
It just seems so odd, given the problems of speech-to-text and Spanish-to-English seems so different from one another (in terms of the problem domain). Seems so unusual to have both handled by one model!
Does knowledge of speech-to-text carry over into knowledge of translation? Does knowledge of translation carry over into knowledge of speech-to-text? So weird.
We also see in image generation models that multi-modal networks are more powerful than single purpose networks. As we move towards more advanced AI systems I suspect we will see more and more generalizable networks with distinct advantages over separate networks that get plugged together.
If you want to give it a shot, you can find the python script in this repo: https://github.com/tobiashuttinger/openai-whisper-realtime
A bit more context on how it works: The systems default audio input is captured with python, split into small chunks and is then fed to OpenAI's original transcription function. It tries (currently rather poorly) to detect word breaks and doesn't split the audio buffer in those cases. With how the model is designed, it doesn't make the most sense to do this, but i found it would be worth trying. It works acceptably well.
Have you thought of using VAD (voice activity detection) for breaks? Back in my day (a long time ago) the webrtc VAD stuff was considered decent:
https://github.com/wiseman/py-webrtcvad
Model isn’t optimized for this use but I like where you’re headed!
Took マッコウクジラ14頭が海岸に打ち上げられる オーストラリア(2022年9月21日) https://www.youtube.com/watch?v=bZkNIzeRBk4
Extracted audio with youtube-dl -f bestaudio https://www.youtube.com/watch\?v\=bZkNIzeRBk4
Converted into [00:00.000 --> 00:13.000] オーストラリア南部の島で、真っ向くじら14棟が海岸に打ち上げられて死んでいるのが見つかり、専門家が調査のため原地入りしました。 [00:13.000 --> 00:25.000] 原地メディアによりますと、オーストラリア南部のキング棟で、19日、少なくとも14棟の真っ向くじらが海岸に打ち上げられて死んでいるのが見つかりました。 [00:25.000 --> 00:31.000] ほとんどが若いオーストを見られ、専門家が現場に重むき調査に当たっています。 [00:31.000 --> 00:41.000] くじらの死害は大きく運んだり埋めたりすることが難しいため、自然に分解されるのを待つ方針が検討されています。 [00:41.000 --> 00:52.000] また、死害を狙い、サメが海に集まる可能性があるとして、原地東局はサーファーなどに周囲に近づかないように呼びかけています。 [00:52.000 --> 01:02.000] 一方、21日にはタスマニア棟でおよそ230棟のくじらが浜辺に打ち上げられた状態で見つかりました。 [01:02.000 --> 01:07.000] およそ半数がまだ生きている模様で急助活動が進められています。 [01:07.000 --> 01:23.000] 見つかったのは、ゴンドーくじらの仲間と見られています。
Here are the exact steps to follow to get it running on Ubuntu 22.04 via WSL and yt-dlp:
1. pip install git+https://github.com/openai/whisper.git
2. yt-dlp -f 'ba' -x --audio-format mp3 https://www.youtube.com/watch/?v\=bZkNIzeRBk4
3. renamed the file to test.mp3
4. whisper test.mp3 --language Japanese --task translate --model large
Amazing work.
I just wonder what system requirements Whisper has and whether there are open source voice recognition models that are specifically built for embedded devices.
Edit: According to this comment[0] the base model runs in real time on an M1 CPU. The tiny model apparently decodes an audio file twice as fast. These are promising results.
You could use really small chunk sizes and process them in a streaming fashion, but that would impact accuracy, as you're significantly limiting available context.
[00:00.000 --> 00:06.500] Since the last one started, the number of times I've eaten has decreased.
[00:06.500 --> 00:11.000] If I get too carried away with the last one, I'll get hungry and do it.
[00:11.000 --> 00:14.500] I don't have time to eat.
[00:15.500 --> 00:18.000] I'm going to eat now.
[00:20.000 --> 00:23.000] It's going to take about 10 minutes from here.
[00:23.000 --> 00:31.000] It's been a while since I've had my last meal.
[00:31.000 --> 00:36.000] I feel like I'm losing my女子力.
[00:36.000 --> 00:39.000] I have to go back to my original self.
[00:39.000 --> 00:44.000] I have to get ready and go to bed.
[00:44.000 --> 00:46.000] It's not good.
[00:46.000 --> 00:51.000] I've been drinking a lot lately, so I'm going home.
[00:51.000 --> 00:53.000] I have to get my nails done this fall.
[00:53.000 --> 00:54.000] Halloween nails.
[00:54.000 --> 00:57.000] Halloween, Halloween, Halloween.
[00:57.000 --> 00:59.000] I'm going to the beauty salon today.
[00:59.000 --> 01:02.000] I'm going to get my nails done the day after tomorrow.
[01:02.000 --> 01:10.000] I used to look at a lot of clothes, but I stopped looking at them.
[01:10.000 --> 01:12.000] I'm going crazy.
[01:12.000 --> 01:22.000] My stomach's stopped in the middle of summer.> The source code must be the preferred form in which a programmer would modify the program. [...] Intermediate forms such as the output of a preprocessor or translator are not allowed.
If I asked a programmer from OpenAI to modify the model to better support Japanese speakers from Hokkaido, their "preferred form" of the model's source code would include the 680,000 hours of audio used to train the model.
Yes that means that there are almost no open source models and yes it's awesome that they released this and made the weights available. Just don't call it open source.
>>A decoder is trained to predict the corresponding text...
Prediction of expected text in the context of the previous text.
While this is valuable in casual transcription, it can be extremely dangerous in serious contexts.
From personal experience, having given a deposition with an "AI" transcription, it will literally reverse the meanings of sentences.
This is because it produces the EXPECTED output in a context, and NOT THE ACTUAL OUTPUT.
Like a speaker that clips the output, these types of systems 'clip' the really valuable information out of a transcription. Worse yet, this is a completely silent failure, as the transcript LOOKS really good.
Basic info theory shows that there is more information contained in 'surprising' chunks of data than in expected ones. These systems actively work to substitute 'expected' speech to overwrite 'surprising' speech.
The transcript I got was utter trash, multiple pages of errata I had to submit when the normal is a couple of lines. And as I said, some literally reversed the meaning in a consequential way, and yet completely silently.
This kind of silent active failure mode is terrifying. Unless it is solved, and I see no way to solve it without removing ALL predictive algos from the system, these types of systems must not be used in any situation of serious consequence, at least not without real redundancy and backup.
Curious what anyone is using these days for a real-time transcription. It doesn't have to be perfect, but just good enough.
My kids watch some youtube vidoes where people will make a mod where it converts them talking to text then look for keywords and spawn a boss in Terraria if you say the wrong keyword etc.
I made a clone of that with the .NET System.Speech.Recognition library. It... works.. but my biggest problem is that #1 it waits until you are done speaking to translate to text on the callback, so there was too much of a delay for it to be fun.. the point is that it will be checking a stream of chatter. #2 is the recognition is pretty crap, I mean it's nearly good enough for my silly purpose but it's still pretty bad.
It's doing seconds of translation per minute for me at least.
https://developer.apple.com/documentation/speech/recognizing...
Also, see `requiresOnDeviceRecognition`
That's so, so far beyond the previous state-of-the-art, it's absurd.
As for speed, to a computer we don't talk very fast, not even that guy.
I wonder if it could handle Rap God by Eminem....Let's find out!
A 3m07s flac took 5m to transcribe:
$ whisper --device cpu 'BLACKPINK - BORN PINK/01 Pink Venom.flac'
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: korean
[00:00.000 --> 00:10.000] Blackpink
[00:11.000 --> 00:14.000] Kick in the door, wave in the coco
[00:14.000 --> 00:16.000] 팝콘이는 친게 껴들 생각 말고
[00:16.000 --> 00:19.000] I talk to talk, run ways I walk walk
[00:19.000 --> 00:21.000] 힘 감고 팝 팝 안 봐도 척
[00:21.000 --> 00:24.000] By one and two by two
[00:24.000 --> 00:26.000] 내 손끝 두 하나에 타면 아지은 중
[00:26.000 --> 00:30.000] 갓 자쇼 지금 화려해 T makes no sense
[00:30.000 --> 00:32.000] You couldn't get a dollar out of me
[00:33.000 --> 00:38.000] 자 오늘 밤이야 눈톱을 품고
[00:38.000 --> 00:41.000] 미혼을 뺏음 down
[00:41.000 --> 00:43.000] Look what you made us do
[00:43.000 --> 00:47.000] 천천히 널 잠재울 파이어
[00:48.000 --> 00:52.000] 잠이 날 만큼 아름다워
[00:52.000 --> 00:53.000] I bring the pain like
[00:53.000 --> 00:57.000] 디스탑, 팽팽, 디스탑, 팽팽, 디스탑, 팽팽, 팽팽
[00:57.000 --> 00:58.000] Get em, get em, get em
[00:58.000 --> 01:00.000] Straight till you don't like
[01:00.000 --> 01:01.000] Whoa, whoa, whoa
[01:01.000 --> 01:03.000] Straight till you don't like
[01:03.000 --> 01:04.000] Ah, ah, ah
[01:04.000 --> 01:05.000] Taste that, pink venom
[01:05.000 --> 01:06.000] Taste that, pink venom
[01:06.000 --> 01:08.000] Taste that, pink venom
[01:08.000 --> 01:09.000] Get em, get em, get em
[01:09.000 --> 01:11.000] Straight till you don't like
[01:11.000 --> 01:12.000] Whoa, whoa, whoa
[01:12.000 --> 01:13.000] Straight till you don't like
[01:13.000 --> 01:14.000] Ah, ah, ah
[01:14.000 --> 01:15.000] Blackpink and Amo
[01:15.000 --> 01:17.000] Got it by the smack ram
[01:17.000 --> 01:18.000] But rest in peace
[01:18.000 --> 01:19.000] Please light up a candle
[01:19.000 --> 01:20.000] This the knife of a vando
[01:20.000 --> 01:22.000] Messed up and I'm still in saline
…SNIP…To be able to give it text and hear the speech. A TTS (text to speech).
As a language learner, the ability to create my own sentences (based on existing ones I have, in changing a word here or there). Would be amazing.
How long till we have this I wonder. I know I could use a service to do this currently. But having something running locally, I'd prefer.
Hopefully someone in the OpenAI team reads this. :)
So I think TTS is a logical part of the system. I also think that there are peculiarities of voice interaction that aren’t captured in text training datasets, so they would need to do some fine tuning on actual voice conversation to make it feel natural.
All in due time I suppose.
On the search part there are lots of vector search companies - Weaviate, Deepset Haystack, Milvus, Pinecone, Vespa, Vald, GSI and Qdrant. But it has not become generally deployed on most systems, people are just finding out about the new search system. Large language models are still difficult to run locally. And all these models would require plenty of RAM and GPU. So the entry barrier is still high.
That's intriguing. You can just set the model to transcribe everything into English, no matter which language the speaker is using, and it just works. Given that many people are much better at understanding English than at speaking it, this might make voice interfaces much more accessible without much work.
On one hand, it may capture something "deeper" about language.
On the other hand, it's likely to do great in general, but miss particularities of some language.
Understanding the coverage of the training model seems a perennial problem. Is there any (shorthand) way to compare language model training corpora?
Clearly if they use common subsets we have a literal comparison. I'm more interested in whether there's progress in characterizing corpora by speech styles, fluency, vocabulary sets, (noise) environment, emotionality, proposition types, etc.
(btw: 25 minutes for a 9-minute segment on a 12-thread x86. Lots of jargon spelled as it sounds. Sentences capitalized but no punctuation. Overall good.)
Some observations:
- The full translation of the 6:22 minute video takes about 22 seconds (17x real time)
- It recognizes the language by default (and did a good job to recognize it was french audio)
- MIT License [3]!
- The quality of the transcription is good, but not perfect.
- The quality of the translation (if you don't consider transcription errors as a translation error) is generally very good.
---
The transcription:
> Bonjour à tous, <error>j'suis</error> espère que vous allez bien, c''est ENTI. Et aujourd', <error>aujourd',</error> on se retrouve <error>un peu physique</error> pour parler de la termo dynamique. Vous ne vous inquiétez pas, ça va bien se passer. On va y aller ensemble, <error>être à par exemple,</error> je vous accompagne à travers une série de vidéos pour vous expliquer les principes de base en termo dynamique. Et bah, c''est parti, on va y aller tranquillement. Lidée, c''est vous puissiez comprendre la termo dynamique dans son ensemble. Donc, je vais vraiment prendre mon temps pour <error>couplisser</error> bien comprendre les notions,
The translation:
> Hello everyone, I hope you're doing well, it's NT and today we find ourselves a little physical to talk about the thermo dynamic. Don't worry, it's going well, we're going to go together and be the same. I'm going to accompany you through a series of videos to explain the basic principles in thermo dynamic. Well, let's go, <error>we're going to go quietly</error>. The idea is that you can understand the thermo dynamic <error>in sound together</error>. So I'm really going to take my time to understand the notions,
---
All in all very happy that OpenAI is publishing their models. If Stable Diffusion is any guide, people will hack some crazy things with this.
[1] https://github.com/openai/whisper [2] https://www.youtube.com/watch?v=OFLt-KL0K7Y [3] https://github.com/openai/whisper/blob/main/LICENSE
2 questions:
1) how does it compare to state of the art FOSS solutions? I'm seeking about DeepSpeech or Vosk
2) would it be somehow possible to associate timestamp to the words recognized? That would be amazing for things such as audio editing or skipping to a particular location on a video
But in general the model is robust and accurate and trained on the amount of speech we never dreamed about in Vosk. We will certainly benefit from this model as a teacher (together with others like gigaspeech models). I recently wrote about it https://alphacephei.com/nsh/2022/06/14/voting.html
for 2), it's actually written in the description: "phrase-level timestamps", so it should be possible (phrase level is neat for skipping to a special location on a video, but maybe not for audio editing).
Skimming the codebase I can't immediately see code to do additional training.
Being able to fine-tune the model to a specific language or case (eg. teach it specifically about some technical topic that might not be so prevalent in the current train set) would be majorly disruptive to current SOTA in "callcenter analytics" tech. Especially when combining Whisper with GPT3.
No surprise that it appears to have successfully transcribed all the recordings of Harvard Sentences I could find. https://en.wikipedia.org/wiki/Harvard_sentences
For a company that raised $1B, that's not exactly living up to their name and original mission.
I feel bad about GPT-3 and DALL-E being released under the terms they were, but I don't feel bad about this. I'm not going to condemn OpenAI for the good things they did, but I will hold them accountable for bad things or good ones they didn't do.
I'd given up on OpenAI being open or ethical, but this is a start. It took them down from "evil super-villain" status to mere villain.
I can already tell this is much better than any of the existing open source projects with the exception of the wav2* sequence of projects and potentially nvidia's nemo.
I can understand not releasing GPT-3, even if I disagree with the decision.
The version I choose to believe: stability.ai ate DALL-E for lunch, and that woke them up.
Why do you disagree?
"tiny.en": "https://openaipublic.azureedge.net/main/whisper/models/d3dd5..."
"tiny": "https://openaipublic.azureedge.net/main/whisper/models/65147..."
"base.en": "https://openaipublic.azureedge.net/main/whisper/models/25a85..."
"base": "https://openaipublic.azureedge.net/main/whisper/models/ed3a0..."
"small.en": "https://openaipublic.azureedge.net/main/whisper/models/f953a..."
"small": "https://openaipublic.azureedge.net/main/whisper/models/9ecf7..."
"medium.en": "https://openaipublic.azureedge.net/main/whisper/models/d7440..."
"medium": "https://openaipublic.azureedge.net/main/whisper/models/345ae..."
"large": "https://openaipublic.azureedge.net/main/whisper/models/e4b87..."
Anyway I am pretty terrible with email but short exchanges can work for me, or maybe we can connect over signal. Send me a message to my email in my profile and I would be happy to sync up!
I’d love to chat and hear about how you use this! My email is in my profile, or I’m @tekacs on Twitter (and everywhere). :)
Comparing the readily available test sets from the paper to some of my personal robust models (for the Talon models, this is greedy decoding, no language model):
Talon Talon Talon Whisper wav2vec 2.0
28M 300M 1B Large 960h
librispeech clean 3.21 2.52 2.40 2.7 2.7
librispeech other 8.21 6.56 5.63 5.6 6.2
common voice 13.88 11.65 8.86 9.5 29.9
tedlium 7.51 6.55 5.47 4.0 10.5
> Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.
It looks like Whisper is probably leaving a lot of accuracy on the table, but initially it does seem to be a lot more robust than general "SOTA" models.
For a quick comparison, Silero's accuracy charts are kind of nice because they post results for a large variety of datasets. Scroll down to the EN V6 xlarge EE model (not the xlarge CE) [1]
[1] https://github.com/snakers4/silero-models/wiki/Quality-Bench...
Now I wonder if it works equally well with Spanish from Spain (and its different regions) and Spanish from the New World (and in its myriads of different flavours).
The Mycroft has done a lot of cool and important work in the field to ship an actual personal assistant product (stuff like wake word detection).
Though I assume the amount of Norwegian it has been exposed to is fairly limited, so in that light I'm actually impressed as well.
I tried it on a news segment from the radio[1], this is the large model output:
[00:14.000 --> 00:17.200] En skamløs krenking av FN pakten.
[00:17.200 --> 00:24.000] USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
[00:25.500 --> 00:29.400] Arbeidsklær som er ment til å være til begge kjønn, har det med å være tilpasset.
[00:29.400 --> 00:33.400] Men hvordan ville det gått, om det var motsatt?
[00:34.100 --> 00:38.900] Dyrevernsorganisasjon vil ha digital merking av regnstyr,
[00:38.900 --> 00:44.900] men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
[00:45.600 --> 00:51.400] Mange strømselskaper er positive til å tilby kundene fastpris på strøm, og det årevis.
[00:51.400 --> 00:59.900] Da risikerer de å måtte betale mye i nettopp åretsvis, sier aktører som aldri tilbyr fastpris.
[00:59.900 --> 01:21.900] Dette er onsdagens Dagsnytten. Jeg heter Espen Ås.
* En skamløs krenking av FN-pakten. USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
* Arbeidsklær som er ment å være til begge kjønn, er som regel tilpasset ... menn. Hvordan hadde det gått om det var motsatt?
* Dyrevernsoganisasjon vil ha digital merking av reinsdyr, men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
* Mange strømselskaper er positive til å tilby kundene fastpris på strøm - og det i årevis.
- Da risikerer de å måtte betale mye i nettopp; årevis, sier aktør som aldri tilbyr fastpris
Dette er onsdagens Dagsnytt 18 - jeg heter Espen Aas.
[00:14.000 --> 00:17.000] A shameless violation of the UN treaty.
[00:17.000 --> 00:24.000] The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
[00:24.000 --> 00:33.000] Work clothes that are meant to be for both genders have to be suitable, but how would it be if it was the other way around?
[00:34.000 --> 00:44.000] The animal welfare organization will have a digital marking of reindeer, but the industry itself insists on the old traditional way of tearing a knife.
[00:45.000 --> 00:51.000] Many electricity companies are positive in offering customers fixed electricity prices, and that is annual.
[00:51.000 --> 00:58.000] Then they risk having to pay a lot in just a year, says an actor who has never offered fixed prices.
[00:58.000 --> 01:20.000] This is Wednesday's Dagsnytt 18. My name is Espen Ås.
* A shameless violation of the UN Charter. The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
* Work clothes intended for both sexes are usually adapted to ... men. How would it have gone if it had been the other way around?
* Animal welfare organizations want digital marking of reindeer, but the industry itself insists on the old, traditional way of marking with a knife.
* Many electricity companies are positive about offering customers a fixed price for electricity - and for years.
- Then they risk having to pay a lot in precisely; for years, says a player who never offers a fixed price
This is Wednesday's Dagsnytt 18 - my name is Espen Aas.
It also didn't handle the hanging "... menn", instead thinking it was the start of the following sentence. Almost everyone would understand it was the end of the sentence based on the context.
The double-A vs Å is not an issue as it's the same letter, double-A is the older form.
The small model was considerably worse than the large one though.
Deepl didn't do any Norwegian last I looked, even though it does most other Germanic languages (including Danish and Swedish).
Duolingo doesn't have a Norwegian class for Germans either, though they do have one with English as the source language.
https://salsa.debian.org/deeplearning-team/ml-policy
BTW, wouldn't you take the existing model and do additional Hokkaido Japanese speaker training on top of it, rather than retraining the model from scratch?
> If I asked a programmer from OpenAI to modify the model to better support Japanese speakers from Hokkaido, their "preferred form" of the model's source code would include the 680,000 hours of audio used to train the model.
Precisely. These 'users' lifting the model can't do it themselves. You will still be contacting OpenAI for support or to add support for another language and they will be the ones able to modify the model.
> Just don't call it open source.
That is true, it is still closed source and already we are seeing the hype squad already apologising to OpenAI as they 'open sourced' a closed model that you can't modify yourself.
OpenAI is still business as usual and nothing has changed.
This isn't quite correct. The model weights are all you need to fine tune the data on your own with your own audio.
Without the original training set this still isn't open source. But you aren't powerless to modify the model without the original training set.
You can do a lot with weights and no training data - for example you can pull the end layer off it and use it as a feature extractor.
And to modify it for Japanese speakers you'd fine train the existing model on additional data. If you wanted to modify the model you can (sometimes, depending on what you want to do) modify an existing architecture by removing layers, adding replacements and fine tuning.
I don't quite know what the right analogy of trained data is. In many ways it is more valuable than the training data because the compute needed to generate it is significant. In other ways it is nice to be able to inspect the data.
> The source code must be the preferred form in which a programmer would modify the program.
As a machine learning programmer I'd much prefer the weights than the raw data. It's no realistic for me to use that training data in any way with any compute I have access to.
YES! The old joke about "Artificial Stupidity" is actually more true than anyone realized.
These statistical so-called-AI systems actually work to actively REMOVE or sanitize out any unexpected information, making it all conform with the EXPECTED results from the training set.
This not only REMOVES the most high-information 'surprising' or unexpected nuggets, it actively HIDES them. When something unexpected comes up, it gets force fit into the expected prediction algorithms and output as if it were good.
I'm not saying that there are no useful things that can be done with this technology — there is a LOT of mundane work out there to be done.
But, we will never get this type of "AI" saying "Huh, that's odd, I wonder why that is?", which is exactly the kind of observation that leads a prepared and fertile mind to great discoveries.
One item I remember was that I said "Dr Kemeny" in relation to Dartmouth College (he was a famous mathematician, invented the BASIC programming language and was president of the college). It replaced those instances with "Jack Kennedy".
In another instance, I said that "Evidently, you have a reading comprehension problem.". It replaced it with "Evidently, I have a ...", completely reversing the meaning.
There was zero problems with the microphones or audio, and it was not rushed or mumbled talk. There were 80+ other examples over a few hours of talking, and some from other speakers. And those were just the obvious ones I could catch.
Another massive problem with this technology is that a human stenographer can notice when s/he missed something and didn't hear and ask the speaker to repeat or clarify what was said, and will often during a pause request clarification on spelling of names, addresses, etc. In contrast, this "AI" technology just barges ahead ASSuming that it knows what it is doing and inserts literally whatever sounds good in the transcript, completely silent that it doesn't have a clue.
Having seen this up close, I'm of the strong opinion that anyone foisting this software on the market without huge warnings that this is not usable for any critical functions is, basically a fraud. They know or certainly should know that these failures not only exist but are common and systemic, yet they barge along like it is OK. It is not.
Also, you are comparing Whisper's highlight reel with everyday performance of other models. Nobody shows their weaknesses in their highlight reel.
Also, I feel like shipping to the cloud and back has been shown to be just as fast as on device transcription in a lot of scenarios. Doing it on device is primarily a benefit for privacy and offline, not necessarily latency. (Although, increasingly powerful smartphone hardware is starting to give the latency edge to local processing.)
Siri's dictation has had such terrible accuracy for me (an American English speaker without a particularly strong regional accent) and everyone else I know for so many years that it is just a joke in my family. Google and Microsoft have much higher accuracy in their models. The bar is so low for Siri that I automatically wonder how much Whisper is beating Siri in accuracy... because I assume it has to be better than that.
I really wish there was an easy demo for Whisper that I could try out.
Maybe I'm missing something, but I don't see the problem here.
I just ran some benchmarks - M1 Max, pytorch, with a 1.29 second flac (looks like the matrix math was running on a single thread):
tiny
146.522ms detect_lang
549.131ms decode_one
0.057ms tokenizer
base
354.885ms detect_lang
1046.679ms decode_one
0.011ms tokenizer
small
803.892ms detect_lang
3194.503ms decode_one
0.017ms tokenizer
medium
2279.689ms detect_lang
10128.255ms decode_one
0.023ms tokenizer
large
3656.478ms detect_lang
17249.024ms decode_one
0.016ms tokenizer> in sound together
That's hilarious and honestly, incredibly bad. "Dans son ensemble" is a very common idiom (meaning "as a whole") while "in sound together" has to be pretty rare. "Son" means "his/hers/its" as well as "sound", and the former meaning is probably more common in general so I have no idea how this result could arise.
"Termo" also doesn't exist in French, it's "thermo", so the transcript even makes orthographic errors.
And I forgot about "couplisser" which is also a hilarious made-up word that sounds like it could mean something, but doesn't! Edit Google finds exactly one reference of this, in a patent with a typo on the word "coulisser".
I'm still impressed by the transcript quality since it covers many languages, but the translation part is quite poor.
Both, wow. This is really interesting.
I have it running right now and it's not touching the GPU.
Although I don't know if they're using anything similar to what you suggest. Very cool idea, anyway!
https://github.com/mozilla/DeepSpeech-examples/blob/master/m...
But yeah you make a good point about LLMs being too large to run on a normal PC. I do somewhat suspect that we might see some rapid acceleration in the size of neural network processors as large models begin to offer more utility. I think for now they have limited appeal but we’re already seeing things like Tesla’s Dojo make large leaps in capability to rapidly process complex networks.
In five to ten years we may see built in accelerators come standard in most computers capable of running very complex models. Already Apple provides ever more powerful accelerators in their phones. You could imagine Adobe offering real time diffusion models as part of Photoshop, among other things.
https://github.com/openai/whisper/blob/main/requirements.txt
should process most formats
On that note, a core Nvidia NeMo developer I follow posted this: https://twitter.com/HaseoX94/status/1572748653189791745
He calls it a "T5 for ASR" paper :) More insights in there, have a look! Curious to see what your blog would put up as well!
I'm guessing maybe audio books along with the actual books would be the best source for such? I mean there's Mozilla Voice, but it's quite limited in the Norwegian department and perhaps not quite as interesting as an audio book would be.
https://mycroft-ai.gitbook.io/docs/using-mycroft-ai/customiz...
I'm currently trying to setup a deepspeech server on my raspberry pi to see if it works ok for commanding spotify.
Edit: just realised you said `TTS` not `STT`
[0]: https://github.com/Sheepybloke2-0/trashbot - It was called trashbot because the final implementation was going to look like oscar the grouch in a trashcan displaying the reminders.
Almond is also interesting as a voice assistant, though I think it doesn't perform speech recognition itself.
Don't believe what's on marketing pages, they rarely transfer to the real world. Will have to make time to try it and see. In theory, given task diversity and sheer number of hours, it should be a lot more robust but will wait on evidence before believing any claims on SoTA.
Original:
Trois mille six cents fois par heure, la Seconde
Chuchote Souviens-toi !– Rapide, avec sa voix
D'insecte, Maintenant dit Je suis Autrefois,
Et j'ai pompé ta vie avec ma trompe immonde !
Remember ! Souviens-toi ! prodigue ! Esto memor !
(Mon gosier de métal parle toutes les langues )
Les minutes, mortel folâtre, sont des gangues
Qu'il ne faut pas lâcher sans en extraire l'or !
> Trois mille six cents fois par heure, la seconde chuchote « Souviens toi », rapide, avec sa voix d''insecte, maintenant dit « Je suis autrefois », et j''ai pompé ta vie avec ma trompe immonde. « Remember, souviens toi, prodigue, est au mémoire, mon gosier de métal, parle toutes les langues, les minutes, mortelles folâtres, sont des gangs qu''il ne faut pas lâcher sans en extraire l''or. »
Not bad! Far from perfect but it's a difficult text. Interesting that it works better with Baudelaire than Pascal.
Original:
> Mes révérends pères, mes lettres n’avaient pas accoutumé de se suivre de si près, ni d’être si étendues. Le peu de temps que j’ai eu a été cause de l’un et de l’autre. Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte. La raison qui m’a obligé de me hâter vous est mieux connue qu’à moi. Vos réponses vous réussissaient mal. Vous avez bien fait de changer de méthode ; mais je ne sais si vous avez bien choisi, et si le monde ne dira pas que vous avez eu peur des bénédictins.
Transcription:
> Mes rêves errent pères, mais l'detre navais pas accoutumé de se suivre de si près ni d'detre si étendu. Le peu de temps que j'sais eu a été cause de l'de l'de l'de autre. J'sais n'detre plus longue que parce que j'sais pas eu le loisir de la faire plus courte. La raison qui m'sa obligée de me hâter vous est mieux connue qu'moi. Vos réponses vous réussissaient mal. Vous avez bien fait de changer de méthode, mais je ne sais pas si vous avez bien choisi et si le monde ne dira pas que vous avez eu peur des bénédictes.
Here there are many more mistakes, so many that the beginning of the text is unintelligible. The language from the 17th century is probably too different. Still on the "medium" model, as the large one crashes the Colab (not sure how to select a beefier machine.)
Still fascinating and exciting though.
My test gives that, much better than yours:
Mes *rêverants* pères, mes lettres n'avaient pas accoutumé de se suivre de si près ni d'être si étendues. Le peu de temps que j'ai eu a été cause de l'un et de l'autre. Je n'ai fait celle aussi plus longue que parce que je n'ai pas eu le loisir de *l'af*faire plus courte. La raison qui m'a obligé de me *ra*ter vous est mieux connue qu'à moi. Vos réponses vous réussiss*ez* mal. Vous avez bien fait de changer de méthode. Mais je ne sais si vous avez bien choisi et si le monde ne dira pas que vous avez eu peur des bénédict*eurs*.
I tried the beginning of L'étranger (because you seem to be a fan of Camus ;-)
Here's the original:
> Aujourd’hui, maman est morte. Ou peut-être hier, je ne sais pas. J’ai reçu un télégramme de l’asile : « Mère décédée. Enterrement demain. Sentiments distingués. » Cela ne veut rien dire. C’était peut-être hier.
> L’asile de vieillards est à Marengo, à quatre-vingts kilomètres d’Alger. Je prendrai l’autobus à deux heures et j’arriverai dans l’après-midi. Ainsi, je pourrai veiller et je rentrerai demain soir. J’ai demandé deux jours de congé à mon patron et il ne pouvait pas me les refuser avec une excuse pareille. Mais il n’avait pas l’air content. Je lui ai même dit : « Ce n’est pas de ma faute. » Il n’a pas répondu. J’ai pensé alors que je n’aurais pas dû lui dire cela. En somme, je n’avais pas à m’excuser. C’était plutôt à lui de me présenter ses condoléances.
Here's the transcription:
> Aujourdhui, maman est morte, peut être hier, je ne sais pas. J''ai reçu un télégramme de l''asile. Mère décédée, enterrement demain, sentiment distingué. Cela ne veut rien dire. C''était peut être hier.
> L''asile de Vieillard est à Maringot, à 80 km d''Alger. Je prendrai l''autobus à deux heures et j''arriverai dans l''après midi. Ainsi, je pourrai veiller et je rentrerai demain soir. J''ai demandé deux jours de congé à mon patron et il ne pouvait pas me les refuser avec une excuse pareille. Mais il n''avait pas l''air content. Je lui ai même dit, ce n''est pas de ma faute. Il n''a pas répondu. J''ai alors pensé que je n''aurais pas dû lui dire cela. En somme, je n''avais pas à m''excuser. C''était plutôt à lui de me présenter ses condoléances.
Except for the weird double quotes instead of the single apostrophe ('), it's close to perfect, and it only uses the "medium" model.
This is extremely exciting and fun! Happy to try other texts if you have something specific in mind!
I know there's some "mini-GPT" type models around, but they don't seem nearly as capable.
As mentioned elsewhere in the thread there are some large models around the 50-200B band that compete directly with GPT-3, but I haven’t used these.
(Work for Google, don't speak for them.)
Especially because ASR is a living organism. You have to constantly update your language model as new people, ideas, and words move into the normal lexicon. As people start talking about "COVID", "metaverse", "king charles", or whatever new things that happen, these need to be added to your language model. You need these updates monthly at a minimum and OpenAI didn't release the raw data which means you can't retrain it even if you wanted to spend the time/resources to.
So, this is an interesting research project and helpful for small teams and side projects, but it's unlikely it makes any real impact on the industry.
[00:00.000 --> 00:09.000] Look, I was going to go easy on you, not to hurt your feelings, but I'm only going to get this one chance.
[00:09.000 --> 00:11.000] Something's wrong, I can feel it.
[00:11.000 --> 00:17.000] It's just a feeling I've got, like something's about to happen, but I don't know what.
[00:17.000 --> 00:21.000] If that means what I think it means, we're in trouble, big trouble.
[00:21.000 --> 00:24.000] Had to be as bananas as you say, I'm not taking any chances.
[00:24.000 --> 00:26.000] You're just one to die for.
[00:26.000 --> 00:32.000] I'm beginning to feel like a rap god, rap god. All my people from the front to the back nod, back nod.So...probably?
Pre-post edit: I can't get it to work.
I've installed pytorch with cuda via pip3, installed the nVidia toolkit and it doesn't see it:
>>> import torch >>> torch.cuda.is_available() False
I've wasted like an hour and a half on it now. I'm not a python dev, and don't have any ML experience so this was just for fun and now it's not anymore.
I tried compiling pytorch with vulkan support, but there are a few LDFLAGS that are wrong. I'll try to solve that some time later.
One piece of advice: use distribution packages! Arch provides pytorch-cuda, and has PKGBUILDS as well.
For reproductibility, I wish we were all on Nix/Guix, but that's not the case (and CUDA+HW dependency would make it complicated).
And yes, what you describe could be done. But no, it won't reduce latency that much, because the model itself learns to delay the prediction w.r.t. the audio stream. That's why ASR-generated subtitles usually need to be re-aligned after the speech recognition step. And that's why there is research such as the FastEmit paper to prevent that, but then it is a trade-off between latency and quality again.
Also, running your "low-latency" model with 1s chunks means you now need to evaluate the AI 30x as often as if you'd be using 30s chunks.
I doubt people run online transcription for long periods of time on their phone very often, so the battery impact is irrelevant, and the model is ideally running (mostly) on a low power, high performance inference accelerator anyways, which is common to many SoCs these days.
As for running the AI 30x, on current hardware that'll make it slower than realtime. Plus all of those 1GB+ models won't fit into a phone anyway.
Whatever the default version that `pip install git+https://github.com/openai/whisper.git` grabbed didn't include it by default.
I found this thread and wondered if Whisper was accessing all the cores or the gpu, so I've spent a couple of hours trying to get whisper to access the gpu - following the points made in this thread, and googling how to install via brew the various components.
Long story short, I keep getting an error message
"RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU."
or when I set --device to gpu, it get the error: "RuntimeError: don't know how to restore data location of torch.storage._UntypedStorage (tagged with gpu)"
it's been a looong time since I wrote any code (remember basic?), so realise I may be missing a lot here!!
does anyone have any pointers?
thanks!
edit: I'm now trying it one more time after trying to set the cpu using this line:
map_location=torch.device('gpu')
and I get this message as whisper begins: ~/opt/anaconda3/lib/python3.9/site-packages/whisper/transcribe.py:78: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead")
then I wait for whisper to do it's magic ...tho it looks like it will remain very slow...
I can't find a source now, but I remember reading some code where the maintainer had to change a huge chunk of code because the point change for a dependency library literally flipped either how the library handled height/width or BGR channels (I can't remember which one but it was preposterous) from the 2.5.4 to the 2.5.5 version. There is no reason for doing that - it breaks everything just for grins and giggles.
Python itself is also a problem, but that's a rant for another day. Ah, how I wish Ruby had become the defacto language of choice for ML/Deep Learning!
I did some basic tests on CPU, the "small" Whisper model is in the ballpark of 0.5x realtime, which is probably not great for interactive use.
My models in Talon run closer to 100x realtime on CPU.
Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow.
I have no experience on the accuracy of Talon, but I’ve heard that most open source models are basically overfit to the test datasets… so their posted accuracy is often misleading. If Whisper is substantially better in the real world, that’s the important thing, but I have no idea if that’s the case.
tiny.en: ~18 sec/sec
base.en: ~14 sec/sec
small.en: ~6 sec sec/sec
medium.en: ~2.2 sec/sec
large: ~1.0 sec/sec (fairly wide variance when ramping up as this is slow to process individual clips)
> Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow.
It's not that simple. Many of the mobile ML accelerators are more targeted for conv net image workloads, and current-gen Intel and Apple CPUs have dedicated hardware to accelerate matrix math (which helps quite a bit here, and these instructions were in use in my tests).
Also, not sure which model they were using at 17x realtime on the 3090. (If it's one of the smaller models, that bodes even worse for non-3090 performance.) The 3090 is one of the fastest ML inference chips in the world, so it doesn't necessarily set realistic expectations.
There are also plenty of optimizations that aren't applied to the code we're testing, but I think it's fairly safe to say the Large model is likely to be slow on anything but a desktop-gpu-class accelerator just due to the sheer parameter size.
Like the colab notebook linked on the official Whisper github project page?
I don't think that's a requirement here. I've been playing with Whisper tonight, and even the tiny model drastically outperformed Siri dictation for me in my testing. YMMV, of course.
Stable Diffusion: Marks images as AI-generated. (invisible watermark, but still, it's there)
Photoshop: Requires time & effort from a human.
Fake news website: Requires time & effort from a human.
https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...
> Nous établissons que l'utilisation de données d'un tel nombre et d'une telle diversité est la raison pour laquelle le système est à même de comprendre de nombreux accents...
It doesn't sound natural at all. An idiomatic formulation would be more along the lines of:
Le recours à un corpus [de données] si riche et varié est ce qui permet au système de comprendre de nombreux accents (With 'corpus', 'données' is implied.)
Of course this is just an example, and I'm sure other French speakers could come up with a different wording, but "données d'un tel nombre et d'une telle diversité" sounds really wrong.
This is also weird and convoluted:
> Nous distribuons en tant que logiciel libre le code source pour nos modèles et pour l'inférence, afin que ceux-ci puissent servir comme un point de départ pour construire des applications utiles
It should at least be "le code source DE nos modèles" and "servir DE point de départ", and "en tant que logiciel libre" should placed at the end of the proposition (after 'inférence').
Also, "construire" isn't used for code but for buildings, and "applications utiles" is unusual, because "utiles" (useful) is assumed. "...pour le développement de nouvelles applications" would sound more French.
I kind of doubt it though -- the speaker doesn't have a Canadian accent (which is hard to miss), and in my (admittedly limited) experience, French Canadian isn't that different from French.
No one says "Nous", there, ever. Perhaps the politicians, while giving a speech. Everyone else uses the more informal "On".
I felt duped by my French classes.
It is often used for larger groups or when the group is not very personally connected. For instance when talking about your company doing something you will often use "nous". I would also use "nous" to refer to the whole list of invitees to a wedding. And in formal contextes like research papers, reports etc. You would never use "on", always "nous".
> We distribute as a free software the source code for our models and for the inference [...]
Should be
> We are open-sourcing models and inference code [...]
Another example
> We establish that the use of such a number of data is such a diversity and the reason why our system is able [...]
Should be
> We show that the use of such a large and diverse dataset leads to improved robustness [...]
Trivial to remove, I give you that. But AFAIK, the original repository + most forks put the watermark automatically unless you've removed it on your own.
almost all of the 'low-vram' variant forks either have an argument to turn off the watermark (it saves a bit of memory) or come with it disabled all together.
My point is that the Python API is more interesting than the txt2img script, and it doesn't add any watermarks.
EDIT: for what it's worth, Nvidia rated the A2 at 18 TFLOPS of FP16, and Apple rates the current A16 Neural Engine at 17 TFLOPS of FP16. I'm sure it's not an "apples to apples" comparison.
Sure, the Whisper Tiny model is probably going to be fast enough, but from my preliminary results I'm not sure it will be any better than other models that are much much faster at this power class.
Whisper Large looks pretty cool, but it seems much harder to run in any meaningful realtime fashion. It's likely pretty useful for batch transcription though.
Even if you hit a realtime factor of 1x, the model can leverage up to 30 seconds of future audio context. So at 1x, if you speak for 10 seconds, you'll potentially need to wait another 10 seconds to use the result. This kind of latency is generally unsatisfying.
----
With Siri dictation, I feel like I usually spend at least as much time correcting its mistakes as I do speaking the dictation itself. In some cases, that is still faster/easier than typing, but I would rather have a voice model that can work in about the same total amount of time without requiring constant corrections. If I speak for 30 seconds, then I can do other things for 30 seconds while my phone processes it… that might actually be preferable if it gets it right. Otherwise, I’ll be spending 30 seconds actively editing it anyways. Even an improvement on the number of edits required per dictation would be nice. Admittedly, I feel like Google and Microsoft already do a much better job here.
It could be interesting to use the tiny model to give a preview of the writing while the large model is taking its time, and then allow the user to tap on words that changed to see the predictions from the tiny model and correct back to them if they want. I was doing some experiments a few minutes ago, and on one audio clip, the tiny model wrote down a very literal interpretation of an uncommon sci-fi word, and that was more accurate than either the medium or the large models. The rest of the time, the larger models did better, as expected.
But, I don’t know. This is interesting to me, but I agree there could be issues with making is workable for real time transcription.
Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly read through hours of conversation and make decisions about whether to invest further resources (which might just mean another hour of listening back to the original audio)? Also absolutely. Bear in mind that a lot of these errors have little to no semantic impact, being on the same level as typos or misspellings in a written communication.
Bear in mind too that if law enforcement (honest or not) is so interested in you that they're willing to record your conversations, your day is already ruined, you just don't know it yet. The change here is one of scale rather than quality.
(edit for clarification: errors are not always something like "[UNINTELLIGIBLE]", where the system knows it doesn't know; they can also be misrecognitions that the system believes in with high confidence.)
Look, I have decades of experience dealing with human speech, and not just as an editor - I can trace the human voice from neural impulses in Broca's region through the physiology of vocal production, mechanical transduction into electrical signals, discrete fourier transforms of the resultant waveforms into spectral information and back again, the reproduction of altered signals from time-aligned speakers to create a sense of spatialization, how those are processed in the human ear, and how the cilia are connected by nerves back to your brain. I'm a good enough editor that I can recognize many short words by sight of a waveform, or make 10 edits in a row by sight and know it will sound good on playback.
So when I say that machine transcription is as good as human realtime transcription now, I say so with the clear expectation that those decades of craft are very close to being rendered obsolete. I absolutely expect to hand off the mechanical part of editing to a machine within 2 years or so. It's already at the stage where I edit some interviews as text, like in a word processor, and then export the edited document as audio and it's Good Enough - not for every speaker, but more than half the time.
NPR and a lot of commercial broadcasters cut their material this way already, because you can get the same result from 30 minutes of reading and text editing that would require 3 hours of pure audio editing with no transcription.
Suppose 90% of the errors are in the 10% where the model is least confident. Then you can review just 10% of your content and take a 2% error rate down to 0.2% error rate.
The principle is that the engines have different failure modes (hopefully) and therefore the 2-3% error rate of each engine is in different areas of the audio. The key underlying assumption is that the events are mutually exclusive.
With 3 engines, you can use something like 2-of-3 stream matches to override the stream that mismatches.
Not having to pause + rewind will save a ton of time for that 3%.
ML systems somewhat notoriously do not necessarily make the same sorts of errors that a human would. And I'd expect a large portion of the errors to be transcribing the wrong words rather that indicating that a word couldn't be transcribed. That sort of error means that you can't really get away with manually reviewing just 3% of the audio.
And there are humans in the loop too, and an enormous amount of redundancy in the questions and answer, so even plausible false transcriptions will get picked up on if they matter. Nobody gets sent to jail simply because the transcription process - human or machine - accidentally substitutes "I did it" in place of "I didn't" midway through a two hour interview.
Humans, however, are not simply metric optimizers. Though it's always in the interest of those corporations producing metric optimizers (i.e. models) to paint humans as such, so their models shine in comparison. They want humans to look like bad machines, so it looks like they should be automated. Not to say they shouldn't in many cases, just that there's a clear one-sidedness in all corporate PR (and funded research, especially that research which is also PR).
All this to say that yes I agree with you. And if we humans don't want our unsustainable economic growth to turn us even more into machines (as our bureaucratic creep has done quite well thus far), we should fight such rhetoric that aims to paint humans simply as machines or task-doers.
1. Set up a computer with voice recognition software that flags certain patterns.
2. Connect computer to voice call communication network.
3. Configure computer to switch between calls every x number of seconds.
Think of it like a system to generate leads for law enforcement that can be integrated with other systems to produce the best quality leads.
>The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.
Besides I wasn't talking about the USA when I said this. I was remembering a conversation I once had with a person who worked as a technician in a telephone exchange.
Power always just finds a way to rationalize what it wants to do.
For entertainment purposes AI transcription is awesome.
For serious business applications the ability to recognize mistakes will continue to be a field to which serious attention is given. It would be interesting to see AI processes double check itself, and also run a logic check on whether the transcription makes sense. So that it can report sections flagged as incongruous or of dubious reliability.
Alignment of video to text is a big problem for me too.
You can even add/modify words that weren't originally there https://www.descript.com/overdub
Would you go as far as to assert machine transcription can be used as an objective benchmark of a speaker’s verbal legibility?
It is fraught with political and interpersonal dynamics to approach someone even privately one on one today and gently suggest their career would get a huge boost if they hired a voice coach to help improve their verbal communication delivery. So even when I don’t directly mention their accent, it becomes a very sensitive subject with many.
However, if audio professionals like you can point to a system and say the raw biomechanics and acoustic physics of the world dictate that this is as physically and psychometrically good as audio parsing of human speech gets regardless whether the system was biologically evolved or ML evolved, the conversation can be couched even more objectively.
I enable recording and voice transcription in every meeting I can (ostensibly for DE&I but really for my own selfish purposes), and already observe in myself I have to work hard to overcome a tendency to gloss over speakers who don’t transcribe well when I review meeting transcripts to jot down any key information I might have missed taking notes upon during the meeting.
Note that I’m perfectly aware that my foreign language verbal skills are nowhere near the English skills of those I have tried to help. If the lingua franca of the coding world switched to Urdu tomorrow, then I’d hire help to learn and polish my spoken Urdu, like I went to a speech coach when learning public speaking because I can always use help in the many skills I lack.
“During his term as President of the United States, Donald Trump made tens of thousands of false or misleading claims. The Washington Post's fact-checker had tallied the number as 30,573 by January 2021, an average of about 21 per day by the end of his presidency.” [1][2][3][4]
I think it’s fair to say there would be a 100 hour long plus video / documentary if they were all compiled into one. lovely!
- [1] Fact Checker (January 20, 2021). "In four years, President Trump made 30,573 false or misleading claims". The Washington Post. Archived from the original on January 20, 2021.
- [2] Kessler, Glenn (January 23, 2021). "Trump made 30,573 false or misleading claims as president. Nearly half came in his final year". The Washington Post. Archived from the original on January 24, 2021. Retrieved January 24, 2021.
- [3] Elfrink, Tim (August 14, 2020). "'Do you regret at all, all the lying you've done?': A reporter's blunt question to Trump goes unanswered". The Washington Post. Retrieved August 14, 2020.