OpenAI – vulnerability responsible disclosure(requilence.any.org) |
OpenAI – vulnerability responsible disclosure(requilence.any.org) |
The original report was that submitting a message close to (but not quite) 1500 seconds to the audio transcription API would result in weird, unrelated, off-topic responses that look like they might be replies to someone else’s query. This is not what’s happening. Our API has a bug where if the tokenization of the audio (which is not strictly correlated with the audio length) exceeds a limit, the entire input is truncated, and the model effectively receives a blank query. We’re working with our API team to get this fixed and to produce more useful error messages.
When the model receives an empty query, it generates a response by selecting one random token, then another (which is influenced by the first token), and another, and so on until it has completed a reply. It might seem odd that the responses are coherent, but this is a feature of how all LLM's work - each token that comes before influences the probability for the next token, and so the model generates a response containing words, phrases, code, etc. in a way that appears humanlike but in fact is solely a creation of the model. It’s just that in this case, the output started in a random (but likely) place and the responses were generated without any input. Our text models display the same behavior if you send an empty query, or you can try it yourself by directly sampling an open source model without any inputs.
We took a while to respond to this. Our goal is to provide a reasonable response to reports. If you have found a security vulnerability, we encourage you to report it via our bug bounty program: https://bugcrowd.com/engagements/openai.
It seems like reporting bugs/issues via that program forces you to sign a permanent NDA preventing disclosures after the reported issue been fixed. I'm guessing the author of this disclosure isn't the only one that avoided it because of the NDA. Is that potentially something you can reconsider? Otherwise you'll probably continue to see people disclosing these things publicly and as a OpenAI user it sounds like a troublesome approach.
I believe the author was referring to the standard BugCrowd terms, which as far as I know are themselves fairly common across the various platforms. In my experience we are happy for researchers to publish their work within the normal guidelines you’d expect from a bounty program — it’s something I’ve worked with researchers on without incident.
A lot of AI products straight up have plan text logs available for everyone at the company to view.
I really hope they fix this bug and start taking security more seriously. Trust is everything.
A model like GPT-4o can hallucinated responses that are indistinguishable from real user interactions. This is easy to confirm for yourself: just ask it to make one up.
I’m certainly willing to believe OpenAI leaks real user messages, but this is not proof of that claim.
Right now there is no real proof, untill you confirm that the data it provided cannot be hallucinated (which could be not feisable).
Also, acknowledging the response fron OpenAI staff dismissing it, would you mind sharing PoC?
You can spot anyone using AI writing a mile away. It stopped saying "delve" but started saying stuff like "It's not X–it's Y" and "check out the vibes (string of wacky emoji)" constantly.
If the story in OP about getting a company's private financial data is true (i.e. the numbers are correct and nonpublic) that could be a smoking gun.
Either way it's a bad look for OpenAI to have not responded to this. Even if the resolution turns out to be that these are just hallucinations, it should've been investigated and responded to by now if OpenAI actually care about security.
> I am issuing this limited, non‑technical disclosure:
> No exploit code, proof‑of‑concept, or reproduction steps are included here.
For real? At least doesn't match the one on https://keybase.io/requilence
I could have actually gone to their office in person if I wanted to be pedantic but it actually seemed like a pretty weird office space lol.
The problem I have with it is that there's no way they could have determined if an API key was stolen or not, even to this day.
Basically, their docs (which seemed auto-generated) pointed to a domain they did not own (verified this). So if you ran any API examples you sent your keys to a 3rd party. I know because I did this. There's no way to know that the domain in the docs is simply wrong.
I tried explaining this to the support people, that I needed to talk with a software engineer but they kept stonewalling. I think it was fixed after 24 hours or so.
Massive security bug, well spotted. It's like Bank of America showing other people my transactions, or Meta leaking my WhatsApp messages.
This raises some serious questions about security.
I certainly wouldn't sign an indefinite NDA for a chance to win:
Average payout: $836.36
openai should be grateful, after all, they want all information to be free
Just to avoid yet another case of hallucinations outputs getting misinterpreted.
Software quality is... Minimal now days.
After some hemming and hawing, my most cromulent thought is, having good security posture isn't synonymous with accepting every claim you get from the firehose
Mozilla's program, which has been around longer than most, doesn't. Google and Microsoft don't. Meta and Apple don't.
This is water carrying, intentional or not, for a terrible practice that should be shamed, so that it doesn't become standard.
You can shame it all you want, but you can also just publish your bugs directly. Nobody has to use the Bugcrowd platform. You don't even have to wait 45 days; I don't buy these "CERT/CC" rules.
Even among 3rd party platforms, of which there are several bigs, the NDAs are not a platform requirement, just an option for participating firms.
NDAs are not the norm. Don't mislead people who would otherwise get into this game with non-issues they need not worry over.
All bets are off with small random startups that do bug bounties because they think they're supposed to (most companies should not run bounties). But that's not OpenAI. Dave Aitel works at OpenAI. They're not trying to stiff you.
Simultaneous discovery (either with other researchers or, even more often, with internal assessments) is super common. What's more, you're not going to get any corroboration or context for them (sets up a crazy bad incentive with bounty seekers, who litigate bounty results endlessly). When you get a weird and unfair-seeming response to a bounty from a big tech company, for the sake of your own sanity (and because you'll probably be right), just assume someone internal found the bug before you did, and you reported it in the (sometimes long) window during which they were fixing it.
I felt like it was a huge deal at the time but it’s surprisingly hard to quickly google it.
OpenAI very well may have a bug, but I'm not clear on this part. How do you know the numbers are real?
I understand you know the name is the company is real, but how do you know the numbers are real?
It's way may than anyone should need to do, but the only way I can see someone knowing this is contacting the owners is the company.
Accurate financial data?
How do we know?
What does using not-web-search not having the data have to do with the claim that private chats with the data are being leaked?
???
