Ask HN: What are other "uncrackable benchmarks"?

3 points by Nail2680 28 days ago | 1 comment

I have my own benchmark for AI just based off my Doctoral Thesis, which isn't public(university constraints). I am wondering if other people have other "hidden" benchmarks? I have a few I have made, and one wonder if talking about them devalues them, and two can a good test set be made that can't be leaked?

retrac 28 days ago |

Effective natural language processing of sign languages. Current LLMs are almost completely incapable. And it's a steep hill to climb. No text corpus. Must be learned from video. I do not believe current LLM approaches are capable of this with the amount of training material available, regardless of compute. Though I'm eager to be shown wrong.