Extracting Steering Vectors from J space(darshanmakwana412.github.io) |
Extracting Steering Vectors from J space(darshanmakwana412.github.io) |
If you provide free chatbot services, but sell advertisers bids on which steering vectors to use to bias towards products, based on an embedding of the prompt, I bet you'd make a ton of money. For example, Coca Cola would bid on prompts about drinks, and bias towards mentioning Coke products.
I wonder if you could also use a similar method to do product placement in GenAI images and videos, and whether ad revenue would be enough to offset the price of generation. Some ad bids can go pretty high...
Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...
Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
https://www.anthropic.com/news/golden-gate-claude
> If you ask this “Golden Gate Claude” how to spend $10, it will recommend using it to drive across the Golden Gate Bridge and pay the toll.
You should even be able to create a j lens from scratch, but it might take a while. I was able to do it in a few hours on an H100. Creating the J lens is basically the equivalent of calculating a few thousand training steps for a model (256,000 backprops in my case). I've got more details in a blog post:
https://blog.lwarfield.dev/layer-scope/
I'm currently at work and can't those matrixes up until I get home. I'll update this comment with a link later.
There is no thinking in these models. The J space is a basic technique measuring how much the influence of shifting a token earlier changes it later. Anthropic can wrap it up in a 100-page paper peppered with language about 'consciousness' and other, but that is basically the gist of the entire method.
All this noise about "thinking" isn't really about what models can do, it's mostly about what what every participant in the conversation privately thinks "thinking" means, but we disagree because we're not all using the word the same way.
So when you admonish someone to say "there is no thinking in these models" while not clearly defining exactly what you mean by "thinking", your assertion that models don't do it are meaningless at best, and false or deceptive at worst.
You can dismiss the criticism with pedantics about what he means by 'thinking', but what I said is very clear if you read the underlying paper.