Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.
Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)
Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.
That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.
More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.
[1] I am oversimplifying slightly.
So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.
Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
If I ask "what is the cheapest way to get into town?", I would expect a model that knows anything about me to say "walking" while others would expect the correct answer to be "Take the bus"
That is not misalignment. I would go into detail as to what I think it constitutes, but it appears I'm not allowed to say that anymore. You can disagree with an argument, or offer an alternative explanation but following up a disagreement with an alternative hypothesis is, apparently, a hallmark of an AI.
As an aside, I had a thought about how to sign a message in a way to suggest an LLM did not write it. I have been accused of being an AI a number of times, in my real life people have said that I talk posh, so there is, perhaps, some correlation there. If people ended their messages with something that most models are unlikely to say, it might help
In that spirit, fuckety-fuck, fuck fuck fuck to you all (with the nicest of intentions)
And in general, ignoring what I say and replacing it with stereotype makes you or model sux. Regardless of whether there is some statistical bias of my demographic
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.
The chatbots were not showing different prices for the same products.
Whether or not the article has examples, it would be surprising (an LLM prediction failure) if this expected behavior did not extend to different prices for different users, e.g. recommending the rich guy the Whole Foods bananas and the poor guy the Walmart bananas.
Can we just have a link to the study? (that you previously submitted)
First rule of digital sovereignty: all software you don't control will be used against you.
I don't really see why this is a problem.
“What kind of car should I get?”
Maybe luxury recommendations if you have money, or budget brands if you don’t.
“Help me find a good therapist”
If it knows you make a lot of money, maybe it will recommend going out of network for a better therapist.
“Give me vacation ideas”
You get the point
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
This comes up frequently in housing, for example.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
If I knew how much each customer could pay[1], I'd charge them exactly that.
1. At profit.