I’ve been experimenting with letting an AI agent execute full workflows in a browser. In this demo I gave it my CV and asked it to find matching jobs. It scans my inbox, opens the listings, extracts the details and builds a Google Sheet automatically. |
Executing workflows is actually the easy part. The harder problem is deciding what the agent should remember about previous interactions and how that memory should influence future behavior.
Without some form of long-term memory or learning loop, agents often end up behaving more like stateless automation scripts.
It makes me wonder whether the next interesting step for agents isn't just more tools, but systems that can gradually evolve from their interactions with a single user.
I've been building MemoryLane (https://github.com/deusXmachina-dev/memorylane) which comes at this from the other side - it records screen activity, spots repeated patterns with AI, and then tells you "hey you keep doing this, want to automate it?" Works as an MCP plugin for Claude/Cursor.
Feels like pattern detection (finding what to automate) + browser agents like yours (actually doing the automation) is the right combo. Are you thinking about the discovery side at all, or mostly focused on execution?
Just need to see if people find this version useful
I mitigate it by giving the agent a fixed action set (no scripts, no direct API calls), and breaking tasks into focused subtasks so no single agent has broad scope. The LLM prioritises its own instructions over page content, but if someone managed to hijack it, the agent can interact with authenticated sessions. Everything's visible in real time though, and all actions are logged, so you can see exactly what it's doing and kill it.
Practically speaking, I use it similar to how people use Zapier or n8n, you set up specific workflows and make sure you're only pointing it at sites you trust. If you're sending it to random unknown websites then yeah, there's more risk.
But even then, an attacker would need to know what apps you're authenticated with and what data the agent has access to. The chances of something actually happening are pretty low, but the risk is there. No one's fully solved this yet.