This shouldn't be ignored in the discussion here:

  The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes."

This is precisely the kind of fine print on white-collar AI capability that companies keep running into: pretty much any non-entry office job worth having involves a lot of undocumented (even undocumentable) problems requiring judgment and experience.

And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"

This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison. I'm going to go out on a limb and guess it's not the LLM.

If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of tax fraud, you're in uncharted waters. In the end, the gun did it, but you were the one holding the gun.

A lot of jobs are like that. You're not as much buying the service as you're buying not having to worry about the service.

Interesting write-up. Having been a bookkeeper a long time ago, I'm not too surprised at this being susceptible to automation by an LLM backed system.

It seems also that the classes of error they encountered could be handled by improved skills/knowledge base access on the fine points of relevant tax legislation.

The important part for their software ofc is, will they take responsibility for the output if HMRC come calling? Without that users are adopting the risk which they may not be keen to do (dealing with HMRC is not fun), with that it could be a very nice saving for a lot of small companies (and bad for the employees of a lot of accountancy firms)

I'd be scared shitless to even try something like this. There is just a pretty website, a video, and a blog post. No info on the founders, I can't find anything on LinkedIn, just a company Vineyard Finance LTD that was incorporated last year.

We're all unhinged about the data we're giving LLMs but here I'd draw the line. I'd rather keep paying the small amount I pay to have my accounts done.

"The VAT return prepared by the model was essentially correct: the most important number in the return, which is how much VAT the company was owed by the tax agency, was off by only 7 pence relative to the human-prepared return."

I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.

The paragraph after this one goes on to explain other mistakes the LLM made? Yikes

> nearly as accurate as a human book keeper

Anything to avoid using the metric system.

Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.

This is interesting to see, but surely any non-trivial business [edit: who needs to file a VAT return] is already entering its invoices into a finance system which can automatically generate a VAT return in a deterministic way.

Unrelated, but I feel it's unfair to rob the word "bookkeeper" of its peculiarity of having three subsequent double letters by inserting a space in the middle.

This doesn't surprise me at all. You can really constrain this problem, give very narrow context, and get pretty reliable and reproducible results.

I've gotten very good results with some vibe-coded deepseek book keeping. https://github.com/traverseda/beansync

Parses emails or other sources, extracts numbers, correlates different transactions, web search, asks questions, stores notes (regex based, very simple).

The hard part is getting good data, I'm sure that lexus nexus or whoever can get API access to my bank account and all my credit cards, but I can't. Email turned out to be the best way for most of my providers. Managed to avoid 2factor auth so far, but it will suck when I need it.

The company I work for, Digits, has been regularly updating our AI-vs-human bookkeeper benchmark. Look at page 8 -- many models are nearly as accurate as a human bookkeeper

https://digits.com/downloads/beyond-the-hype-evaluating-llms...

I just have a folder on my computer where I keep things in beancount. Then I have mercury CLI access with a read token to my business bank account, and I have my emails fully synced in there as well via IMAP. Claude Code with Opus just seamlessly hooks everything up so my accounts are up to date. At the end of the year, I used that information to prepare my tax returns for the business and then later the part that flowed to me as the owner.

I had a fairly complicated tax return in 2025 involving a couple of change of business tax consideration and some money that was accidentally sent to me as a 1099 instead of to the business and I did everything with tax software with Claude Code advising.

The end result was pretty damned good. I was unsurprisingly audited (or at least carefully reviewed) and the only error was in some way where I allocated a small amount of my wife's tax-free disability payments (disability is the mechanism that California uses to provide maternal benefits pay protection; she's not actually disabled). The IRS told me about it, I paid that bit (it was meant to be claimed back from the employer, not the US government) and everything was hunky dory. To be honest, the sum was so small I did not investigate (and haven't yet followed up with getting reimbursed by her employer).

Honestly, almost all of it could have been avoided if I'd paid an accountant and a tax lawyer and they'd told me things and I'd done as they did, but in the end the combination of the fact that the IRS is very reasonable when you explain things and a modern agent means that the entire process was quite simple. In the end, I preferred the interactive mechanism of working with software because most accountants and lawyers will prefer to get all of your documentation all at once and then work on it rather than do it incrementally. In my case, I was able to work on the return incrementally and then have everything plugged in. I could ask a bunch of questions and get clarification.

I think I will probably do all this the same way this year (though of course my taxes will be simpler).

It's not hard to imagine that will be able to do as good a job as a human accountant in the not too distant future.

It's also not hard to imagine tax authorities using AI to audit everyone's tax returns every year.

We sure live in interesting times.

I'm not so interested in having an LLM do my bookkeeping for me. But I'm very interested in whether LLM's can unravel the accounting obfuscations that billionaires use to avoid paying taxes on their wealth.

I can't file "almost" with the IRS. That's not going to end well. Sorry Mr. auditor, this is a non-deterministic filing.

That "nearly" is doing an awful lot of heavy lifting. It doesn't matter if your AI model is 99% or 99.99% accurate. For a tax return it has to be perfect every time or someone is at best getting a fine or at worst going to prison.

Sure, human error happens too, but humans take accountability. That's why accountants are a regulated profession. Until an AI company CEO is willing to go prison if the output of their model is wrong, these tools are worthless.

But don't take my word for it, head on over to Toot's own terms of service https://toot-books.pages.dev/terms#ai-not-advice

    Toot uses automated and AI systems to generate classifications and reconciliation suggestions. Output may be incomplete or wrong and must be reviewed by you.
    Toot is a software tool. It does not provide accounting, tax, legal, audit, or financial advice, and nothing it produces is a substitute for a qualified accountant or tax adviser. You are responsible for checking Output before approving it or relying on it, and for any decision you make based on it. To the extent permitted by law, we are not responsible for outcomes arising from automated Output you approve without review.

Comparing this to a human book keeper is farcical.

Oh, I'm actively doing this at the moment. FreeAgent grabs my transactions from Wise already, and then I give it [Claude Code, in fact] a folder of PDFs to attach to my invoices, including figuring out VAT, and it's uploading what it found using the FreeAgent API. My accountant hasn't complained yet, and it seems considerably more accurate than when my wife was doing it.

Quiet plug for https://github.com/pjlsergeant/byre which I use for all my little projects like this.

> They've done studies, you know. Sixty percent of the time, it works every time.

> If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent.

From my understanding, you are the person signing off on the paperwork that is submitted to the IRS.

There is this cache 22 with taxes.

You are responsible, but you outsource it to a accountant. Because you are not knowledgeable about the taxes. But you are expected to be knowledgeable to understand the tax documents that you submit to the IRS. That is why the accountant always ask you to review the documents and sign them like 20 times.

The same applies when you run a company, depending on the country, you need to prove yourself knowledgeable in accounting, before you are allowed to run a company. Normally that is included in a university degree, but if you have a middle school diploma, you need to do a official examen to get that degree.

Whatever you submit for your company, you are again responsible. Even if you hired a accountant.

So while technically, if a accountant makes gross mistakes, the bill will always fall in your lap, because you are expected to understand the reports you submit to the IRS. And catch any errors before doing so.

With the IRS, the burden of proving your innocents is often put you. Its because the good faith argument can be misused easily. That is why the buck stops at you.

So using a LLM or a accountant, really does not matter. Sure, a accountant can go to jail if there has been major issues (its not going to be with one client issue).

But you can lose your house / company, have your life ruined by whatever you submitted.

“A computer can never be held accountable, therefore a computer must never make a management decision.”

– IBM Training Manual, 1979

https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...

1 Just as with traditional non-ai ways of doing things, there is a deterministic layer that can trigger when things are off. This can range from letting the ai know to stopping before a human confirms.

2 Usually, specially for SMBs, nobody goes to prison over accounting errors. That's because SMB owners make mistakes all the time and, because of that, authorities are fairly practiced in understanding how mistakes look, and how fraud looks.

I agree with you on every point but it is interesting to see real world benchmarks like this. Showing the standard benchmarks that all LLMs use is not only boring but at this point likely gamed or even has issues (according to OpenAI) by every LLM.

Sure! In classic accounting firms you'd typically have one accountant doing the books and then the other reviewing the first accountant's work. Given the model's accuracy (and models will only get better), only by automating the "doing" and keeping a human to do the review part we will already shave huge amount of time and costs without risking the compliance part.

The LLM is, after all, trained on a world with a lot of tax fraud in it.

> This is a prime example of a problem space where accuracy matters, but it also matters who ultimately goes to prison.

I don't disagree that tax fraud is bad but accuracy really doesn't matter 1/10th what people think it does.

Something to keep in mind in the EU with VAT is that there are approximations, errors and then huge frauds like the infamous "carousel fraud". Sadly the carousel fraud is all too common in the EU: this one gets people really sent to jail. But it's not an error a LLM shall make that could lead to a carousel fraud taking place: it's an elaborate scheme.

Now... If you're self-employed and have to pay the VAT (which may or may not be the case depending of the type of self-employment work you do) or if you're a SME and you've got approximations and errors well it's really not a big deal.

Because the public servants in charge of collecting the VAT and verifying that you filled your stuff correctly... Do make shitload of errors too. And then, depending on the country, these public servants have a special privilege: they can decide an amount of VAT+fine (if there's a fine) that you agree with and if you agree with it, their number is the number.

It's everything but a correct number. It's a number they ended up on that is "close enough" (for some definition of "close"). Sometimes by hammering number after number on a little calculator and then ending up saying:

"We ask you to pay 3500 EUR of additional VAT. If you sign here and accept, the IRS (equivalent of the IRS) won't be able to bother you ever again for that year/years".

This notion that, somehow, there's "one" correct number and that public servants paid something like 2 K EUR net per month (which may be the reason why so many are so prompt to ask for bribes btw) can determine it in a few minutes is ridiculous.

It's simply not happening. Usually the accountant doesn't know the exact number. The SME owner doesn't know the real amount of VAT due. And the public servant(s) in charge of a VAT audit, if any takes place, certainly as zero clue as to what the exact VAT number should be.

The rules are way too complicated, there are way too many special cases, and some things are simply extremely hard to take into account (for example during one audit the person told me he disagree with the 90% of 21% VAT deducted for my ISDN line --yup it was a long time ago-- but then I told him it was during a previous TVA audit, a few years before, that the VAT auditor, seen my job, fixed at 90%).

For some stuff you can opt for a fixed number: say a car, there are rules, depending on the country, where you can, say, get back 75% of 50% of the VAT. But people can also, depending on the country, opt-in for the "real usage": you do 12 387 km during the years, how many were really for work? 6 287 km ? Or may 6 275 km?

What if one day while going to work, you took a detour because you want to drop a gift for your grandma's birthday? How were these kilometers accounted for?

Restaurant bills? What if one of the person wasn't work related (your nephew happened to be in town and you had to take car of him and you knew the clients well so they didn't mind: how does the accountant account for that)?

Gifts to client. You got a discount buying 6 packs of bottle of champagne, but not five (no discount if only five), so you bought a sixth one. But that sixth one you kept for you.

Nobody computes those amounts correctly. Nobody.

And don't get me started on the public servants who'll give, during an audit, an unfair advantage to people from their community (whatever that community may be) or those, literally, banging underneath the table while talking about their next vacations: they're telling you an "arrangement" can be made. Basically you give that one person some money (cash bills of course) and, boom, the 20 K EUR of VAT you had to pay suddenly gets lowered down to 5 K EUR.

You refuse? It'd be too bad if those 20 K became 25 K EUR wouldn't it? Mafia-style.

This is all happening. Don't tell me it's not happening because I know people on both sides of these "trades".

Depending on the EU country, it can even be the norm.

As rational people who like determinism (at least I do), at first glance it looks like a SME's accounting is something square but it really ain't. Or let's put it this way: it there's one number that is the really the amount of VAT the SME has to legally pay, I'd guesstimate 0.01% of the SMEs out there pay, to the cent, the proper amount.

All this to say: the approximations, hallucinations and errors of LLMs in accounting for VAT are probably just on par for the course.

P.S: ah yup... The boss (say major airline company) giving a business card and its PIN code and asks to take good care of the clients: and an employee takes them to strip clubs, they all get drunk like crazy, and explodes the expenses on the business card. Way, way, way more than what is OK to put into expenses to entertain clients. Well guess what: I know for a fact this happens too. And I know for a fact this ain't accounted for properly either.

It’s almost impossible to be in a situation where your taxes end up this wrong you’re accused of fraud. I honestly do not believe my accountant does anything better than AI or AI generated code that does calculations to see what you should be paying.