Ask HN: Gpt4 Terrible at Tabular Data?

1 points by soeptical 3 years ago | 4 comments

I pasted an 11x35 csv table of an employee roster showing daily allocations to different areas.

I then ask some basic queries “where is person y on each day this week”

It gets about 80% of the answers right, but complete randomly gets the remainder utterly wrong. I then tell it what is wrong, and what the right answer should be, and it apologises and updates the answer, and then performs equally poorly for the next employee name.

I have spent all day prompt-tuning, and trying to explain exactly what the table means and what im after, and it reassures me that it knows exactly what it is doing, and then fails again completely.

Seems strange that this thing can allegedly score 90th centile in lsat but completely botch very basic table reading?

joegibbs 3 years ago |

What happens if you convert the table to JSON? Does it perform better having keys for each cell rather than having to guess from the header?

soeptical 3 years ago | |

Will try this thanks!

tobinfekkes 3 years ago |

I also tried to get some very basic results regarding Premier League and Europa League stats (i.e. trophies, years, etc), and it was wrong for 90% of the answers.

You'd think this example is near the best case scenario with easily parsable tabular data published widely and publicly every year, all season. Yet it was comically wrong, and embarrassingly confident that it was not.

PaulHoule 3 years ago |

It's notorious that that kind of system is bad at tabular data. It's a research topic to make similar systems that are good at it. I worked for a startup that was trying something like just around the time BERT came out and transformers were not quite ready.