Show HN: An LLM purpose built for data annotation, outperforms GPT-3.5

9 points by DhruvaBansal 2 years ago | 1 comment

Try it out here: https://labs.refuel.ai/playground

Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets.

It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction.

spdustin 2 years ago |

It seems to suffer from choice bias, favoring choices that come earlier. If I add “car components” to the end of the list, the last example reliably returns “car”. If I move “car” to the end of the list, the model reliably returns “car components”