I'm excited to share an experimental study conducted by Pulze.ai to find the most effective approach for classifying prompts given to Language Models (LLMs). Our goal was to improve the overall performance of LLMs. The selected model has now been integrated into the pulze.ai API, providing prompt categorization back to the user.

We tested four different approaches:

1. Classical methods using Term Frequency-Inverse Document Frequency (TF-IDF) with Support Vector Classification (SVC) and Logistic Regression (LG) 2. Sequence classification leveraging BERT and its variants 3. Sentence embeddings with Sentence BERT (SBERT) combined with SVC or LG 4. The SetFit framework that allows end-to-end training of sentence embeddings using Contrastive Learning (CL) and fine-tuning the classification head Our findings suggest that combining pretrained SBERT-based sentence embeddings with SVC provides a robust and efficient solution for prompt classification. Despite the surprisingly effective performance of the classical combination of SVC and TF-IDF, it's suspected to struggle with diverse vocabularies in real-world settings.

The fine-tuned Masked Language Models (MLMs) such as BERT showed mixed performance, with lower suitability for few-shot learning. The SetFit approaches, while promising, appear to need further optimization. We've implemented these findings in the pulze.ai engine with success.

We're planning further research to test these models with diverse real-world datasets and will be sharing more about deploying these models in a production environment in a future post.

We're eager to share these insights with the HackerNews community and look forward to hearing your thoughts and feedback. Please don't hesitate to share your experiences, ideas, or suggestions regarding prompt classification for Language Models.