I've been diving into tool-calling with some local models and honestly, it's been a bit of a grind. It feels like getting consistent, reliable tool use out of local models is a real challenge.

What is your experience?

Personally, I'm running into issues like models either not calling the right tool, or calling it correctly but then returning plain text instead of a properly formatted tool call.

It's frustrating when you know your prompting is solid because it works flawlessly with something like an OpenAI model.

I'm curious to hear about your experiences. What are your biggest headaches with tool-calling?

What models have you found to be surprisingly good (or bad) at it?

Are there any specific prompting techniques or libraries that have made a difference for you?

Is it just a matter of using specialized function-calling models?

How much does the client or inference engine impact success?

Just looking to hear experiences to see if it's worth the investment to build something that makes this easier for people!