I've reached a point where tweaking system prompts, tool docstrings, and Pydantic data type definitions no longer improves LLM performance (tool args accuracy and speed). I'm considering a multi-agent setup with smaller fine-tuned models, but I'm concerned about latency and the potential loss of overall context (which was an issue when trying a multi-agent approach with out-of-the-box GPT-4o).

For those experienced with agentic systems, what strategies have you found effective for improving performance? Are smaller fine-tuned models a viable approach, or are there better alternatives?

Currently using GPT-4o with LangChain and Pydantic for structuring data types and examples. I already explain high level tool usage guidelines in the system prompt, extensive descriptions with examples in tool docstrings, and data type definitions with docstrings, field description, and examples using Pydantic BaseModel. The agent has access to five tools of varying complexity, including both data retrieval and operational tasks.

Any input appreciated!