Show HN: MCP Tool Definition Quality Score (TDQS) Spec

Show HN: MCP Tool Definition Quality Score (TDQS) Spec(tdqs.dev)

8 points by punkpeye 2 days ago | 1 comment

Hey everyone,

You may know me because of my Open-Source work like awesome-mcp-servers, FastMCP (node.js), ViteMCP, mcp-proxy, mcp-remote, and a few other projects in the MCP ecosystem, including Glama.

I was lucky enough to be present when MCP was first announced. That let me to contribute to the foundations of this new protocol and everything that has evolved around it. It also let me to be at the center of a lot of feedback, and by far the biggest complaint about the MCP ecosystem has been the inconsistent quality. Quality here means a lot of things, but server JSON definition is a big part of it. Bad tool definitions mean that tools are not selected when they should be, they are when they shouldn't, they are improperly invoked, etc.

TDQS is an open-source specification (https://github.com/glama-ai/tool-definition-quality-score) for evaluating the quality of the MCP server definitions. It's not a complete solution to the quality problem, but it is a research based rubric that increases clarity over what tools are available, what are their behaviors/purpose, and when/how they are supposed to be used.

TDQS is what Glama uses to score 15,000+ Open-Source and remote MCPs. And https://tdqs.dev is a free website to promote the spec and increase the adoption through better documentation and easy to use playground/CLI/API/SDKs.

Would love your feedback and participation in improving the quality of the MCP ecosystem.

nometalalchemis 2 days ago |

I love the TDQS system. I've personally revised my MCPs to improve the quality of the score and make it more accessible for agents when they call tools. Really love the idea behind this and Glama in general.

One point of feedback, when I uploaded a new version of my MCP (specifically to address a C quality score) after 3 days it still hadn't retested. I tried multiple manual resync's as well as trying through the API, and neither I or my agents could figure it out (saying just wait for the backend to do it automatically). What was confusing was the actual MCP page had the latest version and showed the README plus all the latest github release information (including ~70+ new operations), but the quality score test just wouldn't run on the latest version. I even tried rebuilding and releasing, but that was when I realized I had to add credit card info (because the rebuild and release failed. Props to the AI helpdesk though cause that helped me sort it out a bit, saying no credits). I added a few dollars worth of credits and the rebuild worked. A few hours later the quality score jumped to an A.

Even if it didn't require a credit card, neither Fable high (via CLI/API) or I could figure it out over 4 days of searching the documentation, trying just about everything until it worked. And no, I did not add my credit card upon initial account creation, so I admit that might be my fault... but other users may feel the same way. I'm not sure if you can figure out how to improve the documentation or add a blurb near the quality score to help people find the exact steps needed to trigger a retest?

On the other hand, do you have any recommendations for marketing new MCPs? Its really easy to get lost in the sea of existing ones on the glama site or github in general.

Thanks for your hard work on this project. Looking forward to whatever you build next!