Show HN: Microsoft releases Flint, a visualization language for AI agents

Show HN: Microsoft releases Flint, a visualization language for AI agents(microsoft.github.io)

108 points by chenglong-hn 3 hours ago | 46 comments

Data visualizations are the bridge between user and data.

But building AI agents that can generate visualizations reliably can be very tricky:

- simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability

We figured out it is a limitation on the language issue (not just AI capability thing) -- current visualization languages are a bit too low-level for AI agents, requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler. Flint is a visualization intermediate language to address this issue, allow AI agents to solve this last-mile human-agent interaction problem. It provides a simple semantic-type based specification, and contains a layout optimization engine that can produce good-looking charts (filled with derived low-level details) from simple high-level specs. The result is also very human understandable and adaptable. Flint powers data formulator for generating visualizations (another open source project from microsoft https://data-formulator.ai/).

Flint is available open source, and we built a MCP server that you can directly plug flint in your favorite agent app to play with data.

YuechenLi 56 minutes ago |

>Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout.

Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.

Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):

type SemanticType = "Category" | "YearMonth" | "Profit";

type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed

interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }

interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }

interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }

type SemanticTypes = Record<string, SemanticType>;

interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }

// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };

chenglong-hn 49 minutes ago | |

I do find the chartType part is not quite elegant, since templates should be more extensible. We will need to fix that.

For other parts, it's quite common in visualization and diagram etc libraries to have json, since they are easily portable in different rendering contexts.

YuechenLi 5 minutes ago | | |

I'm sure you know that JSON are object literals in TypeScript, there isn't really even a serialization process that's needed there. The AST/IR can still be in JSON, but the authoring surface can be a restricted subset that does not allow logic to execute, that way you can still get the type safety and functionality of TS when you need conditional/loops/logic without throwing away what TS already gives you.

I mean, it would be great if you guys would have like a "TSON" that is basically "JSON with type declaration and comments" from TS, which I think would just solve a lot of problem straight up. JSON itself is just too restrictive and comes with its own bracket verbosity tax.

rbalicki 1 hour ago |

"For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.

wuliwong 1 hour ago | |

Isn't this literally made for AI agents to be accessed through an MCP server? Seems to me the AI agents part of the marketing is important.

bigfishrunning 18 minutes ago | | |

But why be exclusive? Why not "Chart language for computer programs to generate"?

I don't want to use an agent at all, but i wouldn't mind generating some charts with an easy-to-generate markup language...

chenglong-hn 1 hour ago | |

for agent to generate, but also easy for human to edit (especially with UI) :)

cpard 1 hour ago |

There’s an emerging pattern in agentic systems and this project is a great example.

A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.

I feel we will be seeing this more and more in the near future.

pwarner 1 hour ago | |

When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.

ajrouvoet 1 hour ago | |

A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.

jrflo 56 minutes ago |

I don't really understand the point of this, I feel like LLMs have been able to one-shot matplotlib since GPT 3.5. I have extensively used LLMs to do data viz and haven't run into any problems. What is a specific instance where an agent struggles to generate a visualization and Flint solves it?

chenglong-hn 51 minutes ago | |

This comes with a bit of last mile issue, if just chatting with GPT in the chat panel and keep steering it, it's generally fine (as a power user).

But when building it in a tool that serve end users, we are starting to see that a 80% success rate in generating good looking charts can become a big issue. We experienced this when building some data analysis system. So the reliability, expressiveness, and costs (in terms of time and tokens) are hard to achieve all together with directly generating matplotlib, vega-lite etc.

So we essentially designed the langauge as a trade-off across the three, by moving some decisions to the compiler to reduce generation cost while maintain good expressivenss.

jrflo 40 minutes ago | | |

Ah got it, that makes more sense. Thanks for the info!

nrub 1 hour ago |

> simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability

N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?

mbreese 1 hour ago | |

A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.

Or at least, maybe that's the idea?

IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.

chenglong-hn 1 hour ago | |

we are considering also reliability, interactivity besides expressiveness. Simpler spec with good expressiveness comes handy when you want the agent to be reliably for non-expert users and with small models.

kveykva 2 hours ago |

Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/

My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.

chenglong-hn 2 hours ago | |

Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.

Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.

So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.

NicuCalcea 1 hour ago | | |

I'm sorry, but as someone who creates data visualisation as a big part of my job, I wouldn't say the charts on the website look good. Most aren't awful either, but by no means are they an improvement over what I'd get by telling any coding agent to make a chart with Vega-Lite or Observable Plot, and probably worse than if I had some decent instructions/skills.

I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.

theK 2 hours ago |

> requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler

Isnt graphviz there for the same reason?

Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!

simlevesque 33 minutes ago | |

Absolutely. This is DOA honestly and not really better than what we had before.

chenglong-hn 2 hours ago | |

In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).

Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.

theK 2 hours ago | | |

> Json as a common language for human in visualization has been around for a while

Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.

chenglong-hn 3 hours ago |

Project page: https://microsoft.github.io/flint-chart/

MCP setup: https://microsoft.github.io/flint-chart/#/mcp

neomantra 2 hours ago |

This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...

and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.

But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...

https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...

chenglong-hn 2 hours ago | |

This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).

Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45

FailMore 2 hours ago |

The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.

I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.

Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc

Source: https://github.com/espressoplease/smalldocs

giancarlostoro 2 hours ago | |

> mermaid diagrams

I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.

FailMore 2 hours ago | | |

Yep I find them to be very useful for explaining a system.

They can do a lot of cool things! Mermaid gallery here: https://smalldocs.org/s/xZrc-lNW1kbXpoIuU3l_ky#k=-0ehGe2B-hR...

altmanaltman 2 hours ago | |

interesting how you don't discuss literally anything about the project actually posted and spam your thing. Not pointing you out, seen many other comments like this on HN but always felt a bit weird about them

FailMore 1 hour ago | | |

Fair point. Sorry

dvt 2 hours ago |

This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.

Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.

chenglong-hn 2 hours ago | |

That's awesome!

I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!

ietcd 2 hours ago |

https://github.com/scicloj/kindly

but enterprise

santiagobasulto 2 hours ago |

Forget AI agents, this DSL is better even for humans. Cool project!

chenglong-hn 2 hours ago | |

For AI agents and Humans :)

animal_spirits 2 hours ago |

It compiles into Echarts, but echarts already has a JSON co figuration spec

chenglong-hn 2 hours ago | |

It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.

In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!

lzupart 48 minutes ago |

This happens if a company has a CEO who presumably can no longer successfully go to the toilet without AI assistance.

Agents, npm, typescript, MCP. All buzzwords are there. Will anyone look at the slop charts? Of course not, the tokens are the goal.

MSFT stock is at 2024 levels. Maybe someone should produce a flint chart and present the agentic work to Nadella. No one buys this AI slop any more.

chenglong-hn 24 minutes ago | |

I actually made a chart with Flint to show MSFT stock, and with sparkline chart to compare with other companies... :)

bigtechisajoke 54 minutes ago |

Tweakers.

Make something people want.