Show HN: Jq-Like Tool for Markdown

Show HN: Jq-Like Tool for Markdown(github.com)

325 points by yshavit 1 year ago | 75 comments

There have been a few times I wanted the ability to select some text out of a Markdown doc. For example, a GitHub CI check to ensure that PRs / issues / etc are properly formatted.

This can be done to some extent with regex, but those expressions are brittle and hard to read or edit later. mdq uses a familiar pipe syntax to navigate the Markdown in a structured way.

It's in 0.x because I don't want to fully commit to the syntax being stable, in case real-world testing shows that the syntax needs tweaking. But I think the project is in a pretty good spot overall, and would be interested in feedback!

verdverm 1 year ago |

> GitHub PRs are Markdown documents, and some organizations have specific templates with checklists for all reviewers to complete. Enforcing these often requires ugly regexes that are a pain to write and worse to debug

This is because GitHub is not building the features we need, instead they are putting their energy towards the AI land grab. Bitbucket, by contrast, has a feature where you can block PRs using a checkbox list outside of the description box. There are better ways to solve this first example from OP readme. Cool project, I write mainly MDX these days, would be cool to see support for that dialect

bradgessler 1 year ago | |

Not only is GitHub focused on AI, but they’re also making their UI slower and jankier by rewriting it in React.

I feel like a “Linear for GitHub” is due.

nicce 1 year ago | | |

Is it true that it is React’s fault? Is there any other replacement for heavy user interaction, that is clearly more performative? You cannot do that on server side.

colonial 1 year ago | | |

Holy crap, is that why it's felt so syrupy recently? What was wrong with the old implementation?!

iansinnott 1 year ago | | |

Linear itself has a workflow for Github: https://linear.app/docs/github

whateveracct 1 year ago | | |

Linear? That website makes my laptop heat up like nuts.

sorrythanks 1 year ago | | |

https://pierre.co/

yshavit 1 year ago | |

The Markdown parsing library I'm using supports MDX, so it shouldn't be too difficult to come up with syntax for those components. I haven't done that yet, but mostly because I didn't want to go down that path until I knew there was interest and had a concrete use case or two to inform the query syntax.

If you want to open an enhancement request issue, I'm happy to take a look (PRs also welcome, but not required). If you're not on GitHub, let me know and we can figure out some other way to get the request tracked.

Thanks for taking a look at the project!

verdverm 1 year ago | | |

I don't write rust and already have an MDX toolbox that fits my needs. Browser, GH, and IDE search / TOC are good enough for me.

I'm currently in a phase of trying to shed tools and added complexity, rather than add them

hamandcheese 1 year ago | |

GitHub was ignoring users needs long before the AI craze.

jvanderbot 1 year ago | | |

It's hard to remember, but as soon as gitlab showed up, GitHub went from a "maybe someday if I make it" site to a "let's just use GitHub for everything" site.

Prior to gitlab ratcheting up the usability, features, and cost effectiveness, I preferred hosted git for 99% of use cases.

echelon 1 year ago | |

> This is because GitHub is not building the features we need, instead they are putting their energy towards the AI land grab.

You throw the ball to where it's going. Gitlab might be delivering more value in the short term, but if things wind up looking significantly different in ten years, they might be in for a world of hurt. Innovator's dilemma is real.

It's a danger to ignore the tectonic changes happening. It's also incredibly risky to lean fully in, because we're not sure where the value accrues or which systems are the most important to build. It doesn't seem like foundation models are it.

It's smart to build basic scaffolding, let the first movers make all the expensive mistakes, then integrate the winning approaches into your platform. That requires a lot of energy though.

mtndew4brkfst 1 year ago | | |

> That requires a lot of energy though.

So do the plagiarism machines!

codelion 1 year ago | |

it's a shame when core feature development seems to lag. i've also been working w/ MDX lately & agree that support would be a great addition.

kqr 1 year ago | |

> Bitbucket, by contrast, has a feature where you can block PRs using a checkbox list outside of the description box

I'm not sure this is better. I like the idea of the full context of the PR being available in a small set of relatively standardised fields. Smaller, non-semantic sets are easier to standardise.

verdverm 1 year ago | | |

I'm not sure how having a list of remaining tasks that can be added to adhoc and block PRs from merging is not better than check boxes in markdown description box that are little more than aesthetic...

It has saved us from self induced pain and is a great coordination point

bastardoperator 1 year ago | |

Or maybe GitHub built these features over 10 years ago and millions upon millions of people use them daily without issue. You can literally have any semblance of what you're describing with a PR check, and that feature is also pretty old. The API is right there, you just have to use it.

lanstin 1 year ago |

Ironically one of the reasons markdown (and other text based file formats) were popular because you could use regular find/grep to analyze it, and version control to manage it.

unglaublich 1 year ago |

My flow is to go through the Pandoc JSON AST and then use Jq. This works for other input formats, too.

yshavit 1 year ago | |

I'm curious how ergonomic you find that? I did look at the pandoc JSON initially, and found it fairly awkward to work with. It's a great interchange format, but doesn't seem optimized for either human interaction or scripting. (It's definitely possible to use it for scripting, it just felt cumbersome to me, personally.)

saghm 1 year ago | |

I've never had a need for parsing markdown like this, bit I have to wonder, would it make to go through HTML instead, given that it's what markdown is designed to compile to? At that point, I'd assume there's any number of existing XML tools that work work, and my (maybe naive) assumption is that typical markdown documents would be relatively flat compared to how deeply nested "native" HTML/XML often gets, so it doesn't seem like most queries would require particularly complex XPath to be able to specify.

MathMonkeyMan 1 year ago | | |

I did this for a tool that checks relative links in markdown files, e.g. readmes in a repo.

markdown -> xhtml -> sxml -> logic (racket)

dleeftink 1 year ago |

Kind of aligned with this is MarkdownDB, providing an SQLite backend to your Markdown files [0]. Cool to see this, I feel the structure of .md files is not always equally respected or regarded as a data serialisation target.

[0]: https://markdowndb.com/

broodbucket 1 year ago |

I think you'd benefit of having some more real-world-ish examples in the README, as someone who doesn't intuit what I'd want to use this for.

yshavit 1 year ago | |

That's a great idea, thanks! I'll do that tomorrow or so.

As a preview, two specific cases I've seen:

1) In PRs, some companies like to have semi-structured metadata, like a link to a related ticket under a heading "Ticket". In mdq, you could find that using `# Ticket | [](^https://issues.acme.com/)`

2) Many projects ask people who submit bugs to check off whether they've searched for existing bugs. `- [x] I've looked in the bug tracker for existing bugs`

iJohnDoe 1 year ago | |

Agreed. At least 5 examples of output shown being used against a standard markdown document.

pokstad 1 year ago |

Please don’t reimplement JQ. That problem is already solved. Instead, just provide a tool that can convert your target syntax into JSON, then it can be piped to JQ for querying.

kbd 1 year ago |

Cool thanks for sharing! I'll have to check this out. I've wanted something similar.

After trying a bunch of the usual ones, the only "notes system" I've stuck with is just a directory of markdown files that's automatically committed to git on any change using watchexec.

I've wanted to add a little smarts to it so I could use it to track tasks (eg. sort, prune completed, forward uncomplete tasks over to the next day's journal, collect tasks from "projects", etc.) so I started writing some Rust code using markdown-rs. Then, to round-trip markdown with changes, only the javascript version of the library currently supports serializing github flavored markdown. So then I actually dumped the markdown ast to json from rust and picked it up in js to serialize it for a proof of concept. That's about as far as I got so far. But while markdown-rs saves position information, it doesn't save source token information (like, * and - are both list items) so you can't reliably round-trip.

FWIW, the other thing I was hoping to do was treat markdown documents as trees (based on headings) use an xpath kind of language to pull out sections. Anyway, will check out your code, thanks for posting.

threecheese 1 year ago |

Interesting; one thing you may have learned researching existing tools and libraries: many of them serialize markdown to html before running structured extraction/manipulation - even stuff like converting to pdf.

The core assumption here is that Markdown was/is designed to be serializeable to html - this is why a markdown document/AST is mostly not a tree structure, for tree-ish elements such as sub-sections. Instead, it is flat, an array of elements in order of appearance in the document. Apparently this most closely matches the structure of html, at both the block and inline levels. Only Lists and Blockquotes (afair) support nesting.

Ex: h1 -> paragraph -> h2 -> paragraph is not nested, it is an array of four ordered elements.

Anyway, you might throw a task at Cursor or Copilot to see how an equivalent implementation using html fares against your test suite, you may be able to develop more quickly.

aqueueaqueue 1 year ago |

Why not MD -> json, then use jq! That would be half a static site generator there!

spiffyk 1 year ago |

Thanks for sharing! No immediate use-case for me right now, but good to know something like this exists.

I wanted to point out little nitpicks for the documented shell invocations:

    cat example.md | mdq '# usage'

This can be changed into a stdin file redirect to avoid invoking an extra `cat` process (see Useless use of cat [1]):

    mdq '# usage' < example.md

In a similar fashion, you can avoid an extra `echo` process here:

    echo "$ISSUE_TEXT" | mdq -q '- [x] I have searched for existing issues'

by changing to this:

    mdq -q '- [x] I have searched for existing issues' <<< "$ISSUE_TEXT"

[1]: https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat

mzl 1 year ago | |

From the linked web-page

    A cat written with UUOC might still be preferred for readability reasons, as reading a piped stream left-to-right might be easier to conceptualize.[14] Also, one wrong use of the redirection symbol > instead of < (often adjacent on keyboards) may permanently delete the content of a file, in other words clobbering, and one way to avoid this is to use cat with pipes.

I personally think the cat-style is easier to read since it only uses commands and pipes, with no need to keep track of redirection directions.

twinkjock 1 year ago |

Thanks for sharing this Yuval! Thanks as well for using permissive licenses so I can use this at work.

imglorp 1 year ago | |

Curious, which license can't you use at work for a simple shell tool? Considering you're not linking against it, even GPL3 should be okay, right?

twinkjock 1 year ago | | |

Most big employers consider GPL unusable and will only allow Apache, MIT, BSD, and other permissive patent-free licenses.

frankfrank13 1 year ago |

I worked on a project converting word docs to markdown so they could more easily be ingested into an LLM, one issue was that context windows used to be very short, so we would basically split on `\n#` to get sections, but this turns into a whole thing where you have to make guesses about which header level is appropriate to split at, and then you turn each section into a separate chunk in FAISS. Anyways we ended up using HTML instead of MD but theres so much tooling for traversing HTML and not MD. This would have been helpful for that

foo42 1 year ago |

This is one of those moments where you come across a tool _just_ at the right moment. I have a task for which this will be perfect

infogulch 1 year ago |

I've always wanted a "literate programming" / jupyter-style notebook based on markdown. Maybe this could help make something like that possible.

arminiusreturns 1 year ago | |

Emacs org-mode and babel are what I use instead. Very powerful alternative to Jupyter imho, but of course not for everyone.

I have used commit hooks so that non-emacs users could push updates that still get executed to avoid the emacs tie in though, the biggest issue with it I've found.

zerkten 1 year ago |

Thanks! I have to grapple with some markdown across multiple repos and this'll be a helpful tool in the toolchest.

linklater12 1 year ago |

congrats on your tool, will check it out. I have a side question on markdown: cursor messes up markdown generation quite often for me. I think its responses are always in markdown with sections for code and asking it to generate markdown breaks it. So the question: any ideas on how to have cursor generate markdown?

nodesocket 1 year ago |

How is it parsing? Just normal string and regex matching or transforming markdown to an intermediate structured language?

yshavit 1 year ago | |

For the markdown, I'm using https://github.com/wooorm/markdown-rs, which is a formal parser that produces an AST. For the query language, I have a very simple hand-rolled parser.

dcreater 1 year ago |

What purpose does this serve that grep doesn't?

moonshotideas 1 year ago |

Love this! One persons opinion - I’d change it to mq - less chars are always better for command