A Scholarly Markdown

A Scholarly Markdown(scholarlymarkdown.com)

177 points by droque 11 years ago | 96 comments

lorddoig 11 years ago |

And for everything else, there's AsciiDoc.[0]

For an extra five minutes learning you get a boatload (think container ship) more features[1] - it compiles to DocBook: a mature, actually standardised, highly structured format, and from that you get HTML, EPUB, PDF, slideshows, and man pages for free.[0] For math you get MathML, ASCIIMath, and LaTeX (along with a number of ways to render them.) It has a super nice syntax, is equally good at little docs and huge books, and you could theoretically write a proper academic paper in it with the LaTeX backend. And you always know what's going to happen when you try to mix bold and italic...

Also endorsed by Linus.[2]

    [0]: http://www.methods.co.nz/asciidoc/#_overview_and_examples
    [1]: Well, five minutes to be able to do everything Markdown can do; everything else will take a bit longer
    [2]: https://plus.google.com/+LinusTorvalds/posts/X2XVf9Q7MfV (comments)

asgard1024 11 years ago | |

Thanks, this is exactly what I was looking for. A simple syntax for documentation that is extensible with macros. I am glad AsciiDoc includes them, they are very useful.

timtylin 11 years ago | |

I love AsciiBook. Everything but the name. If we ever get an amazing collaborative development tool like Github that has built-in AsciiBook rendering, this project would have been called "Scholarly AsciiBook (but we got Unicode support too I swear really)".

Edit: I was wrong about Github not supporting AsciiDoc

boundlessdreamz 11 years ago | | |

Github does render asciidoc using asciidoctor. What is missing in that that is possible in Markdown?

picks_at_nits 11 years ago | | |

In addition to supporting previews of AsciiDoc files, you’ll find that GitHub also does the “rendered prose diff” trick for commits and pull requests that include changes to AsciiDoc files.

fiddlosopher 11 years ago | |

AsciiDoc is a nice project, but I think that pandoc's variant of Markdown has a lot of advantages over AsciiDoc for academic writing. Let's just compare support for math and citations, for example. Pandoc's citation support is output-format independent: you can write the citations, specify a CSL stylesheet and bibliography, and you'll get the same output in every format pandoc supports. In AsciiDoc, as I understand it, if you want automatic citation support you need to use LaTeX, and then you're limited to output in LaTeX and PDF. Similarly for math: pandoc actually converts your LaTeX math to MathML (for formats that like that) or to native Word equations (for docx); in AsciiDoc, as I understand it, your LaTeX math will work if you target LaTeX, but if you want MathML you need to put MathML in the source. If you want to target both, then maybe you need two different source documents?

I don't have much experience with AsciiDoc, but I've encountered other limitations in writing pandoc's AsciiDoc renderer. For example, I didn't see any way to include multiple paragraphs (or other block content) in an AsciiDoc footnote. That's a deal-breaker for many academics.

I also want to emphasize something that is often not mentioned in comparisons of Markdown and AsciiDoc. As John Gruber emphasizes in his Markdown documentation, Markdown emphasizes ease of reading in source format. AsciiDoc has different priorities, and it sacrifices the readability of the source document to get them. Here's an example involving nested lists, from the AsciiDoc manual:

    1. List item one.
    +
    List item one continued with a second paragraph followed by an
    Indented block.
    +
    .................
    $ ls *.sh
    $ mv *.sh ~/tmp
    .................
    +
    List item continued with a third paragraph.
    
    2. List item two continued with an open block.
    +
    --
    This paragraph is part of the preceding list item.
    
    a. This list is nested and does not require explicit item continuation.
    +
    This paragraph is part of the preceding list item.
    
    b. List item b.
    
    This paragraph belongs to item two of the outer list.
    --

Markdown equivalent:

    1.  List item one.
    
        List item one continued with a second paragraph followed by an
        Indented block.
    
            $ ls *.sh
            $ mv *.sh ~/tmp
    
        List item continued with a third paragraph.
    
    2.  List item two continued with an open block.
    
        This paragraph is part of the preceding list item.
    
        1.  This list is nested and does not require explicit item continuation.
    
            This paragraph is part of the preceding list item.
    
        2. List item b.
    
        This paragraph belongs to item two of the outer list.

Note that the AsciiDoc is at least as easy to write, perhaps easier, because you don't need to worry about indentation. But the Markdown source is more readable; the indentation makes clear the structure of the list in a way that mirrors how it would be displayed in a browser or on the page.

lorddoig 11 years ago | | |

The CommonMark is definitely more readable, but Markdown is just so sparse by design that every time I tried to use it for anything more serious than e.g. a blog post I found myself banging my head against the wall, so it's a tradeoff I'm happy to make - and we're talking about languages whose purpose is to produce readable output, so while a readable source file is definitely a nice thing I think it's hard to make a case for valuing that over functionality. To title a paragraph in markdown you can choose from an <h[1-6]> or a <strong>, that's it, and it's so tightly coupled to HTML (it's more of an HTML generation format like HAML than a document format really) that if you need to do anything remotely complex you're forced to either inline the HTML or bind yourself forever to external tools - in which case the goal of universal readability is defeated anyway.

Markdown's syntax constrains how much it can ever do, whereas AsciiDoc is designed to be extensible, providing macros at the "language" level and underneath that there's the whole DocBook toolchain: it seems to me that the clever move is leverage all this, rather than create yet-another-Markdown (and it's not like the syntax tradeoff is at all dramatic.) I think the footnote issue you mention can be done in block macros, but I'm not sure.

I might be getting a bit utopian here, but what would be really great is if all these markup languages sought to have their canonical implementation in pandoc, which would allow for the standardisation of a set of pass-throughs/filters/annotations for things like equations and citations.

Thanks for pandoc, btw. Just out of interest, is there a reason you haven't attempted an AsciiDoc reader? (Or have you?) Assuming it'd be quite a bit more work than the others, what with macros and so forth.

tjl 11 years ago | | |

Markdown (even this Scholarly Markdown) is a non-starter for me. It doesn't seem to support individual equation numbers for aligned equations. Plus, Scholarly Markdown oddly decided to put them at the beginning of the line rather than after. Why? The equation is more important than the number so it should be first. Plus, it's nice to have the numbers all aligned to the right margin.

You stated about the footnote support being a deal-breaker for many academics, the poor equation support is a deal-breaker for me. Most of what I write is full of equations, especially aligned equations. It's necessary to reference a specific equation that might be on a different page (or far above it on an HTML page).

It looks like Asciidoc has different ways of handling LaTeX equations for non-LaTeX output. It can render a PNG, it can use a Javascript solution, or it can output MathML.

treerex 11 years ago | |

I have yet to see a good stylesheet for AsciiDoc's DocBook output for creating a decent PDF. Do you have recommendations? I love AsciiDoc, but am only using it to generate HTML right now.

boundlessdreamz 11 years ago | | |

As @jauco said try Asciidoctor - http://asciidoctor.org/

In addition to default asciidoctor style, there are other themes available - http://themes.asciidoctor.org/preview/ (See bottom right for theme switcher).

You can also take a look at Pro Git book styles - http://git-scm.com/book/en/v2 which was written in asciidoc - https://github.com/progit/progit2

lorddoig 11 years ago | | |

Unfortunately not, XSL isn't really my thing. I'd probably consider trying to go via LaTeX, but that's because I like the way LaTeX looks out of the box and probably isn't appropriate for most things.

jauco 11 years ago | | |

Try asciidoctor and its stylesheets.

coned88 11 years ago | |

Are there any good wiki softwares that use Asciidoc?

boundlessdreamz 11 years ago | | |

The github wiki software supports asciidoc and many other formats - https://github.com/gollum/gollum

rhythmvs 11 years ago |

John MacFarlane (Pandoc’s author) is heavily committed to the Common Mark standardization effort of Markdown (in fact he’s the principal designer of the Common Mark spec).

Both Common Mark and Pandoc serve different purposes: the first is an initiative to counter fragmentation/balkanization of the Markdown ecosystem and in being so has to reckon with backward compatibility, consensus and adoption. The latter is a document conversion library, which, by design, needs to reckon with interchangeability between formats and may hence be hampered by the lowest common denominator as regards feature support. Internally, Pandoc keeps an Abstract Syntax Tree (accessible in json format), and defaults on its own flavour of Pandoc Markdown (featuring a Markdown superset of content element types).

Both the Common Mark community and John MacFarlane have made it clear their first and foremost focus is on standardization, not so much on extending the feature set. Yet, scholars and technical writers are in dire need for something more heavy-weight than the rather small set of features offered by Common Mark implementations or Pandoc Markdown. Hence the Scholarly Markdown initiative and the scholdoc reference implementation (Pandoc fork).

More on how Scholarly Markdown came about, can be read on the blog of one of it’s pacemakers, Martin Fenner; e.g. http://blog.martinfenner.org/2013/11/17/the-grammar-of-schol..., http://blogs.plos.org/mfenner/2012/12/18/additional-markdown...

_-_-_-_ 11 years ago |

I'm still skimming the documentation so I apologize if the answer is obvious, but does anyone know offhand why this is implemented as a fork of Pandoc? Pandoc already has extended markdown features and the creator of pandoc is very much an academic (http://johnmacfarlane.net/), so is there are a reason why these contributions aren't part of pandoc proper?

ivansavz 11 years ago |

A similar effort of introducing math/figures/refs into markdown is the `softcover markdown` syntax. It's basically markdown, with latex commands allowed:

https://raw.githubusercontent.com/softcover/softcover_book/m...

The beautiful idea in markdown is that it allows you to mix (non-container) HTML tags in with the .md and it just works. Softcover markdown is in the same vein, allowing the more readable markdown for main copy, and intermix LaTeX tags as required. Beautiful if you ask me. Or at least beautifuler than ```math ... ```math.

The "backward compatibility" of ScholarlyMarkdown with basic markdown is a cool feature as many tools/plarforms exist that "support" .md now, but to preview you'll still need something that renders the equations, so strictly speaking ScholarlyMarkdown is a new markup langauge.

ciroduran 11 years ago |

It seems that Markdown is starting to get a lot of disgregated reference implementations. Some months ago some important Markdown users were behind CommonMark (http://commonmark.org/). I'd think that it would be best to join this conversation rather than splintering Markdown even more.

netheril96 11 years ago | |

I'm in fundamental disagreement with the principle behind CommonMark. They prioritize standardization over practical usability, and for that reason people will keep splintering markdown.

For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown. Then, because a lot of people actually need this feature, they still have to patch or extend or plug-in the functionality into any implementation of CommonMark, leading to fragmentation again. And frankly, the reason that I start writing more markdown is precisely because of the syntax highlighting ability. Oh, and to write scientific articles, math formulas are a deal-breaker.

They want to have a standard, unambiguous syntax specification, a suite of comprehensive tests and a cleanly implemented parser. They want to unify the community of markdown users and developers. All of those are commendable goals. But at the end of the day, it doesn't satisfy my needs, so I'd rather use a messy, poorly specified markdown flavor or even just render the markdown with GitHub's service.

fiddlosopher 11 years ago | | |

"For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown."

To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks.

JetSpiegel 11 years ago |

Why? This really feels like cramming a square peg into a round hole. Anytime anyone wants something more complex than simply lightly formatted text they will have to use other tool (LaTeX), so why not just use LaTeX in the first place?

droque 11 years ago | |

The way I see it, the main feature is being able to plug math formulas in, rather than formatting documents. It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics.

mturmon 11 years ago | | |

"It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics."

That's what LaTeX does too. Its markup, generally, is semantic.

Dewie 11 years ago | |

Alternatively you can use org-mode and extend the document with latex where appropriate.

stared 11 years ago |

This thing is certainly needed (vide this discussion: https://hackpad.com/New-scientific-markup-language-utAjFcYuv...), but what would be really convincing is examples (or I am missing them?).

As a side note, a lot of Markdown + LaTeX + Code can be done in IPython Notebook. (Though, there are some things absent, like referencing citations or other equations).

lorddoig 11 years ago | |

I read your link and, per my main comment, I'm pretty sure AsciiDoc answers 99% of your needs.

http://www.methods.co.nz/asciidoc/#_overview_and_examples

stared 11 years ago | | |

Cool! Any examples of using AsciiDoc for math or scientific notes? (I.e. with formulae, references...)

timtylin 11 years ago | |

I've just updated the website to include a link to a repo where I am putting all the examples I have permission to share.

https://github.com/timtylin/scholdoc-examples

More will be added as I tweak the syntax. My hidden goal is to add enough to ScholarlyMarkdown to be able to reproduce around 70% of ArXiv.

onalark 11 years ago | |

Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.

dllu 11 years ago |

I created a similar thing for my personal website:

http://www.dllu.net/programming/dllup/

which handles math using svgtex instead of clientside MathJax for faster rendering. It also compiles to both html5 and LaTeX. However, dllup is overall less polished, missing some features (labelling equations, sections).

samatman 11 years ago |

Interesting to see this come up. I've been migrating a couple projects to babel, the emacs org-mode literate programming style. Starting with the hello-world of babel, my emacs config file.

Babel and Org have clunky syntax but the mode takes care of that. The combination isn't perfect but it's very powerful, in particular the ability to chain several languages together in flexible ways.

Coming up with more ways to combine pretty formatting with syntax highlighting is polishing a pretty smooth surface. I'm dreaming of a really slick syntax and operating environment, like babel if it wasn't tacked onto org. In the meantime, using what we have.

nikdaheratik 11 years ago |

Alot of interesting work, judging from the site, but this still feels like reinventing the wheel to me. In addition to AsciiDoc, there's MultiMarkdown (http://fletcherpenney.net/multimarkdown/features/ feature list is nearly identical), and probably another 3-4 that I haven't come across.

Of course, having spent the past few months writing an app that basically reinvents this same wheel (http://www.eqeditor.com/writer/) I suppose I'm in good company.

nikdaheratik 11 years ago | |

Another thing I'm curious about is how the ScholarlyMarkdown authors would go about trying to get papers formatted in this way to go into a database like PubMed but that's pretty far down the track, and not addressed by any of the other solutions either.

timtylin 11 years ago | | |

The goal is eventually to be able to, like what Authorea is trying to do now, just "slap on" various LaTeX style templates during final rendering (similar to how static blog engines format plain md posts using templates). It's going to be PAINFUL but I think it is eventually doable with enough elbow grease. Markdown is a nice starting point because the features are so spartan that you actually have some hope of finding the "greatest common denominator" document model that can pretty much map injectively to all the major journal-specific LaTeX document-classes.

I'm imagining some kind of online component like ShareLaTeX that will become a clearinghouse for a number of tested and proven conversion paths, and can handle compiling a ScholarlyMarkdown document to different formats. This project won't go anywhere if this can't at least be done for major houses like Elsevier, PNAS, Phys Rev, etc.

xiaq 11 years ago |

If you want to write scientific documents with markdown, beware that pandoc already supports inline TeX (math with $$ and TeX commands starting with \) when doing md -> latex conversion. That has been my setup for quite some time.

j2kun 11 years ago |

If you want me (as an academic) to use ScholarlyMarkdown, then you can't force my collaborators to use ScholarlyMarkdown as a consequence.

There should be a nice ScholarlyMarkdown -> LaTeX cross compiler for starting simple documents in TeX and then sharing with collaborators (pick some obvious defaults or allow a config file to get fancy). But more importantly, if I am joining a project that already has a bunch of LaTeX wizardry going on, I should be able to seamlessly and implicitly edit the text parts in ScholarlyMarkdown without my collaborators knowing.

Can ScholarlyMarkdown do this? If not then I'm not really interested.

bshimmin 11 years ago |

I wonder how long it'll be before Gruber complains about the name.

Skywing 11 years ago |

Part of the beauty of markdown, imo, is its simplicity. If I ever have a question about markdown syntax, I can glance at the original docs and have the answer in less than 30 seconds.

0942v8653 11 years ago |

Looks like an interesting project. I may be missing something, but I think in terms of math especially it would be hard to use Markdown instead of something more robust like LaTeX which supports proper mathematical notation.

Edit: yes, I was missing something. Scholdoc supports MathJax (which has browser, Office, and LaTeX support).

Small issue: The title isn't showing up in Firefox (or Chrome) because there are two title elements and the first one is blank.

mitchi 11 years ago |

I actually like the HTML/CSS solution too. http://thomaspark.me/2015/01/pubcss-formatting-academic-publ...

It's more difficult but I feel like there's a lot more potential.

timtylin 11 years ago | |

Yep, that's the idea. Published work today should be able to take advantage of modern screens and its ability to resize, reflow, and animate.

zzleeper 11 years ago |

This looks very interesting but I have a problem with this. I think MOST of these features could have been build just with clever uses of Pandoc filters. This means that i) it's fully compatible ii) it's very easy to extend

2mur 11 years ago |

Why not asciidoc?

smilekzs 11 years ago |

Around 2 years ago I wrote a patch for chjj/marked that supports MathJax. Despite heated discussion, author never noticed and it wasn't merged. Just a glance at how fragmented this markdown world has become.

kbd 11 years ago |

I'm annoyed that there's still no way to underline in markdown.

samuell 11 years ago |

My spontaneous reaction is: Why don't use the existing MediaWiki syntax? It has gone through the test of years (decades?) of real-life needs, has support for (latex) formulae, everything you might need around images and references etc, one of the most powerful template systems I've ever seen, and the list goes on and on.

Additionally there is a very stable, performant and flexible reference implementation implemented as a web app (mediawiki) with excellent import/export in XML format (and the list goes on ...), versioning with syntax-highlighted diff, etc etc etc.

What have I missed? :)

duskwuff 11 years ago | |

MediaWiki syntax is incredibly grody ('''bold''' / ''italic'', tables, template logic, etc), and has exactly one implementation in common usage (MediaWiki itself -- written in PHP, and horribly convoluted), and has no formal specification. The diff feature you're referencing as a "syntax-highlighted diff" is a simple diff of the source -- it's entirely ignorant of formatting, and the XML export is similarly just an export of the raw source.

Nobody likes MediaWiki syntax, not even its own users. It's awful. The only reason it's still used at Wikipedia is because there's too much content to reasonably convert.

bambax 11 years ago |

> It is a fork of Pandoc and is build upon the same parsing engine.

=> built