JSON Canvas – An open file format for infinite canvas data(jsoncanvas.org) |
JSON Canvas – An open file format for infinite canvas data(jsoncanvas.org) |
BUT, I imagine this being more useful for creating non-frontend tools. For instance a server that returns subsets of nodes for a particular viewport. Or something that may index nodes, or produce search results. Or tools that simply generates canvases from other data as a one-way operation.
Braintree payments did this well when they were still a startup. They had to collaborate with their direct competitors to create a standard for the entire POS induustry.
What prevents us from doing this here as well?
Are there other solutions that take JSON and do the same thing just as easily?
I've been looking for a good equivalent to QGraphicsView and HTML Canvas is not it (HTML Canvas is a raster image, QGraphicsView is a size-independent fixed-sized, scrollable canvas with indexed objects).
To take this specific example, some of it feels very similar to HTML. label, links, sections, groups, anchors, background fill options...I would have been tempted to define as a subset of HTML that is supported. Then if I wanted JSON, say how JSON maps to HTML. voila suddenly everything is standardly named and creatable. This means backgroundStyle is replaced by background-size = cover or contain. It means that those six preset colors are replaced by all HTML standard colors. Voila no one needs to learn different concepts or definitions. Try that existing standard, try 2 more and only if they don't work invent a new one. Please. I say this as someone that inherited standards invented by teams that I then had to try and train hundreds of users on. Funnily enough the previous people left when the coding was done, without teaching the users. THey probably left to implement version 2 elsewhere. :)
https://obsidian.md/blog/json-canvas/
We just released it today, so this is still a very nascent project. A little over a year ago we released Obsidian Canvas. The .canvas file format has felt stable enough to give it a name and resources that other apps can freely use. See the original Show HN:
https://news.ycombinator.com/item?id=34066824
The spec is conservative, and definitely does not support many features canvas apps will want to implement (yet).
The purpose of giving JSON Canvas a name and site is to encourage an interoperable ecosystem to grow around this format. We're definitely looking for feedback of all kinds!
It's great to see all the suggestions already shared in this thread because it starts to provide a roadmap for how this could become a more useful format for other apps.
Which existing formats were considered before building your own, e.g. SVG/Excalidraw/draw.io/...?
I built a library at AWS for a general canvas editor called Diagram Maker. It recently got archived so I stood up a fork here: https://github.com/sameergoyal/diagrammer and the data format we use is strikingly similar. Check it out here: https://sameergoyal.github.io/diagrammer/?path=/docs/docs-us.... The key differences are panels, workspace & editor.
I dont actively work on the project outside of bugs, but maybe there are ways to collaborate here, like moving my project to use & extend the JSON canvas spec.
JSON Canvas seems to follow in that spirit by being very lightweight, so a lot of implementation details (i.e. how are files rendered, what file formats are supported, etc), edit tags, etc) are left open to implementation.
Markdown and JSON are meant to be non-opague file formats that prioritizes portability and human readability over other features. An application format like Sqlite has a lot of benefits over markdown, but it loses the benefits of text based formats like being compatible with git and is less portable.
What I would like to see is a convention for extending the node and edge definitions, similar to frontmatter in markdown files- something that is not required for basic rendering but is a nice-to-have for applications to consume) - that way portability between apps of varying complexity can be maximized while still allowing for more complex features that some apps might implement. Markdown has the benefit of supporting extensions (for example like tables in GFM) - apps that are not compatible can still render the unsupported markup. But there should be an explicit way to extend open JSON formats.
Some feedback off the top of my head and from reading the comments:
1. *Specifying node ordering*. Obsidian seems to just send whatever is last touched to the top, but this makes a common feature in many apps (move up/down/back/front) more difficult to impement
2.*More explicit definition for line shape*. Adding a way to "bend" a line in a specific way. Useful for creating more complex diagrams.
3. *Relations between nodes*. Group nodes contain child nodes, but the spec doesn't specify how the child nodes are defined. I would expect it to have a `children` property to nest nodes. Obsidian seems to implicitly link nodes to groups based on whether their bounds intersect. This makes it difficult to implement some common features:
a. nodes that exist outside of the bounds of its group, for example a node that "floats" just outside of the edge of the group's borders.
b. nodes that are not part of a group even though it exists within the bounds of that group.
There are many different ways for a canvas app to extend the spec to implement those features, but it seems like something that should be defined in the spec to maximize portability
4. *Extensibility.* Either explicitly support or provide a standard for defining more styles for nodes and edges, such as stroke width, stroke style, rotation, etc. It seems like "color" should be a part of this as well, rather than being an explicit property of a node.
5. *Embeds.* Supporting "embeds" as a node type. I even think the "file" node should be redefined as `embed` with a `uri` property to support different schemes (`file://`, `oembed://`, `https://`) and maybe a `mime-type` (`text/markdown`, `image/webp`). The file node's "subpath" property seems to be only relevant for markdown files, so I think that should be an extension rather than an explicitly defined.
6.*YAML* :) (Should just seemlessly convert from json, but yaml is more readable than json)
Being able to design standards that evolve over time and making tough decisions about what what to make explicit and what to leave implicit is a skill I want to improve on as a developer this year. Does anyone have any resource recommendations or best practices to recommend for me to research?
Please don't, it has one of the most confusing syntax out there with lists and maps, and it won't do well for parsing.
I don't think human readability is a critical feature of JSON at this point. If that's your priority, you can use YAML. Readable JSON is nice because for small files you can read or edit small sections of it, and it's easy to debug when manipulating it with machine code. But there are plenty of cases where a huge JSON file is still useful even if it's barely human readable.
My heuristic has always been: use YAML if you expect humans to create the file (or maintain large chunks of it), otherwise use JSON. For example, Kubernetes config is YAML because humans create it from scratch, and it would suck to do that with JSON. Whereas package.json is JSON because machine code initializes it and humans only make minor edits to specific fields.
In the case of this canvas format, I wouldn't expect humans to create the file from scratch, so use JSON over YAML. Then the question is, will humans even care about reading the raw JSON? Probably not. So why not use something like SQLite or Protobuf? The most compelling reason would be that humans writing code to interface with the format can use parsing tools from their language's standard library.
The purpose of a spec is to specify, and if you don’t specify and leave things open to interpretation, then that completely defeats the purpose.
Anybody who’s worked with a poorly defined spec knows exactly how bad this can be. A good example would be the shambles that is the HL7 spec used in healthcare.
A former colleague had a phrase for this: “once you’ve seen one HL7 message… you’ve seen one HL7 message”. Which really highlights the issue of a standard that’s open to interpretation.
The issues raised (in the comments here) seem to hint at a lack of specificity. That is something that they should really look at improving.
I think overall any group that tries to come up with a standard that can unify a field should be lauded and supported. But perhaps calling this a 0.1 release, and taking the feedback on board, would be the best way forward.
What Markdown got right was creating a nicely readable lightweight markup syntax.
And Markdown also demonstrated how to create a bad precedence for future consolidation by being so loosey goosey and underspecified (and with a bad reference implementation). That there is a Commonmark at all is solely because of others picking up the slack and doing the unthankful gruntwork of creating 100 if-then-else statements in a semi-formal prose format.
- How do coordinates work? Does +Y point up (OpenGL) or down (web)? Is the origin meaningful? What are the units - how does this interact with font sizes? High-DPI displays?
- What’s the difference between a file node and a URL node linking to file://./? Are files supposed to be transcluded? What filetypes are allowed? The home page seems to have an image — was this done using a file node or a markdown node with an <img /> element?
- What HTML tags are allowable in markdown? Is JavaScript allowed?
- Why does the group node allow a background image? If both image and color are specified, which takes precedence? How are children of the group specified?
A couple feature requests for extensibility or interoperability with Excalidraw and TLDRaw and friends: drawings / scribbles, predefined shapes like circle or rectangles, ability to specify fill style, edge width, transparency, ability to embed images, more detailed placement for connector start/end points, etc.
This strikes me as exactly the type of application data that would benefit from being represented in SQLite. Of course, JSON is a `JSON.parse` away, but now you're building your own...everything else. Storage/retrieval, validation, querying/filtering/indexing, graph traversal, etc. It's all yours.
There's so many benefits to building this kind of thing in SQLite. You get data validation, atomic transactions in memory and on disk, a high-level query interface, lazy loading (i.e., only load nodes at most 2 edges away), triggers (when I delete this node, automatically delete edges that point to it), and a stable on-disk format, to say nothing of the fact that SQLite itself is just about the most stable software there is.
By the way, no disrespect to JSON Canvas, it looks like good work, just trying to offer the perspective of someone who has done stuff like this in the past.
Sure, they could screw me over and start charging absurd amounts of money for their app, but high quality open source alternatives would pop up immediately.
Meanwhile, as long as they don't screw me over, it's unlikely an open source alternative is going to be able to catch up to a profitable business that keeps their users happy.
It's an interesting approach, focused on incentive alignment, which is the best way to ensure quality long term.
file (required, string) is the path to the file within the system.
What kind of path, within what system? It's not clear that the 'file' type couldn't just be another kind of 'link'. If various fields like 'background' were defined to be URLs, that would offload a lot of complexity onto existing web specs.
> cover fills the entire width and height of the node.
Does that work like the CSS background-size: cover; or background-size: 100% 100%;?
> ratio maintains the aspect ratio of the background image.
Does that mean CSS cover? contain? Something else?
Colors can be specified in hex format, e.g. "#FFFFFF". Six preset colors exist, mapped to the following numbers:
1 red
2 orange
3 yellow
4 green
5 cyan
6 purple
Maybe the file format isn't meant to reproduce the exact same look in different software, but merely communicate user intent. Your guess is as good as mine.
It looks really promising though! I'm definitely interested in seeing this grow.
But to be a total downer, this spec looks like an extremely rudimentary graph file format, of which there are already like a hundred and all define more visual aspects than this spec.
Also, this is not very json-ish, but optimizing your serializer so your metadata is always written first is pretty handy for embedding, since it allows you to use a pull parser and do useful things before the entire doc is parsed. (e.g. picture a huge doc being embedded and starting out as just a box and having its elements filled in async. You can't do that well if you don't know the bounding box ahead of time)
I don't think this really has much to do with the "infinite" nature of the canvas. Individual instances of infinite canvases have a finite size. :)
Summary: "node: { type: ..., x/y/color }; edge: { from/to: ..., color/label/... }"
Refreshingly simple, especially paired with their "gif of usage": https://obsidian.md/canvas
UPDATE: Figured out how to create one:
1. Install and then open Obsidian
2. Click the "Create new canvas" icon - third down of the icons on the left
3. Add some stuff to the canvas - I double clicked to create a few boxes, put some text in them and then dragged lines between them
4. In the ... menu on the top right click "Reveal in Finder"
You can then open the file it reveals in a text editor to see the JSON Canvas format.
https://github.com/obsidianmd/jsoncanvas/blob/main/sample.ca...
If I'm reading between the lines, this is only supported by Obsidian (as it's by Obsidian)? Considering the complexities and 'malleability' of infinite canvas tools, it would have been prudent to have involved or approached some of the major players in this space, like Excalidraw, Draw.io, Microsoft, Figma. Or at least started at version 0.1 and once it gained a wider consensus, release 1.0.
Besides Argdown, Markdown itself provides some built-in features that can be utilised to construct graphs. Coordinates for instance, can be stored as HTML comments or link alt texts, e.g. `[node](# ("x:25,y:50"))`. Edge, shape types and other data could similarly be stored in alt text fields as serialised JSON or in separate blocks using link reference definitions. [1]
One step further, Markdown lists could be used to store subtrees while cycles as Obsidian block links. This also allows you to encode ancestral, sibling and descendant relations:
- [root](# ("x:25,y:50"))
- leaf
- [link](#^id)
- another leaf^id
[0]: https://argdown.org/
[1]: https://github.github.com/gfm/#link-reference-definition
As far as the spec, I don't really like the idea of forcing well-known types for the nodes. A generic spec should allow for entirely generic nodes that can represent themselves to consuming functions with a 'type' property as a key, as well as arbitrary data types linked to arbitrary nodes. For instance: one of my use cases is an 'addition' node, which would take two number values and produce a number value. This node would also use an entrance execution pin as well as an exit execution pin.
If the spec were to include a 'pin' data type and capture the type keys and labels for pins, those pins could be stored as a list on the node. Then, the type property could just tell the executing context how to route the node data and the pin properties would bring type safety to the functional inputs passed to the mapped function.
Anyway, I assume all of that is out of scope for initial offerings, but that's my two cents on a generalized node spec. Regardless, thanks again for the sweet, simplistic node graph implementation!
This is instead about whiteboard-style graphs. Which is useful, but I find the branding "An open file format for infinite canvas data" to be confusing. Nothing in it implies whiteboards or graphs to me. The fact that the canvas is infinite doesn't even have an obvious influence on the file format, apart from the absence of a canvas size property.
Node: only id, x, y would be necessary. This would allow for point nodes. We could even imagine letting go of x and y to signify that the position of the node is not fixed and could be recomputed in real time by the program.
Even ids could be optional, why require them if they are not referenced?
An added bonus of having point nodes is that you get freehand drawings for free: every stroke is a series of connected dots. Maybe it is an anti-feature though, depending on your vision.
not a great experience on mobile (iOS/safari) - pinch to zoom and drag to scroll are both a bit busted, the zoom ui controls can disappear altogether, and there’s no ui indication that the box on the right, that contains the details of the spec, is actually a scrollable container (ie no scroll bar visible)
The items on our infinite canvas are more akin to a desktop app, where most objects are application windows.
If you click "Toggle output" on the linked page, you can see the code for the page itself
It has basically changed the way I interact with LLMs and research and plan things
Do you plan to make the TypeScript definition part of this new site?
https://github.com/obsidianmd/obsidian-api/blob/master/canva...
For me it's easier to read TS format.
I've stayed away from their Canvas feature largely because it is.. not that. Not because the Obsidian developers have kept it locked down in some crazy proprietary format, but like a JSON file representing the canvas is pretty useless without something to interpret it and these days Obsidian is still the only implementation.
So I kind of hope this takes off. Having a second source available would make me feel a lot more comfortable trying out the canvas feature.
- I'd like to see the `file` node have a mime type field.
- Why limit links to URLs, as opposed to URIs?
- You might want to put a version field in the top level!
https://youtu.be/GblI7GI0jQ4?t=85
(Alas, the original desktop app (iMapping, in Java) as shown in the video is no longer being developed, and now they are only developing the web app (Infinitymaps) only run in the browser, which imo is not the best fit for infinite canvas apps which can be very resource-intensive)
Also see the app author's own all-purpose mega-map that he used to organize everything: https://youtu.be/bTQWL5wmdZY?si=6VrnPIErOzasisEe
I agree with you that I don't think it maps well to the mental model of the brain. Seeing the youtube video link in the sibling comment, when the user is completely zoomed out and can see everything, I just feel overwhelmed looking at it all. Maybe it's the nature of the boxes having different scales that you can't compare them as easily to each other compared to just a regular canvas where things just place in two dimensions. Each time you zoom into a canvas, the transition causes a lost sense of place and space, akin to walking into a doorway to a room and wondering why you walked there to begin with.
I assume no.
> What are the units
Arbitrary. The website says pixels, but the demo lets you zoom in and out, so I think defining the unit as pixels is pretty meaningless, except as a hint to the viewer of the initial scale for the canvas. Even then I can see good reasons for a viewer program to just ignore that and use whatever initial scale allows everything to fit on screen.
> how does this interact with font sizes?
Seems like the font size can't be changed, but I'd imagine it's a specific number of units.
> High-DPI displays?
Not sure what kind of answer you're looking for here. You can just scale everything, so support for High-DPI displays would be up to the viewer program.
I assume yes, as it makes placement of things much easier if they're all relative to a known origin. The limits of the canvas would be "realistically infinite" in all directions though I assume...
If using integers, I'd cap it at 2^53 to align with js's max safe integer which I think is just a double.
This is a data format. This is an abstract data model.
So the spec doesn't specify rendering.
Rendering could however be specified separately (and its reference implementation is Obsidian).
Interacting with SQLite from different programming languages is easier than most other formats, but you still need a SQLite binding. They're available for every language but that's still a not-completely-trivial dependency.
I expect most tools that people build against JSON Canvas will run in a web browser. Adding SQLite as a dependency means you need SQLite running in WebAssembly - totally possible, and even officially supported these days (the SQLite team run their own WASM builds now) but still a sizable piece of extra complexity over just using JSON.parse(...)
Also: SQLite files aren't very easy to diff, so they're not great for collaboration in version control. JSON is better for that.
I'm 100% with you on the schema changes and versioning challenge. The best way to address this IMO would be for the spec to include a top-level "version": key which indicates the version of the spec that a file was created against.
Handled carefully and introduced right at the start of the project this could ensure an ecosystem grows up around the standard such that older spec versions can always be opened by newer implementations, and any implementation can fail-fast if it is given a file that it doesn't yet know how to handle.
> Also: SQLite files aren't very easy to diff, so they're not great for collaboration in version control. JSON is better for that.
Yeah, if we're talking about diffing the literal file itself, then that changes things. At that point, we're not just talking about a storage format, we're talking about interchange as well. In that case I'd ask - of course this is application specific, not general - what data are we putting on the wire? Where does that data live?
For example, if the main state state lives in a your browser instance, and you ship updates (i.e., "CREATE", "EDIT", "DELETE" or some such) back and forth between collaborators, then diffing the state of whatever you have is fairly easy, SQLite or JSON or whatever else. But if we're shipping the actual file itself over the wire or attempting to version control it, then you're absolutely right and diffing the SQLite file is inferior.
There are some interesting tradeoffs in this space. This is a fun discussion!
Adding C sqlite to a golang project adds a significant hit to build times and cross-compilation/static linking complexity[0]. When I looked into the native Go implementations of sqlite I came away with the feeling it wasn't worth the tradeoffs compared to using the C version, but now I still have to deal with the issues above.
I haven't looked deeply into how sqlite works, but my instincts tell me the reason we don't have high quality implementations in every language is because it's actually too complex to treat as a protocol.
I would love to see something fill the void between plain text and sqlite.
I agree that going from text to sqlite is a bit of a hurdle, especially if you're not writing C :)
If you want to improve on JSON, you would have to go into an other direction. Maybe something like postgis would be helpful for extremely large canvasses.
JSON Schema is pretty powerful by the way. Checkout the documentation. SQLite is absolutely no match there. What Sqlite could bring is speed, but I dont see how in the contxt of canvas it would be of any help here.
There is a somewhat recent STRICT mode that strengthens them: https://www.sqlite.org/stricttables.html
SQLite as an Application File Format, https://sqlite.org/appfileformat.html
See Mattermost for an example of a similarly positioned product that is fully open source.
If it was open source that would be more likely to happen
Sample output (should be viewable since I put `#public` at top of the document): https://bigasterisk.com/vault/esp%20cams.md
Server code: https://bigasterisk.com/code/vaulterrific/files/tip/
In-fact, I don’t really like Obsidian on Mobile, so I use iA-Writer to edit/view the Markdown files that I managed with Obsidian on the Desktop.
Then, conforming implementations could render any document just by following the instructions, while editors that actually understand them can provide their own high-level control.
The trick is keeping it editable, which postscript doesn’t do well.
Example: if the language is strong enough to, say, implement force-directed node layout, an editor that doesn’t understand it could still add nodes and they would move around according to the document author’s wishes whereas perhaps the original editor might have more powerful editing capabilities.
I like to think of it like the unist ecosystem for ASTs. Unist provides a baseline spec that compatible tools can use to comb an AST. Then, specific AST tools like hast for HTML or sast for CSS/SCSS can add their own metadata on top.
I'm imagining an ecosystem of "adapters" that would help you translate some of the metadata across providers.
[1] https://twitter.com/KinopioClub/status/1768037179400331692?t...
Wow you have kinda lost the plot on a few things.
JSON was designed to be human readable and writable. YAML was designed to be a human readable format for the automated interchange of data between automated systems. Human writability was neither a goal for YAML nor its intended use. Like everyone else on the frakking planet, you’ve misunderstood what YAML was intended and designed for. YAML was never intended for human-written configuration storage, which is what everyone used it for the instant after they became aware of it.
YAML can bite you very hard if you misunderstand it. JSON is simply invalid if you misunderstand it when writing it.
If you don’t need human readability, use a binary format. Binary formats are so freaking fast compared to literally any structured text format, past, present, or future. High speed and low latency matter and binary formats make both of those easier.
If you need to inspect the binary data, write a viewer using the code you use to read it. It’s a lot simpler than people believe it to be. I find Protobuf to be more of a hassle than writing the code myself, and protobuf is very easy to use, and I’m quite a moron. Binary stuff is not hard.
Though yes, they could have explicitly defined a z-index or defined a convention on how the ordering should work (first nodes top and last nodes bottom or vice versa?). It's interesting to think about the trade offs between explicitly defining these things vs. leaving the application to implicitly make the choice. JSON Canvas seems to be designed to use in tandem with markdown files in a note taking app, so it makes sense why they opted for the more implicit design to be similar to markdown
> Nodes are placed in the array in ascending order by z-index. The first node in the array should be displayed below all other nodes, and the last node in the array should be displayed on top of all other nodes.
Sure but I can't interact with them the same way as I do in Obsidian. It's an electron app so it's already heavy on the web based tech.
Currently I export my notes as a webpage and edit using my Nextcloud instance. It works, but it's not very nice.
It's not just a file "of some kind", it's a text file. That's one of Obsidian's key selling points - that a vault is simply a collection of text files. The "text" part is important to both Obsidian's philosophy and the majority of its users.
Sqlite as an application file format is great [1], but for a knowledge base / note taking app the benefits are not worth the tradeoffs in my opinion.
Sqlite is more performant more performant and provides lots of built-in features. However, most note taking users do not have enough notes or files to benefit from that performance. Sqlite will also lock the user into the application, whereas a "pack of files" can be used in the shell as a text editor. Using markdown files + a open json format has the benefit of being supported by multiple applications (e.g. sometimes i open my obsidian vault in vscode), while a sqlite database would need a proprietary schema coupled with a single application
I prefer an open file format that isn't tied to a vendor. A "data bridge" might handle syncing and diffing more efficiently than plain files, but it is still tied to the vendor. For example, I prefer not to pay for Obsidian Sync, and I'm able to use a git plugin and storing my files on nextcloud to sync between my devices. This leverages existing tech without having to implement features from the ground up
And this spec is for complicated layouts, not trivial notes you're comparing it to, so your intuition from simple notes doesn't translate to this use case
I'm not sure I understand. Can you clarify?
> Sqlite is more performant more performant and provides lots of built-in features. However, most note taking users do not have enough notes or files to benefit from that performance.
For a graph with lots (thousands+) of nodes/edges, SQLite is probably capable of being more performant than a JSON file, depending on whatever specific kind of performance we're measuring. That said, to me, the most interesting thing that SQLite gives for applications when compared to flat files is data integrity, via transactions and schemas/constraints. Performance is nice but not even close to the most interesting thing about SQLite in most applications. Performance has never been the reason I've chosen SQLite over flat files for my applications.
> Sqlite will also lock the user into the application, whereas a "pack of files" can be used in the shell as a text editor. Using markdown files + a open json format has the benefit of being supported by multiple applications (e.g. sometimes i open my obsidian vault in vscode), while a sqlite database would need a proprietary schema coupled with a single application
In a sense you're right about this! I'll grant it's easier to open a JSON file in vscode and edit it if you already know vscode and JSON. That said, SQLite is in the public domain with a well-defined, stable format and there are countless free and open source database editors/viewers out there.
SQLite is self-describing, also. You open `sqlite3` and type `.schema` and it shows you the database schema. You enter a query and get some results. It's all right there. So, while a database might have a schema that was designed for a particular application, that doesn't mean you as the end user can't tinker with it, and the number of people who know SQL is rather large.
My feeling is that xml and json is always easier to parse than SQL. What I often see is that a large amount of application logic is hidden in how the tables are joined or selected. In SQL this is almost always the case and of course this is often the case in json as well to some extent but usually alot less.
In the end it comes down to that xml was not the end all of integration formats, neither are its replacements. Actually being able to read and understand what you parse without tooling is an immense help.
Saying that SQLite doesn't have "real types" is simply false. If one doesn't want to learn how to use a tool, blaming it for that failure is poor form.
INT
INTEGER
REAL
TEXT
BLOB
ANY
Unfortunately, that is not the same as typing. Sqlite lacks typing because, as the sqlite author explains in the docs, flexibility is the goal. He continues with "But other developers are aghast", and so strict tables where born, but you can clearly see this cannot overcome real concerns. Try to look for the DATETIME datetype in that list.
Deep bow to sqlite, its design goal was to be the ini file replacement and it has outperformed itself on that one.
Thanks for the extension link. Although constraints are not reuable type definitions, they would still be helpful in this context. Pity that json doesn't have a type for dates, one has to rely on string formats: https://json-schema.org/understanding-json-schema/reference/...
I don't consider this a valid distinction where databases are concerned.
If you define a datatype in an ordinary SQL database, and try to pass it invalid data, it will fail at runtime. How else could it work? There's no compile-time interaction between the value and the database.
If you define a field as BOOLEAN in Postgres, then the value must be 0 or 1, or the database will refuse to write it and return an error. In SQLite this is spelled INTEGER NOT NULL CHECK (col_name = 0 or col_name = 1). More verbose? Yes. Identical semantics? Also yes.
It would certainly be nice if SQLite had a datetime validator that could be used as a check constraint! Hold, up, I got you fam: CHECK(date IS strftime('%Y-%m-%d', date)). If you need a different format, those are,, available.
I guess it has a date type after all! Learn something new every day.
NULL
INT(EGER)
REAL
TEXT
BLOB
(INTEGER PRIMARY KEY)> You like C++ because you're only using 20% of it. And that's fine, everyone only uses 20% of C++, the problem is that everyone uses a different 20% :)
https://eli.thegreenplace.net/2009/10/17/the-c-bashing-seaso...
The YAML footguns are too numerous to reproduce here, so here are some sources:
https://stackoverflow.com/questions/3790454/how-do-i-break-a...
https://www.arp242.net/yaml-config.html
YAML isn't terrible if you only ever have to read what you wrote. Now consider that there are 63 different ways to write multi-line strings in YAML -- how many of those have you committed to memory? Yeah... now throw 10-100 developers into the mix, each with their own favorite alternative syntaxes -- good luck making sense of your YAML.
There can only be so much nesting before you lose track of what item belong to which parent. Copying some yaml structures over to another level requires care, as the result might look correct, but the white space parser thinks otherwise.
I have lost hours of debugging yaml files when a dash was missing somewhere or when I needed one more leading space. The parser accepts it happily, but half of the typical javascript programs will only detect things are wrong when it has already executed on half of your spec. The other half will just run with input that wasn't intended that way.
I remember writing artillery.io test specs where all those problems pop up.
Now the good thing from JSON is JSON Schema. The latest spec allows you to specify quite advanced validations. Yaml has no such thing.
As to your remark: Yaml for front matter is defensible, as you dont have deeply nested structures. Though, as an obsidian user you want to make sure your front-matter is conforming to your own schema. That would require writing a json spec and then have your yaml internally converted to json before handing it over to the validator.
A spec is worthless if you cannot validate against it. Json and xml have a good story there. I concede that yaml is more human-readable than json without an editor. Correctness is the holy grail though.
CMakePresets.json is an offender here...
A tool may parse the json canvas file and extract information such as "there are three ideas linked to this book". Another tool may output an SVG.
I agree with you that posting Obsidian's spec to start the conversation would have been welcome, but this is not what they did.
You shouldn't need to ask permission or consult anyone to do this. That's a silly bar, 'egregious' even. When markdown was created, Gruber wasn't asking all his friends making text editors whether they'd support it. He just released it, and others chose to adopt it on its merits.
I don't meant this to sound harsh, but just to call it as I see it - your comment is basically creating a 'damned if you do, damned if you don't' scenario and an example of why people building things shouldn't worry much about comments in forums.
Many of the best projects that become defacto standards start as a solution to a very specific and real problem that had no public input.
If JSON canvas isn’t a good solution for broad adoption/interoperability, it won’t become one. The world is not harmed by its release, and at worst, it’s now far easier for people building software to understand and make tools that can interact with Obsidian. At best, it becomes a solid foundation and option for tools going forward.
Obsidian didn’t have to do anything here. I’m glad they did.
SVG really stretches the “human readable” thing to its limit, and I personally would not adopt an SVG-like format for anything that isn’t SVG.
(Given that this is coming from obsidian, I kind of expected something actually markdown-like - which would certainly be a challenge, but not an impossible one.)
Granted, this notation is a little higher level than SVG, so the model mismatch might be a bigger problem anyway?
Config language design seems to have a surprisingly "bumpy" design space, where optimizing for one thing (human readability, or human familiarity, or tooling support, or flat data, or nested data, or strong types, or DRY, etc...) necessitates tradeoffs in other areas.
No wonder there's so many config languages!
Parties that do not read the database contract will get caught by runtime validation. However, any code that targets the database contract could make use of that contract, e.g. with scaffolding. This might enable tighter integration with type checking in the client program.
What you propose is not a contract, your code doesn't understand it, so now you introduce a new problem. (I think that is why the sqlite author doesn't seem to be too enthusiastic about bolting on strictness checks, as its potential is really limited and it contradicts its design).
Admittedly these sorts of tricks are obscure, if by 'obscure' we mean "you have to feed a search engine a string like 'Validate SQLite datetime' and read some sources". But to reiterate my point slightly differently, the verbosity of these CHECK constraints doesn't indicate that they're doing anything different from a "typed database".
Out of curiosity, I asked ChatGPT, which got the "well formed" version, when I pointed out it would accept 2023-03-34, it gave a correct explanation of what SQLite would do with that date, and suggested `CHECK(date_column = strftime('%Y-%m-%d', date_column))`, which is more satisfying than the other one, and has the same effect. Really gotta keep an eye on the chatbot.
If I had the final say for how this format were to be publicized, I would probably just include the JSON schema “as-is” in the Obsidian developer docs. Then, it’s just as usable by third parties without making promises about interoperability.
But Obsidian is thinking ahead here. They’ve intentionally given us a duck to poke at[1], knowing we’d offer feedback and critique. If the goal is to get people to imagine a better implementation, preying on HN’s “I could implement this so much better” sense is one hecking smart way to get the community to pitch in.
I'd say this is the point.
If it's "published as part of Obsidian" it implies Obsidian can break it at any time and the interop anyone else has built will need updating. But "published as a standalone spec" means Obsidian is saying "we won't change this without warning" (at the very least).
I haven't seen anyone mention it in this thread, but my immediate reaction to seeing JSON Canvas was that it looks like a spiritual sibling to Markdown, a famously (and intentionally) informal spec.
There's a real wisdom to this informal spec approach, and it's worked incredibly well for Markdown despite grumblings about its lack of standardization.
I imagine this is the Obsidian team's intention with JSON Canvas. Obsidian has really benefitted from the informality of Markdown, which meant they were free to extend it in ways that made sense to the product (with things like cross-page ^ references, yaml frontmatter, etc), without triggering the Spec Police. The same perhaps applies here to Canvas products.