Select Transform: JSON template over JSON(selecttransform.github.io) |
Select Transform: JSON template over JSON(selecttransform.github.io) |
XML -> JSON
WSDL -> JSON schema
XPath -> JSON path
XSLT -> JSON template?
SOAP -> Swagger and friends
On a merits-based thing, if you're just looking at XML, it has a lot of things going against it .If someone sends me something "in JSON", I can have a good guess what it'll look like. Not as much luck with XML. XML punishes you for using attributes (because you can't place composed types within them), but the awkward alternative is placing attributes in sub-elements.
I have a list of products with prices. I store the price as a quantity attribute on a <product>. I decide later on to store currencies for all my "money" objects. Do I now double the amount of attributes? It's all super awkward, and JSON's mental model is more straightforward, even if you end up with approximations.
"You know how you feel when you have to parse XML from 2005? That’s how we’ll all feel about JSON in 2025. It’s not bad, just bloat-y."
XML, JSON, gRPC… These are all technologies to solve a problem. The trouble is people confuse technology with a business solution and they start applying it as though by "having a REST API" you've done something. You can't say you've accomplished anything until you actually demonstrably deliver value!
And so you couldn't just READ that, you had to have a parser that fully understood EVERYTHING going on there. JSON won't get namespaces, and therefore it will remain good. We will continue to replace what namespaces did for XML, but will keep that data in its own metadata document, outside the main document. I think that's why JSON won't be replaced as quickly as XML.
Also XML didn't kill XML, webservices (ws-* standards) killed XML.
There's so much quality work done with XML that all this effort is a complete waste. From streaming parsers to the XQuery, and great projects such as VTD-XML [0], it's sad to see us spinning wheels. I see TOML as another wasteful attempt at something that's only slightly better but requiring a new toolchain and an ecosystem.
Well in green-field projects it's certainly advisable to minimize what is "done in"/"expressed in"/achieved-via the current-interchange-format de-jour --- but in messy real-world projects such rabbit holes can unavoidably open. And then you sooner or later end up thinking, as I did back then, "XML is kinda sucky but XPath and XSLT make it actually pretty nifty & fun to deal with and somewhat escape its own very suckiness".
XML can be really powerful, but a lot of real users actually don't know how to use XML. Many just use it as an enterprise version of JSON in which case I am happy to just work with JSON, plain and simply.
Also JSON processing can be easily parallelized, because it is easy to express in a newline delimited format (http://ndjson.org/).
XML comments -> JSON: nah -> YAML
XML CDATA (multiline raw strings) -> JSON: nope -> YAML[0]: http://jmespath.org/
Too many tools? Are you not selective about what tools you use anyways? The field is decades old. Do you program in assembly? Fortran?
[0]: http://json5.org/
[1]: http://hjson.org/
The thing is, when creating webservices you like creating tests. With WSDL you have tools which given a WSDL will generate a test-suite for you and some mocked services.
With JSON you'll have to hack-up something using Swagger.
Despite this, many languages aren't great at this! we have tools that work alright on first-order operations but fall apart on higher level things.
Even really simple things like "conditionally include a key" are rarely supported in a way that leads to concise code. How many times have we all written
foo = {...}
if bar:
foo['a'] = x
foo = { ... , 'a': x if bar }
Haskell lenses kinda goes into this stuff, though Haskell itself is a bit hard to use in a lightweight fashion. Clojure has Specter. But still looking for some more stuff to fill this hole, especially in the "transitionally typed" stuff like Typescript or Python + Mypy
return {
labels: data.items.map(function(value){
return {
type: "label",
value: value > 10 ? value : value * 100
};
})
};
return { labels: data.items.map(value => { type: "label", value: value > 10 ? value : value * 100 }) };
I am a bit concerned that there is no discussion in the docs of the potential security risks of allowing direct native JS execution of arbitrary instructions passed in by an untrusted source.
I use a project called JSONLogic (jsonlogic.com) which bears some similarity to ST in terms of being able to select and transform values. The biggest advantage I see with it is, unless you explicitly plug in a rule that parses and executes user data, there is no way for the data to "escape the sandbox". This means you can safely build a query syntax on top of it where you can directly consume the arbitrarily complex query from an untrusted source and execute it in a secure manner.
I thought I would provide some context on why I wrote this library, and how I'm using this right now. So here it goes:
Other than ST.js, I also work on another open source project called Jasonette (https://www.jasonette.com), which lets you write an entire native iOS/Android app in nothing but JSON markup.
And when you can express the entire app logic--from model to view to controller--in JSON, you can split them up whichever way you want and load them from anywhere (from a file, from cache, from a remote server, or from local memory).
But the biggest benefit of all is: you can load an entire ios/android native app from the server in realtime, just like how web browsers load HTML/JS/CSS in realtime.
When working on Jasonette, implementing model and view was relatively simple. For model it's natural since JSON is all about describing data. For view i just needed to come up with a syntax to describe layouts and all the standard mobile UI components in JSON.
However the biggest challenge was how do i actually describe functions in JSON. Without a function, it's just a mockup and won't really do anything meaningful.
Which brings us to ST.js.
When you think about what a function is, it takes one value and turns it into another. So basically what I needed to do was build something that will take one JSON, and turn it into another JSON, but most importantly I would have to do it using JSON. Basically I needed to implement a finite state machine in purely JSON.
And this is what templates do. So I set out to build a JSON template engine that turns one JSON into another using a template which itself is written in JSON.
What's really cool about this is, since the template is JSON (As far as I know there doesn't exist any prior art that uses JSON as a template, otherwise I would have used it instead), it has all the benefits of the JSON format itself:
1. You can store it anywhere (Most DBMS nowadays support JSON natively)
2. You can send it over the Internet
3. You can compose them easily
4. You can validate them using JSON schema
5. etc.
To use a more specific example, I use ST.js in both Android and iOS versions of Jasonette as the template engine. And a JSON template is absolutely necessary in this case.
For example, if I want to take a piece of device-generated data and render it, I need to be able to somehow parse it client-side (can't send it back to server to re-generate a JSON) http://docs.jasonette.com/templates/#3-device-api-generated-...
This also applies to many cases where the client-side data contains privacy-sensitive data. The only way to dynamically parse something that happens on the client side is by writing the template itself in JSON and sending it over as data.
Anyway, I hope you guys take a look at the examples on the homepage to get a glimpse of what makes ST.js powerful. Each example is almost its own library, except that you don't need a separate library for those purposes since all you're dealing with is JSON.
I work on a similar problem though using a UI based approach, I gave a demo to the Kubernetes SIG App group a while back which be found here: https://www.youtube.com/watch?v=alEzE8MSNaI&t=30m
Btw Ethan, maybe my email is landing in your Spam folder? I've tried reaching out :)
In essence ST.js is a finite state machine implemented purely in JSON. So I can imagine it being used for things I haven't thought of, which is why I open sourced it. Would be really cool.
Please feel free to reach out at ethan.gliechtenstein[at]gmail thanks!
I can't wait to play around with this.
Would that be a good use case for this?
Basically you can send the JSON template over JSON to the server, and let the server transform the raw data using the template before returning.
The most important benefit here is that, since everything can be expressed in JSON, it can be stored or transmitted over the Internet, which enables interesting cases like this.
I also would like to know in which situation this is preferable to jq.
Not sure which one is better at this point. Using jq still often feels like trying to decode a code golf contest entry.
Json should strictly be used as a serialization format and everyone who touches it should immediately parse it into a typed data-structure that represents your domain better.
In fact you really should represent your data in your code using whatever native object hierarchy / algebraic data types / option types / etc. your language supports. (Algebraic datatypes are awesome at this sort of thing, it allows you to avoid the aforementioned conditionality)
The problem is the right way to do this is also the hard way, because people hate writing "boilerplate" classes and serialization code and once you start thinking about making it "easy", you end up using json as the in-application data structure because "who needs types anyway".
foo = {
'a': 1,
'b': 2,
**({
'c': 3,
'd': 4,
} if bar else {}),
}
[1] https://docs.python.org/3/whatsnew/3.5.html#pep-448-addition...
Nice that this works on lists as well.
They seem neat but I haven't come across a case where whipping up, in vanilla old-school Haskell, a custom set of small simple ADTs and the related handful of recursing walk-and-transform functions didn't do the job perfectly cleanly and comprehensibly.. this sort of stuff is already very compact to write and code-gen like TH seems almost overkill to me. Guess I'm itching to come upon a much more painful scenario to finally give them "lenses" a fair chance.
let x = y { foo = foo y { bar = bar (foo y) + 1 } }
let x = over (foo.bar) (+1) y
A decent amount, but I'd rather have all of that separate because it makes the code easier to read.
I'm not interested in how concise a block of JSON text can be with all of my transformations of conditionals and loops. That's going to be difficult to read without highlighting. I also can't put a breakpoint in the middle of some JSON text without the appropriate support from the language/runtime/IDE.
However, this syntax is much better than JSON Patch which I've considered using for document templates.
I think there's a bit of a conflict with regards to immutability. When I see a variable in the code, it's much easier to handle a single definition rather than a "half-definition" followed by a bunch of conditionals.
I've also seen some waaay to concise code that becomes super legible after just one extra line of code to split things up.
I've run into the opposite problem a lot though. Even if every operation is simple, the length of code makes the intent unclear, especially in cases involving aggregation. Death by a thousand cuts
results = []
for elt in other_array:
if elt is None:
results.append(0)
else:
results.append(f(elt))
results = [ f(elt) if elt is not None else 0
for elt in some_other_array ]
Clarity is the most important, and after a couple conditionals it makes sense to be more explicit about control flow. But many common tasks are not _about_ control flow, even if C-isms make us write them as if they do. "if" statements generate so many bugs, eliminating them is great.
1. You can store JS anywhere (all DBMS support text).
2. You can send it over the Internet... Like everything else nowadays.
3. You can compose JS easily.
4. You can validate JS using linters.
5. etc.
Besides... Aren't you only disguising functions in strings? You can eval strings anyway.
I've done something similar to ST out of the same frustrations at work, but my use case was way more specific, and you took it way farther with better execution.
One of the use cases for ST.js in Jasonette is dynamic client-side rendering, where it takes dynamic data and renders it against the template, after which it becomes native components.
Another use case is implementing actual functional programming. This one is not as straight-forward since it involves maintaining another separate state machine for function call stack. But this is also achieved using ST templates. Here's a blog post that talks about it in detail http://blog.jasonette.com/2017/02/15/functional-programming-... but it's a bit involved.
Also Jasonette has something called mixins, which you probably can guess based on its name. It lets you mix in JSON objects into another, so that you can make the app modular. That's also achieved internally using ST.
Overall, I believe even I'm just scratching the surface of what this can do, which is why I decided to take some time to clean things up and prepare and open it up as its own repo, because I think there's tons of other things that can be done by taking advantage of the fact that all that's happening is:
JSON A + JSON B = JSON C
Hope this makes sense!
XSLT is very powerful but also incredibly messy because it has to be expressed in XML.
Doing the same in JSON seems like it would result in the same problem.
Being expressed in XML does not help, but the language is just bad with implicit data flowing through and very odd constructs, there are XML template languages which I found much more readable than XSLT e.g. I remember my experience with genshi much more fondly than the years I spent with XLST.
dbaseIIIplus was the one that actually almost managed this feat.
This particular implementation looks a bit messy though - XSLT is valid XML and Xpath, while this solution uses templating syntax. That's more human readable, but also slightly inconsequent - you wouldn't be able to validate such a template with json schema for example.
Could you elaborate?
This is the guy: https://stackoverflow.com/users/317052/daniel-haley
I mean, sure, it could do the job, but did anyone ever sit down at a file of XSLT and think "yes, this is the best way this information about transforming this XML could be represented"?
JSON is even less suitable for this. It's horrible to read this stuff, horrible to write.
There's a steep learning curve full of false summits, for sure, but once you grasp it, there is always an elegant way of solving a problem using it. Sometimes there's even a fast way!
Use of it in the wild, though, leaves a lot to be desired. The false summits lead people to write bad code, the bad code gives it a bad rep, the bad rep leads people to dislike using it, people's dislike leads to then writing bad code... and so on.
I remember our chat 8 months earlier on Reddit about this: https://www.reddit.com/r/functionalprogramming/comments/5ufm...
Yeah. You're still aiming for "functional programs encoded as (essentially) s-expressions but written in JS-Object-Notation (JSON) instead of LISPy parens". And that's still fine if you see that as a major leap forward. Not sure about the audience, those who want to "functionally program a mobile quasi-native APP that can self-update from a server" and know what the "functionally program" part means, wouldn't they reach for JS via React Native? Same selling points implicit already. Is it for those who never programmed but are expected to declaratively express functional idioms in your JSON notation without needing to install all sorts of dev tools and SDKs? Not entirely implausible at all I guess. Curious to see who will end up as your target audience. =)
let x = Y (Foo (old_bar + 1) quux) baz
where Y (Foo old_bar quux) baz = yXML is not a good form for a transformation language. It's not the quality of the code written in it or anything like that.
Take XSLT and change the syntax, and sure, it might be great. You can't ignore the terrible, terrible choice of making the transformation itself XML. It's completely unreadable and horrific to write.
If most people agree a representation is terrible, it's a bad choice for anything serious, because suddenly only a subset of your team or contributors can work on it effectively.
There is a reason XML isn't used as syntax for programming languages. Anyone claiming that an `if` construct, for example, is well represented by a mess of XML tags has a different brain to me and the vast majority of people I've met.
XSLT was meant to be a general-purpose tool. Having a syntax that the vast majority of people find incredibly hard to work with is clearly a huge problem with the spec.
With JSON, the answer is always a name/value pair.
A schema language in which subordinate values being presented as attributes vs. elements was a presentational option in the concrete XML representation rather than a dictate of the schema language would solve this simply. There are already multiple schema languages for XML; needing a better schema language doesn't begin to provide a reason for a different representation language, since schema and representation language are separate concerns.
{
"employees": [
{
"name": "John Crichton",
"gender": "male"
},
{
"name": "Aeryn Sun",
"gender": "female"
}
]
}
<employees>
<employee>
<name>John Crichton</name>
<gender>male</gender>
</employee>
<employee>
<name>Aeryn Sun</name>
<gender>female</gender>
</employee>
</employees> employees:
- name: John Crichton
gender: male
- name: Aeryn Sun
gender: female <employees>
<employee name="John Crichton" gender="male" />
<employee name="Aeryn Sun" gender=female" />
</employees> <employees>
<employee>
<name>John Crichton</name>
<gender>male</gender>
<salaries>
<salary>10000</salary>
<salary>10000</salary>
<salary>10000</salary>
<salaries>
</employee>
<employee>
<name>Aeryn Sun</name>
<gender>female</gender>
</employee>
</employees>
JSON is not only less verbose, it is also more flexible and easier to read and understand. You don’t have to worry about should I use tags or attributes for that because I late might have to use sub tags, and that makes it far easier to use (and honestly parse as well because many XML documents I have gotten are very inconsistent).
Using a List (XSD List) type:
<employees>
<employee name="John Crichton" gender="male" salaries="10000 10000 1000" />
<employee name="Aeryn Sun" gender="female" />
</employees>
<employees>
<employee name="John Crichton" gender="male">
<salary>10000</salary>
<salary>10000</salary>
<salary>10000</salary>
</employee>
<employee name="Aeryn Sun" gender="female" />
</employees>