Jo – a shell command to create JSON (2016)(jpmens.net) |
Jo – a shell command to create JSON (2016)(jpmens.net) |
"If the first version of your shell script is more than five lines long, you should have written it in Python."
I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or Javascript, it would have been far more obvious what the resulting output would have been, considering that those languages already use {} for objects and [] for lists.
For example, take this example:
jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false
json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False})
Taking these two command lines:
jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null
python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null
2022-02-05 21:25:38.357228500 start-jo-x86
2022-02-05 21:25:45.319337500 stop-jo
2022-02-05 21:25:45.319338500 start-python2-x86
2022-02-05 21:26:18.876235500 stop-python2-x86
2022-02-05 21:26:18.876235500 start-jo-arm
2022-02-05 21:26:22.316063500 stop-jo-arm
2022-02-05 21:26:22.316064500 start-python3-arm
2022-02-05 21:26:40.379063500 stop-python3-arm
Test script is at https://pastebin.com/4tTVrDia
$ time python3 -c ''
real 0m0.029s
$ time python2 -c ''
real 0m0.010s
$ time bash -c ''
real 0m0.001s
I also wrote my own tool, xidel [1]:
time for i in $(seq 1 $count); do xidel -se '{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": false()}' > /dev/null; done
But if you actually want to repeat something a thousand times, you would use a loop in the query for 0,017s:
time xidel -se 'for $i in 1 to 1000 return {"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": false()}' > /dev/null
$ time cat<<EOL
{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17},
"sunday": false}
EOL
{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17},
"sunday": false}
real 0m0.002s
user 0m0.000s
sys 0m0.002sI've never really found any language that feels good for that kind of thing, there's definately a middle ground where it's getting too much for bash, but jumping to a language loses too much at the initial conversion to make it feel worth it until you are well past the point where your future self will think you should have made the switch.
Some languages have things like backticks in php to inter-operate but it's still not great experience to mix between them. For my own little things I'm currently looking at fish, but bash is omnipresent.
This tool seems to delay that point even further, as currently dealing with generating json is definately a pain point (whereas manipulating it in jq is often really good).
But if anyone can point to good examples of this transition in python then I'd be very interested.
edit: jq is more powerful than I thought for creating json, see https://spin.atomicobject.com/2021/06/08/jq-creating-updatin...
The problem with python here is that while python-sh might be nice, you have to install any extra libraries you need with it, and that's not a trivial problem for installing scripts into prod.
Xonsh is better since you kind of get both the benefits of python and a shell like language, but frankly it's broken in a number of ways still. I use it daily, but hopefully you don't need to ctrl-c out of something since signal handling is iffy at best. It is kind of nice to be able to import python directly on the command line and use it as python though...
Up until now I've been creating temp JSON files to feed the commands and I thought jo would be a great tool to make this easier. Now that I know jq can also create JSON, I'll just use that instead.
> "If the first version of your shell script is more than five lines long, you should have written it in Python."
Seriously, these kind of "common knowledge", "universal truth in a sentence" sayings are often the mark of wannabe guru mid career engineers that have no clue what they are talking about.
That is going a bit far. By all means use Python. Go ahead and attack people who use the shell. But let's be honest. The shell is faster, assuming one knows how to use it. A similar claim is often made by Python advocates, something along the lines of Python is not slow if one knows how to use it.
The startup time of a Python interpreter is enormous for someone who is used to a Bourne shell. This is what always stops me from using Python as a shell replacement for relatively simple jobs; I have never written a large shell script and doubt I ever will. I write small scripts.
If anyone knows how to mitigate the Python startup delay, feel free to share. I might become more interested in Python.
Anyway, this "jo" thing seems a bit silly. Someone at Google spent their 20% time writing a language called jsonnet to emit JSON. It has been discussed on HN before. People have suggested dhall is a perhaps better alternative.
That's not a shell command any more than running "python" is. `jo` is its own executable.
And doing the sub-commands are not necessary; `jo` supports nested data natively.
Besides that, looking at json only, it's part of the standard library so is more likely to already exist on any given machine rather than this.
echo '{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": false}'
And this lets you write json without (or with less) braces, commas and quotes. That alone is already a big win.
Paired with httpie [2] aliases [3] it produces concise APL-like syntax:
POST https://httpbin.org/post test:=j`a=b c=`e=3` l=`*1 2 3``
http POST https://httpbin.org/post test:="$(jo a=b c="$(jo e=3)" l="$(jo -a 1 2 3)")"
{"a":"b","c":{"e":3},"l":[1,2,3]}
$ http pie.dev/post test[a]=b test[c][e]:=3 test[l][]:=1
Btw loving your new desktop app so far!
`jq` can construct JSON "safely" from shell constructs, but is rather more verbose - e.g. with the same examples:
$ jq -n --arg name Jane '{"name":$name}'
$ jq -n \
--argjson time $(date +%s) \
--arg dir $HOME \
'{"time":$time,"dir":$dir}'
$ jq -n '$ARGS.positional' --args spring summer winter
$ jq -n \
--arg name JP \
--argjson object "$(
jq -n \
--arg fruit Orange \
--argjson point "$(
jq -n \
--argjson x 10 \
--argjson y 20 \
'{"x":$x,"y":$y}' \
)" \
--argjson number 17 \
'{"fruit":$fruit,"point":$point,"number":$number}' \
)" \
--argjson sunday false \
'{"name":$name,"object":$object,"sunday":$sunday}'I'm very skeptical of this. If I put x=001979 in as a value I dont think I want you trying to guess if that's supposed to be an integer or a string.
This sounds like the Norway Problem waiting to happen.
But, clearly there is a use-case for producing json with integer, null, and boolean values.
jo -- -s opaque_id=001979
{"opaque_id":"001979"}Avoid using this command in prod.
> jo normally treats value as a literal string value, unless it begins with one of the following characters:
> value action
> @file substitute the contents of file as-is
> %file substitute the contents of file in base64-encoded form
> :file interpret the contents of file as JSON, and substitute the result
This will be a source of bugs or security issues for any script generating json with dynamic values.
$ ./jo foo=1 bar=2 obj=$(./jo -a 1 2 3 "</script" '"')
{"foo":1,"obj":[1,2,3,"<\/script","\""],"bar":2}
$ ./jo foo='abc
> def
> ghi'
{"foo":"abc\ndef\nghi"}
$ cat jo
#!/usr/local/bin/txr --lisp
(define-option-struct jo-opts nil
(a array :bool
"Produce array instead of object")
(nil help :bool
"Print this help"))
(defvarl jo-name *load-path*)
(defun json-val (str)
(match-case str
("true" t)
("false" nil)
("null" 'null)
(`{@nil` (get-json str))
(`[@nil` (get-json str))
(@else (iflet ((num (tofloat else)))
num
else))))
(let ((o (new jo-opts)))
o.(getopts *args*)
(when o.help
(put-line "Usage:\n")
(put-line ` @{jo-name} [options] arg*`)
o.(opthelp)
(exit 0))
(if o.array
(let ((items [mapcar json-val o.out-args]))
(put-jsonl (vec-list items)))
(let ((pairs [mapcar (lambda (:match)
((`@this=@that`) (list (json-val this) (json-val that)))
((@else) (error "~a: arguments must be name=obj pairs" jo-name)))
o.out-args]))
(put-jsonl ^#H(() ,*pairs)))))For anyone else wondering: https://www.nongnu.org/txr/
In the spirit of "do one thing well", I'd so rather use this to construct JSON payloads to curl requests than the curl project's own "json part" proposal[1] under consideration.
Curl does do one thing: make network requests. This feature is making it easier to make network requests, i.e. it makes it better at doing the one thing that it does.
curl looks like it has hundreds of flags.
There's one for yaml too that works well in my experience: https://github.com/redhat-developer/yaml-language-server
https://play.google.com/store/apps/details?id=com.alexsci.an...
I wonder if a formal syntax would help? Perhaps including relevant shell syntax (interpolation, subshell). It could clarify issues, and this different perspective might suggest improvements or different approaches.
It's written in C and is not actively developed. The latest commit, it seems, was a pull request from me back in 2018 that fixed a null-termination issue that led to memory corruption.
Because I couldn't rely on jshon being correct, I rewrote it in Haskell here:
https://github.com/dapphub/dapptools/tree/master/src/jays
This is also not developed actively but it's a single simple ~200 line Haskell program.
$ alias fooson="node --eval \"console.log(JSON.stringify(eval('(' + process.argv[1] + ')')))\""
$ fooson "{time: $(date +%s), dir: '$HOME'}"
{"time":1457195712,"dir":"/Users/jpm"}
#!/usr/bin/env node
console.log(JSON.stringify(eval('(' + process.argv[2] + ')')))
#!/usr/bin/env node
for(const [k, v] of Object.entries(process.env)) {
if (!global.hasOwnProperty(k)) {
global[k] = v;
}
}
console.log(JSON.stringify(eval('(' + process.argv[2] + ')')))
$ export BAR="\"'''\"\""
$ fooson '{bar: BAR}'
{"bar": "\"'''\"\""}
$ fooson "{'time': $(date +%s), 'dir': '$HOME'}"
{"time":1457195712,"dir":"/Users/jpm"}
$ export BAR="\"'''\"\""
$ fooson '{"bar": "$BAR"}'
{"bar": "\"'''\"\""}
jo -- -b foo=no -s a=12 q=[1,2,3] {"foo":true,"a":"12","q":[1,2,3]}
I actually think the "Norway problem" is a PEBKAC from users not learning the data format. But this tool may confuse some people or applications who don't know what a boolean, integer, float or string are, and try to mix types when the program reading them wasn't designed to. Probably the issue will come up whenever people mix different kinds of versions ("1", "1.1", "1.1.1" should be parsed as an int, float, and string, respectively)
Should “a” be number or a string in the resulting JSON?
And if it’s number, how can I tell it to output a string?
Disclosure: I'm the author
Project link: https://github.com/ngs-lang/ngs
Enjoy!
You guys do that for "written in Rust"
python -c "import json; print(json.dumps(dict(a=10, b=False)))"But jo does come into its own when you're wanting to use shell variables.
> jo mypid=$$ set_or_not=$WEASEL
> python -c 'import json,os;print(json.dumps({"set_or_not":os.getenv("WEASEL"), "mypid":os.getpid()}))'In the normal case, you'd have variables interpolated in there, not static JSON. And then you run into the quoting problems that jo was created to work around...
> [...] are often the mark of wannabe guru mid career engineers that have no clue what they are talking about.
is the "toxic" part.
At least he says "are often," as not to accuse the anonymous mid career Google engineer.
If you do take the time to deal with pip (which, yes, is a problem) you get access to even more batteries that would have been a pain or just flat out impossible with shell.
(& on many distros, you can use your system package manager. I'm not seeing a material difference between a shell script that requires "apt-get install jo" and a Python script that requires "apt-get install python3-requests" or something.)
But either way, for circumstances where shell is the wrong tool for the job, "invoke python in the middle of this shell script because it's a better tool for this particular part of the job" is a strategy I've used before & will keep using, b/c it produces code that isn't riddled with bugs.
Bash on error will just move on to the next line as if nothing ever happened unless you:
set -eu
set -o pipefailThe final syntax is heavily inspired by the HTML JSON forms one. But we made it stricter, added type safety, and some other features. It also supports all the existing functionality like embedding raw JSON strings and JSON/text files by paths.
The final implementation[2] is completely custom. We have plans to ship it as a standalone tool as well, publish the test suite in a language-independent format and write a formal spec so that other tools can easily adopt it. This spec will eventually be a subset of one for the overall HTTPie request language, which is currently tied to our CLI implementation but we want to decouple it.
Happy to hear you like the desktop app!
[0] https://github.com/jdp/jarg
[1] https://www.w3.org/TR/html-json-forms/
[2] https://github.com/httpie/httpie/blob/master/httpie/cli/nest... — this is the path parser
jo -- -s a=123 -s b=00123 -s c="$(date)"
{"a":"123","b":"00123","c":"Sat 5 Feb 21:29:08 GMT 2022"}
Implicit typing sucks: https://www.destroyallsoftware.com/talks/wat
"Even though Python isn't the fastest language out there, it's likely still faster than the shell command above."
Which is most definitely is not - it's 5x slower.
(Probably not a huge issue in the real world if you're writing a shell script, mind, given that bash itself isn't a performance demon. But claims have to be tested.)
If you are in a shell session and have to choose between executing python -c or calling jo, the latter is faster as you’ve demonstrated. But that’s not a realistic assumption.
Statements like these are almost certainly part of some combined work. The data you’re feeding to jo comes from somewhere. Its output is written somewhere.
You can’t convince me that if you’re already inside some Python script, that invoking json.dumps() is slower than calling jo from within a shell script.
At no point did I claim that launching Python AND running that json.dumps() is faster than running that shell command. I only stated that the json.dumps() is.
You're not going to shell out to `jo` and that's fine - it's not what `jo` was created for; it's explicitly a shell command to help you work around the annoyance of getting quoting right when constructing JSON from the command line (which I've had to do a lot and I'm pretty sure many people have to.)
> If you are in a shell session [and want to create JSON] ... that’s not a realistic assumption.
Of course it is. People create JSON in shell scripts all the time! That's why things like `jq` exist - because this is what people do!
Example for jo:
docker run --rm -it debian bash
apt update && apt install -y jo nano
nano bash-loop.sh && chmod +x bash-loop.sh
#!/bin/bash
for ((i=0;i<1000;i++));
do
jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false
done
time ./bash-loop.sh >/dev/null
docker run --rm -it debian bash
apt update && apt install -y python3 nano
nano python-loop.py
import json
for i in range(1000):
print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))
time python3 python-loop.py >/dev/null
Debian GNU/Linux 11 (bullseye)
jo 1.3
Python 3.9.2
real 0m2.230s
user 0m1.106s
sys 0m1.076s
real 0m0.027s
user 0m0.021s
sys 0m0.005s
Note: jo seems to pretty print because of the "-p" parameter, which is not the case with Python, might not be a 1:1 comparison in this case. Would be better to remove it. Though when i did that, the performance improvement was maybe 1%, not significant.
Admittedly, it would be nice to test with actually random data to make sure that nothing gets optimized away, such as just replacing one of the numbers in JSON with a random value, say, the UNIX timestamp. But then you'd have to prepare all of the data beforehand (to avoid differences due to using Python to get those timestamps, or one of the GNU tools), or time the execution separately however you wish.
Edit to explain my rationale: Why bother doing this? Because i disagree with the sibling comment:
> The claim was that the Python snippet would be quicker than the jo snippet.
In my eyes that's almost meaningless, since in practice when you'll actually care about the runtimes will be when working with larger amounts of data, or alternatively really large files. Therefore this should be tested, not just the startup times, which become irrelevant in most real world programs, except for cases when you'd make a separate invocation per request, which you sometimes shouldn't do.
Edit #2: here's a lazy edit that uses the UNIX time and makes the data more dynamic, ignoring the overhead to retrieve this value, to get a ballpark figure.
Use time value for jo:
jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=$(date +%s)) sunday=false
import time
...
print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": int(time.time())}, "sunday": False}))
real 0m2.794s
user 0m1.422s
sys 0m1.313s
real 0m0.027s
user 0m0.020s
sys 0m0.006s
Edit #3: probably should have started with a test to verify whether the initially observed performance differences (Python being slower due to startup time) were also present.
Single iteration results for jo:
real 0m0.003s
user 0m0.000s
sys 0m0.002s
real 0m0.022s
user 0m0.017s
sys 0m0.004s
It’s like saying that it’s fine if an appliance might randomly burn your house, because there’s a button you can press to not burn your house.