What even is a JSON number?(blog.trl.sn) |
What even is a JSON number?(blog.trl.sn) |
The documentation is quite paranoid that if you are dealing with untrusted inputs, you could parse two JSON numbers from the untrusted source fine and then performing an addition on them could cause your memory to fill up. Exciting new DoS vector.
Of course in practice people end up parsing them into custom types with 64-bit integers, so this is only a problem if you are manipulating JSON directly which is very rare in Haskell.
Sounds like Haskell made the right call: put warnings in the docs and steer the user in the right direction. Keeps implementation simple and users in control.
To the point of the article, serde_json support is improving in the next version of BigDecimal, so you'll be able to decorate your BigDecimal fields and it'll parse numeric fields from the JSON source, rather than json -> f64 -> BigDecimal.
#[derive(Serialize, Deserialize)]
pub struct MyStruct {
#[serde(with = "bigdecimal::serde::json_num")]
value: BigDecimal,
}
[^] Is every part of your system, or your users' systems, going to parse with full precision?
Serde has an interface that allows failing. That one should fail. There is also another that panics, and AFAIK it will automatically panic on any parser that fails.
Do not try to handle huge values, do not pretend your parser is total, and do not pretend it's a correct value.
If you want to create an specialized parser that handles huge numbers, that's great. But any general one must fail on them.
8 billion digits (~100 bits?) is far more than should be used.
Would it possible to use const generics to expose a `BigDecimal<N>` or `BigDecimal<MinExp, MaxExp, Precision>` type with bounded precision for serde, and disallow this unsafe `BigDecimal` entirely?
If not, I expect BigDecimal will be flagged in a CVE in the near future for causing a denial of service.
An 8 billion digit number is 2.5G? (Did I do my maths right?) All I need to do is shove 1,000 of those in a JSON array, and I'll cause an out-of-memory anyway.
On the other hand, any limit low enough that I can't blow up memory by making an array of 100K or so is going to be too low for some people (including me, I often make numbers of low-million numbers of digits).
Providing some method of putting a limit on seems sensible, but maybe just make a LimitedBigDecimal type, so then through the whole program there is a limit on how much memory BigDecimals can take up? (I haven't looked at the library in detail, sorry).
My 2cent anyway.
So we had to go through all the code adding quotes to all the ID fields. That was a giant pain in my ass.
I don't follow. 1-(Number.MAX_SAFE_INTEGER / 2*63) ~ 99.9%, so don't you have a >99% chance of generating an ID that gets truncated in js?
https://en.wikipedia.org/wiki/Double-precision_floating-poin...
That's still going to be a greater than 0.1% chance of hitting a non-representable value though.
Obviously, not all int64 values are representable in float64 (double).
nice pics here: https://en.wikipedia.org/wiki/Floating-point_arithmetic
https://pkg.go.dev/encoding/json#Decoder.UseNumber
That allows you to capture/forward numbers without any loss of precision.
To wit, Python's json module has `parse_float` and `parse_int` hooks:
https://docs.python.org/3/library/json.html#encoders-and-dec...
Example:
>>> json.loads('{"int":12345,"float":123.45}', parse_int=str, parse_float=str)
{'int': '12345', 'float': '123.45'}
- “+1” (not a valid number, according to ECMA-404 and RFC-8259)
- “+0” (also not a valid number, but trickier than “+1” because IEEE floats have “+0” and “-0”)
- “070” (not a valid number, but may get parsed as octal 56)
- “1.” (not a valid number in json)
- “.1” (not a valid number in json)
- “0E-0” (a valid number in json)
There probably are others.
I'm confused by this.
What is the precision of 0.1, relative to IEEE 754?
If I read it correctly, that statement is saying:
json_number_precision(json_number) <= ieee_754_precision
It does seem unclear what it means to exceed precision (given rounding is such an expected part of the way we use these numbers). Magnitude feels easier as at least you definitely run out of bits in the exponent.
So including the string "0.1" in a message is fine because v = 0.1 implies 0.05 < v < 0.15, but including 0.100000000000000000000000000000000000 would not be.
That's what prometheus is doing for example. https://prometheus.io/docs/prometheus/latest/querying/api/
I mean that rapidjson (C++) parsed the string "0.99999999999999999" as the number 1.0000000000000003. Apart from just looking weird, it's a different float64 bit-pattern: 0x3FF0000000000000 vs 0x3FF0000000000001.
Similarly, serde-json (Rust) parsed "122.416294033786585" as 122.4162940337866. This isn't as obvious a difference, but the bit-patterns differ by one: 0x405E9AA48FBB2888 vs 0x405E9AA48FBB2889. Serde-json does have an "float_roundtrip" feature flag, but it's opt-in, not enabled by default.
For details, look for "rapidjson issue #1773" and "serde_json issue #707" at https://nigeltao.github.io/blog/2020/jsonptr.html
Good thing there's msgpack I guess.
The only sane thing with JSON is to avoid numbers altogether and just use decimal-encoded strings. This forces the person parsing it on the other end to at least look up the actual limits defined by your schema.
At least 122.416294033786585 is between ...888 and ...889, though it's much closer to the former.
Say what you want about NaN, but IEEE 754 is the facto way of dealing with floating points in computers and even if NaNs and Infs are a bit "fringe" it's unfortunate that the most popular serialization format can not represent these.
They're not really noisy, but if an application would work with some random noise added, it will probably work with floats, and if it wouldn't work with noise added, it's probably easier to just not use floats and expext people to reason about IEEE details, while risking subtle bugs if different float representations get mixed.
Of course I'm not doing a lot of high performance algorithms, I would imagine in some applications you really do need to reason about floats.
This "Averia Serif Libre" is unreadable for me.
Array
Binary
Date
Decimal128
Document
Double
Int32
Int64
MaxKey
MinKey
ObjectId
Regular Expression
Timestamp
It also specs proper decimal values, mitigating the issues presented in the OP.
I must admit I totally forgot about the JSON number issue. Our files include fields for various monetary amounts and similar, and in XML we just used "xs:decimal".
Most will be less than a million and requires less than four decimal digits. But I guess someone might decide to use it for the equivalent of a space station or similar, ya never know...
Encoding numbers as string because you are using a language and parser that can't deal with numbers properly (even 64 bit doubles), is a bit of a hack. Basically the rest of the world giving up because Javascript can't get its shit together is not a great plan.
As the article said
> RFC 8259 raises the important point that ultimately implementations decide what a JSON number is.
Any implementation dealing with money or monetary rates should know that it needs to deal with precision and act accordingly. If you want to use JavaScript to work with money, you need to get a library that allows you to represent high precision numbers. It's not unreasonable to also expect that you get a JSON parsing library that supports the same.
oh, TIL that you can support large numbers with the default JavaScript JSON library https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
But do note, that in currency, there are multiple, actively used currencies that have zero, three, five (rare) or even eight (BTC) decimals. That some decimals cannot be divided by all numbers (e.g. only 0.5)
Point being: floats are dangerously naive for currency. But integers are naive too. You'll most probably want a "currency" or "money" type. Some Value Object, or even Domain Model.
XML offered all this, but in JSON there's little to convey this, other than some nested "object" with at least the decimal amt (as int), and the ISO4217 currency. And maybe -depending on how HATEOAS you wanna be- a formatting string to be used in locales, a rule on divisibility and/or how many decimal places your int or decimal might be.
(FWIW, I built backbends for financial systems and apps. It gets worse than this if you do math on the currencies. Some legislatioins or bookkeeping rules state that calculation uses more or less decimals. E.g. that ($10/3)*3 == $10 vs == $9.99. or that $0.03/2 == 0.1 + 0.2, e.g. when splitting a bill. This stuff is complex, but real domain logic)
¹IANAL. But this was told when legal people looked at our architecture.
For example, you say you store monetary amounts as cents. What if you needed to store US gas prices, which are normally priced in amounts ending in 9/10ths of a cent? If you want to keep your values as integers you need to change your precision, which will likely mess up a lot of your code.
What's worse is that these things can also change over time and there is sometimes disagreement over what the canonical value is.
E.g. ISO 4217 (used by Safari, Firefox and NodeJS) will say that the Indonesian Rupiah (IDR) uses 2 decimal digits, while Unicode CLDR (used by Chrome) will say that they use 0 decimal digits. The former is the more "legalistic" definition, while the latter matches how people use the currency in reality.
This is not a real issue if you transfer amounts as decimal strings and then pass those to the Intl API for formatting (the formatting will just be different but still correct), but it's catastrophic if you use scaled-up integers (all amounts will be off by magnitudes).
For this reason I would always store currency amounts in an appropriate DECIMAL type in the DB and send currency amounts as strings over the wire.
It's not widely known, but US gasoline prices are actually in a defined currency unit, the mill (https://en.m.wikipedia.org/wiki/Mill_(currency)).
For most purposes, using mills as the base unit would be sufficient resolution.
My intuition tells me that "x * 1000 / 1000 == x" might not be true for all numbers if you're using floats.
if (x == 0) ...
instead of something like
if (abs(x) < eps) ...
where eps is a suitably defined small number.
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
available and widely used, good interoperability can be achieved by
implementations that expect no more precision or range than these
provide, in the sense that implementations will approximate JSON
numbers within the expected precision. A JSON number such as 1E400
or 3.141592653589793238462643383279 may indicate potential
interoperability problems, since it suggests that the software that
created it expects receiving software to have greater capabilities
for numeric magnitude and precision than is widely available.
Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
I miss XML these days. Sure, it was verbose and had a bunch of different and probably excessive numeric types defined for XML Schema... but at least they were well-defined (https://www.w3.org/TR/xmlschema-2/#built-in-datatypes). And, on the other hand, without a schema, all you had were strings. Either way, no mismatched expectations.
$ python3
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 100000.000000000017
100000.00000000001 Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from decimal import Decimal
>>> Decimal('100000.000000000017')
Decimal('100000.000000000017')
>>> import json
>>> json.loads('{"a": 100000.000000000017}')
{'a': 100000.00000000001}
>>> json.loads('{"a": 100000.000000000017}', parse_float=Decimal)
{'a': Decimal('100000.000000000017')}Alright, if "you" have only ever used python. In C, for example, we have hexadecimal floating point literals that represent all floats and doubles exactly (including infinities and nans that make the json parser fail miserably).
Although it would be good to move in the direction of using a BigDecimal equivalent by default when ingesting unknown data.
Having user-set generic limits would be cool, and something I considered when const generics came out, but there's a lot more work to do on the basics, and I'm worried about making the interface too complicated. (And I don't want to reimplement everything.) D
I also would like a customizable parser struct, with things like localization, allowing grouping-delimiters and such (1_000_000 or 1'000'000 or 10,00,000). That could also return some kind of OutOfRange parsing error to disallow "suspicious" values, out of range. I'm not sure how that to make that generic with the serde parser, but I may some safe limits to the auto serialization code.
Especially with JSON, I'd expect there's only two kinds of numbers: normal "human" numbers, and exploit attempts.
C <- alpha*AB + beta*C
The key here is that beta is likely to be an exact value that is entered as a constant, and detecting it allows for a worthwhile optimization.
If it looks like a number, and it quacks like a number, sooner or later people are going to treat it like a number.
Which is perfectly fine; just don't treat it like an int32. :-)
Well it's a pretty abrupt change. 53 bits work fine, at 54 bits a quarter of numbers get truncated, at 55 it's half.
The goal is not to avoid rounding errors (which would be quite difficult when the true account value can be an irrational number, as with 3% APR compounding monthly), but to have the exact same rounding errors that are prescribed by the accounting practices. Which may vary depending on legislation.
A decimal floating point is usually a better starting point than integers are.
Which is why passing around ints is not the solution. And why I specifically mention Domain Models and/or Value Object.
A domain model would throw an exeption or otherwise dissalow certain divisions for example. What I often do, is something like `expense.amount.divide_over(3, leftover_to_last)` or `savings.balance_at(today).percentage_of(3.1337)`.
Sometimes, in simpler setups and when the language allows, I'll re-implement operators like *, / and even + and -. But when actual business logic is needed, I'll avoid these and implement actual domain methods that use the language the business uses.
But never, ever, do I allow just math-ing over the inner values.
So, I disagree: Both decimal floating point and integers are just as "bad". Maybe for the inner values in the domain model or value object, they are fine, but often there integers are a slightly better starting point because they make rounding and leftovers very explicit.
⸻
1. This is a bit on my mind lately as I discovered that LibraryThing’s sort by LoC classification seems to be broken so I exported my library (discovering that they export as ISO8859-1 with no option for UTF-8) and wrote a custom sorter for LOC classification codes for use in finally arranging the books on my shelves after my move last year.
>>> Decimal('100000.00000000000000000000017') + Decimal('1')
Decimal('100001.0000000000000000000002')If you just store cents you can't represent them. You either have to guess at the beginning the smallest unit or store the precision along with it.
Just use strings, it's much simpler.
1e10 + 1e-10 = 10000000000.0000000001
1e10000000000000000000 + 1e-10000000000000000000 = ...
[EDIT]: Oh I see, fix the parser to disallow such large numbers from entering the system in the first place, then you don't have to worry about adding them together. Yeah that could be a good first step towards safety. Though, I don't know how to parametrize the serde call.
But for addition, the idea is to give the complete number if you do `a + b`, otherwise you could use the context to keep the numbers within your `ctx.add(a, b)`. But after the discussions here, maybe this is too unsafe... and it should use the default precision (or a slightly larger one) in the name of safety? With a compile time flag to disable it? hmm...
This example on MDN seems to indicate that you can, am I misunderstanding it?
const bigJSON = '{"gross_gdp": 12345678901234567890}';
const bigObj = JSON.parse(bigJSON, (key, value, context) => {
if (key === "gross_gdp") {
// Ignore the value because it has already lost precision
return BigInt(context.source);
}
return value;
});
The optional `context` parameter is a tc39 proposal. The feature compatibility matrix on the bottom of the MDN page is really confusing because it's showing only when `JSON.parse` was added, not whether the optional `context` parameter is supported.
I've confirmed it's available in:
- V8 11.4.31
- Node 20.12.0 (with `--harmony`)
- Node 21.7.1 (without requiring `--harmony`)
- Chrome 123.0.6312.107
Firefox 124.0.2 (ironically)
Safari 17.3.1
https://tc39.es/proposal-json-parse-with-source/
https://github.com/tc39/proposal-json-parse-with-source
This issue links to the various browser engine tracking bugs:
https://github.com/tc39/proposal-json-parse-with-source/issu...
Which are:
• Chrome/V8: https://bugs.chromium.org/p/v8/issues/detail?id=12955
• Firefox/SpiderMonkey: https://bugzilla.mozilla.org/show_bug.cgi?id=1658310
• Safari/JavaScriptCore: https://bugs.webkit.org/show_bug.cgi?id=248031
With Zod, you can use z.bigint() parser. If you take the "parse any JSON" snippet https://zod.dev/?id=json-type and change z.number() to z.bigint(), it should do what you are looking for.
But the resulting type is still going to have its own limitations and sharp edges. Floats are not the right tool for every job but they are quite good at the jobs they are right for. Learning how they work is more useful than lamenting their existence.
But floats are not the right representation for values that need to exactly match, like an ID, to be sure.
If I’m off by half a cent it’s annoying. If I’m off by half a row I get nothing.
The thing is that almost all of the problems we had in my initial story came from choosing system defaults. All except the PK algorithm.
e.g. it doesn't make sense to support billing in sub-currency unit amounts just by allowing it in your API definition, as you're going to need to batch that until you get a billable amount which is larger than the fee for issuing a bill. Even for something like $100,000.1234, the bank doesn't let you do a transfer for 0.34c.
For cases where sub-currency unit billing is a thing, it should be agreed what the minimal unit is (e.g. advertising has largely standardised on millicents)
https://en.wikipedia.org/wiki/ISO_4217
Or choose a different standard, I don’t know what else is out there, but you probably should choose an existing one.
If you're doing ads and going for millicents, something like $0.01234 should be encoded as 1234 millicents.
Obviously you have to agree on what you're measuring in the API, you can't have some values be millicents and others cents.
How do you do that?
A couple of resources I found but which I’m not sure if are about exactly what you speak of
https://stackoverflow.com/questions/65480947/is-ieee-754-rep...
https://gcc.gnu.org/onlinedocs/gcc/Hex-Floats.html
Furthermore, what exactly do you mean by “all floats and doubles exactly”?
// define a floating-point literal in hex and print it in decimal
float x = 0x1p-8; // x = 1.0/256
printf("x = %g\n", x); // prints 0.00390625
// define a floating point literal in decimal and print it in various ways
float y = 0.3; // non-representable, rounded to closest float
printf("y = %g\n", y); // 0.3 (the %g format does some heuristics)
printf("y = %.10f\n", y); // 0.3000000119
printf("y = %.20f\n", y); // 0.30000001192092895508
printf("y = %a\n", f); // 0x1.333334p-2100000.000000000017
And then you print it.
Does it preserve the exact value?
But also, this sounds like a premature optimisation. Most applications will never reach a level where their performance is actually impacted by string comparison, and when you reach that stage, you're likely have already thrown out a lot of other common sense stuff like db normalisation to get there, and we shouldn't judge "regular people" advice because it doesn't usually apply to you anyway.
Out of curiosity, have you ever seen an application that was meaningfully impacted by this? How gigantic was it?
----
Scratch that. I've actually thought about it some more, and now I'm not 100% sure it's premature, I have to investigate further to be sure. Question still stands though.
Due to the type of aggregate queries that typify analytics workloads, almost everything turns into a scan, whether it be of the a table, field, or index. Strings occupy more space on disk, or in RAM, so scanning a whole column or table simply takes longer, because you have to shovel more bytes through the CPU. This doesn't even take into account the relative CPU time to actually do the comparisons.
I've never personally worked with a system that has string keys shorter than 10 [1][2] characters. At that point, regardless of how you pack characters into a register, you're occupying more bits with two strings of character data than you would with two 64-bit integers[3]. This shows through in join time.
[0]: Even modestly sized companies tend to have at least a few tables that get into the millions of records.
[1]: I've heard of systems with shorter string keys
[2]: Most systems with string keys I've encountered have more than 10 characters.
[3]: The vast majority of systems I've seen since the mid-2010s use 64-bit integers for keys for analytics. 32-bit integers seemed to phase out for new systems I've seen since ~2015, but were more common prior to that.
Don't most databases set a length limit on ID strings?
If you're setting a length limit, and it's made out of digits with no leading zeroes, then you might as well store it as a number. Is there a downside?
A numeric identity is an identity and so is a string.
If you want to math it, it is a number, otherwise... string.
"Will you ever want the 95th percentile PID? Then it is not a number. Move on."
If you need to do math with the thing, use an appropriate type of number, of course.
If you're using 64 bit numbers as a high cardinality identity that can be randomly generated without concern for collision (like a MAC address with more noise) -- well, that's an identity and doesn't need to have math applied to it. For example: "What's the mean IP address that's connected to cloudflare in the last 10 minutes" or "what's the sum of every mac address in this subnet?" are both nonsense properties because these "numbers" are identities not numbers, and using a data type that treats them as numbers invites surprising, sometimes unpleasantly so, results.
Of course, because these are computers, all strings are ultimately numbers but their numberness is without real meaning.
However I read the post I responded to as decrying all floating-point formats, regardless of base. That leaves only fixed-point (fancy integers) and rationals. To represent numbers with the same range as double precision, you'd need about 2048 bits for either alternative. And rational arithmetic is really slow due to heavy reliance on the GCD operation.
If you print your variable with the %a format, then YES, the exact value is preserved and there is no loss of information. The problem is that the literal that you wrote cannot be represented exactly. But this is hardly a fault of the floats. Ints have exactly the same problem:
int x = 2.5; // x gets the value 2
int y = 7/3; // same thing