TNetstrings May 14, 2011
TNetstrings (http://tnetstrings.org/) could be awesome. Its simplicity kicks the ass of BSON (http://bsonspec.org/), another JSON-inspired serialization format. Additionally, it is *very* easy to parse. Unfortunately, it requires a schema if you want to serialize JavaScript Numbers. This is entirely fixable with only a very minor addition.
I propose TNetstrings adds a "." FP64 type.
Add IEEE 754 Number Type so it is isomorphic with JSON.
The most important reason to do this is because TNetstrings is currently failing in its stated objective to "allow for the same data structures as JSON". JSON only specifies 7 data types. One of those is the IEEE 754 double precision floating point Currently, TNetstrings has an "INT" type that is an ASCII-encoded integer, but it has no representation of a float.
Now, there are problems with serializing floats: they are large, and different languages may support many float types with varying precision. At 8 bytes, it is rather large, so TNetstrings should not remove the INT type as it would make for a fast and small presentation of numbers between 0 and 9999999. The remaining issue is then precision. The TNetstrings site says they would add a float type "if we can find the specification for that which doesn't determine accuracy." This is directly at odds with the stated goal to provide the same data structures as JSON.
Zed Shaw says
We'd add floats if we could come with a "reliable" spec. So far all we've found it "it's a fucking %f output, deal."
The IEEE 754 double precision floating point number is very well specified and widely supported. It is also the JS number type. TNetstrings could provide for arbitrary serialization of JSON objects if and only if it supports the humble double.
The current suggested solution is:
Send it as a string, have the receiver know it's a float. That's all we got for now.
There are two problems with this: requiring schema, and being an incomplete standard.
Requiring Schema
Requiring that the receiver know the structure of the object ahead of time means that any *deserializer* has to be able to accept some kind of schema in order to know which strings are actually floats in disguise. Alternatively, you could structure your code such that your application code walks the deserialized object and performs the string to float type conversion. Either way, it means that any infrastructure that would use deserialized objects has know the schema ahead of time.
In CS lingo, it means that TNetstrings is not isomorphic -- TNetstrings.deserialize(TNetstrings.serialize(object)).deepEqual(object) is false if any of your numbers have a fractional component (nonzero digits to the right of the decimal, like 1.2.) A serialization format that is self-describing with a type system is retarded if it is inconsistently isomorphic depending on the value of the data being serialized.
Requiring a schema complicates the interface and introduces an unnecessary burden on the programmer -- the type information is available at runtime, why require it at code time?
If TNetstrings had a float type, then it would be a viable "drop in" JSON replacement with no additional work. The difference between a drop-in replacement and a replacement that breaks unless you specify every place where a Number might be a Float is huge. It means that framework designers and library providers could use TNetstrings instead of JSON with no additional work for their consumers.
You are going to want floats if you are tracking averages, doing anything with lat/long, or modeling speed and direction in a game. Currently, I use JSON for IPC and for persistence. I'd *much* rather be able to use a faster standard if there was a drop-in replacement. However, I don't want to rework my code to include schemas if I can avoid it.
Incomplete Standard
Since TNetstrings does not specify a floating point type, it means that you also must have a standard for specifying the schema of the messages and a standard for a representation of the double type (though 8:, is the only sane one.) Since TNetstrings doesn't standardize the number-description schema, it means that application authors must provide their own -- which reduces interoperability and code-reuse.