Simple JSON Encode/Decode in Pure Lua
I've coded up some simple JSON encode/decode routines in pure Lua and thought I'd share them in case anyone else would find them useful. I use them in Adobe Lightroom, but they're pure Lua 5, so can be used anywhere Lua is.
JSON Encode/Decode in Pure LUA by Jeffrey Friedl is licensed under a Creative Commons Attribution 3.0 Unported License.
Full docs and changelog are in the code itself, but basic use is:
JSON = (loadfile "JSON.lua")() -- one-time load of the routines local lua_value = JSON:decode(raw_json_text) -- decode example local raw_json_text = JSON:encode(lua_table_or_value) -- encode example local pretty_json_text = JSON:encode_pretty(lua_table_or_value) -- "pretty printed" version
Enjoy.
(You might also be interested in this comparison of Lua JSON packages.)
Leave a comment...
HI Jeffrey,
When decoding a 900K json file, json lib will cost 14M+ memory. I review json.lua, and find use many stringcat .. operation at grok_string function.
so I change VALUE = VALUE .. c to tmp_stream:insert(c). diff code is here: https://gist.github.com/lvzixun/80e5b900b82059ebf5d7/revisions. using table.concat would be faster and reduce string memory allocation.
you can review code : https://gist.github.com/lvzixun/80e5b900b82059ebf5d7 😉
Unfortunately, whatever this “insert” function is, it’s not available in the version of Lua that I use. —Jeffrey
Forked this repo for LuaRocks. Do you have GitHub account for this?
https://github.com/jiyinyiyong/json-lua
No, sorry, no GitHub. —Jeffrey
Hi Jeffrey,
thanks for the code. This is exactly what I was looking for. Works like a charme – in a Lightroom plugin!
So far, only simple JSON responses are decoded (did this with a regexp before), but it will probably become more important when using more sophisticated WebAPIs of the Synology PhotoStation.
Greetings from Berlin,
Martin
Hi Jeffrey,
Thanks a lot for this code ! Excellent work ! We are using this serilizer together with Zabbix JSON-RPC API in a large distributed monitoring network. This code provided us with an excellent jumpstart at first when we made a prototype of the gateway from Zabbix to internal ESB. After a while we have decided not to touch anything as it works perfectly and performance is sufficient for this project.
Thanks a lot again,
Greetings from Kazakhstan,
Pavel.
Hello
I am new to Lightroom plugin development: I can’t find the correct syntax to load your JSON.lua file. I have put the JSON.lua file in the plugin folder. I have tried some code found on this page:
JSON = (loadfile “JSON.lua”)()
JSON = assert(loadfile “JSON.lua”)()
JSON = loadfile(LrPathUtils.child(_PLUGIN.path, “JSON.lua”))
without success.
Could you please give the correct syntax to use (Lightroom 6 Plugin SDK ).
Thanks a lot
Thierry
In Lightroom you can use JSON = require "JSON.lua", but note that Lightroom won’t recognize any file added to a plugin folder after it starts, so after having added the JSON.lua file, you have to restart Lightroom before anything will work. —Jeffrey
During encode(), numbers larger than 1e14 were being changed to scientific notation.
Here is some lua code to show how tostring() handles large numbers
> x=100000000000001
> print(tonumber(x));
1e+14
> print(string.format(“%.0f”,x))
100000000000001
I changes the encoding to use the “%.0f” format -> works great
Here is the GIT DIFF
@@ -837,7 +837,10 @@ function encode_value(self, value, parents, etc, options, indent)
—
return “-1e+9999”
else
– return tostring(value)
+ return string.format(“%.0f”, value);
end
Ah, good catch, but unfortunately it’s not quite that simple. Try with your x value set to 10000000000054321, 10000000000054322, and 10000000000054321+1 and you’ll see (if your system is the same as mine) that at some point the string.format() version is wrong, due, I’m sure, to precision being lost in encoding big numbers in too-small a number of bits. I’d rather see that loss of precision represented, instead of a wrong number presented precisely.
There’s a range of decimals for which string.format() is indeed better, as your example shows; if I can determine for sure such cases, I’ll incorporate a change for them. I just tried the obvious way and it was lacking, but I’ll dig further.
Of course, we still need to make sure that a number like 3.14159 is output correctly.—Jeffrey
I am not able to do loadfile with LUA called from HAProxy config
JSON = (loadfile “JSON.lua”)()
The program exists for loadfile call.
The caller lua and JSON.lua are placed in same folder.
Is there specific configuration needed for HAProxy?
Any thoughts or ways around having decode properly handling nil values from a table?
For instance:
{
“somevalue”: null
}
will create a empty output since lua doesn’t support nil values as a key in a table element.
I’ve locally (and horribly) solved this by patching if nil to return a function that returns nil — but looking at the table I have to account for that too — then on the encode I just say if it’s a function then it is nil — good enough for me, but not for a general stance.
local function grok_object(self, text, start, etc)
[…]
VALUE[key] = new_val or function() return nil end
function encode_value(self, value, parents, etc, options, indent)
if type(value) == ‘function’ then value = nil end
if value == nil then
return ‘null’
Other than this small hiccup, I really love how the rest works… Efficient and stable!
As of version 20160526.15 I’ve added a “null” field to the encoding-options table. If your Lua table has a string value with the same value as the “null” option, it gets encoded into JSON as a raw null. Using null="\0" might be appropriate, for example. —Jeffrey
Thanks a lot for this! I’m using to create and load save files for a game I’m working on, and it works great. One small suggestion would be to ignore functions while parsing, rather than throwing an error. This is only my opinion, of course.
Anyway, I am quite happy with this library. It was the first result when I searched for “lua json parser.”
As an aside, what do you use this library for in Lightroom?
Since you asked where I’m from: Portland, Oregon.
I’m not sure what you mean by “ignore functions while parsing”; JSON has no concept of “functions”. I use this in my Lightroom plugins to handle responses from a variety of photo-hosting-site’s APIs. —Jeffrey
I have found a bug when decoding an empty object, it is interpreted as an empty array, at least that is what shows on encode().
look at the field: ‘attributes’, it is an object in the string and then an array on decode()
lua
> JSON = (loadfile “./JSON.lua”)();– one-time load routines
> x='{“id”:”UserInfo_BILL”,”actions”:[],”attributes”:{}}’
> y=JSON:decode(x)
> print(JSON:encode(y));
{“actions”:[],”attributes”:[],”id”:”UserInfo_BILL”}
As mentioned in the docs, it’s handled correctly if you “
JSON.strictTypes = true“. It’s not the default because supporting it requires adding metatables to the data, which can break some systems into which Lua is embedded. —JeffreyThank you for making this module!
I think I have spotted an unintended global. On line 516, you have:
> local isNumber = {
> __index = isNumber,
> — <snip>
> }
However, when __index = isNumber on line 517 is parsed, nothing has been assigned to the local isNumber variable on line 516 yet, so at that point isNumber is treated as an uninitialised global, and __index is assigned a value of nil. (At least, this is how it works in Lua 5.1.)
I haven’t followed all the paths through the code to see if this causes any problems, but I guess that this isn’t what was intended.
Holy cow, you’re right. What a nubie mistake on my part. Good catch, thanks. Fixed. —Jeffrey
Hi!
This is an awesome module, thanks a lot!
Wanted to know, sadly in the project I’m testing, ‘the order’ of some keys in the json file needs to be preserved, my question is, is it possible when reading a json to keep the original order in which they were read?
decoding/encoding the same file changes the order of key-value pairs.
It’s not currently possible, sorry. Lua tables have no order for non-numeric keys, so to preserve the order you’d have to use metatables somehow, which just opens up a can of worms for handling that metadata (e.g. what to do when new keys are added in Lua). —Jeffrey
Hello again, and thank you for the updates to the module dealing with trailing garbage. I’ve spotted another nil global bug in the new code, so I thought I would let you know.
On line 1065, you have this variable declaration:
> local error_message
This declaration is in the scope of the if block starting at line 1060.
> if next_i ~= #text + 1 then
The problem is that on line 1070 you try and use error_message:
> return value, error_message
But at that point we have left the scope of the if block, so error_message is a nil global.
If you could update the code to fix this, it would be much appreciated.
Oops, sorry, I fixed this earlier in the month but goofed up pushing out the fixed version here. It (version 20161109.21) is here now. —Jeffrey
Hey, do you have the old versions 1-20 somewhere online?
It would be cool to have a look at them (push them to git for proper version history/changes etc).
Cheers
You can append the version string to the download link to get old ones, e.g. http://regex.info/code/JSON.lua-20140920.13 —Jeffrey
Wow Jeffrey,
Thank you a million times for this library.
Jon
local stock= ‘{“open”:332.44,”close”:329.87,”high”:332.5,”low”:328.15,”volume”:11038600.0,”actual_close”:331.29}’
local s = JSON:decode(stock)
return s[“high”]
This gives me a result of 332
or if I add JSON.decodeNumbersAsObjects = true I get: (empty list or set)
What do I need to get the right answer, 332.5?
Are you sure you’re not using it in some kind of integer context once you get the value back from whatever function this is? I changed your ‘return’ line to a ‘print’ line, and I got the proper 332.5. —Jeffrey
You’re right, it turns out Redis turns floats into integers. Not sure about the behavior of JSON.decodeNumbersAsObjects = true.
Thanks for the library, I’m new to lua so still asking stupid questions.
hi, Jeffrey
I add a ‘array_newline’ option, which show array
[11,22,[33,44]]
as
[
| 11,
| 22,
| [
| | 33,
| | 44
| ]
]
https://github.com/yurenchen000/json-lua/commit/e870c6e702deacd3b9c5c2b1135893791fc51e16
thank you~
Seems like a good idea… I’ve added it, thanks! —Jeffrey
Hi Jeffrey,
Thanks for your module! I am running into a problem and I cannot figure out how to solve it. I would appreciate some help, please.
I have a json object being returned from a server, which seems to be properly decoded into a LUA table as I can reference it’s properties. One of the properties contains another json like string and I am trying to convert it to a table but keep getting “JSON.decode must be called in method format.” Here is the complete JSON string:
{“Success”:true,”GalleryId”:20094,”postURL”:”https://upload.blah.net/upload/processUpload/?websiteId=5140\u0026eventId=6500″,”FileUploadId”:”b5ymbczi”,”UMIs”:”{\”IMG_3271.JPG\”:\”109678\”,\”IMG_3272.JPG\”:\”109679\”,\”IMG_3273.JPG\”:\”109680\”,\”IMG_3274.JPG\”:\”109681\”}”}
I can isolate the last property called response.UMIs and print it out as the following string.
2017-06-05 18:59:19 +0000, TRACE UMIs as string: {“IMG_3271.JPG”:”109678″,”IMG_3272.JPG”:”109679″,”IMG_3273.JPG”:”109680″,”IMG_3274.JPG”:”109681″}
UMIS = {}
When I try UMIs = JSON.decode(response.UMIs) is when I get the error.
The JSON string is verified by a JSON linter so I am not sure what I am doing wrong.
Thanks!
Use “JSON:decode” rather than “JSON.decode” and it should work. —Jeffrey
Thanks man, this is exactly what I needed. I was just messing around with modding a game that I am playing at the moment. I was looking to export some data as JSON, so that I persist state between gaming sessions. This did the trick perfectly. You’ve saved me a bunch of time, and I really appreciate that you make this free to use.
Thank you!
I’d like to request support for boolean keys, right now it says “can’t encode table with a key of type boolean”. A simple fix is to have
…
if type(key) == ‘boolean’ then
table.insert(string_keys, tostring(key))
elseif type(key) == ‘string’ then
…
Makes sense. Done. —Jeffrey
Using VERSION = ‘20170416.23’
At line 1325:
local encode_value – – must predeclare because it calls itself
function encode_value
Actually, you don’t need to predeclare a recursive function in Lua. The interpreter includes the function in its own _ENV before calling the function. You only need to predeclare pairs of recursive functions, like this:
local a
local function b()
a()
end
function a()
b()
end
I know about that property of Lua and use it elsewhere in the file, so I suppose there was something esoteric happening at one point during development that made me feel the need to do it explicitly. The Lua docs are pretty clear that the two expressions are identical, so I’m at a loss to guess what it might have been, but in any case, you’re right and I’ve tidied it up. Thanks. —Jeffrey
Using VERSION = ‘20170416.23’
I’m using you package for debugging code, so I need to encode table which includes functions. For my purposes, I just added this to encode_value():
elseif type(value) == ‘function’ then
return tostring(value)
However, as a suggestion, I think it would be nice if you either:
1- included some kind of types whitelist as an option for encoodin; or
2- made JSON:onEncodeError() interactive, as is JSON:onDecodeError(); or
3- both of the above.
Anyways, I’m quite happy with this package. Very nice implementation, quick, efficient and very useful.
Good idea, thanks. Check out unsupportedTypeEncoder() in the new version I just posted. —Jeffrey
Hi Jeffrey,
Great package! We’ve been using it for a while now and it works pretty well. This seems to have been mentioned before but as of Version 20170823.25 (version 25: August 23, 2017) we haven’t been able to pass a null option for *decoding* JSON strings. This is useful to us because we need to retain keys in the lua table (since there is no distinction between keys that don’t exist and keys assigned as nil values). Having null configurable when encoding a JSON string is not sufficient because we decode, modify, and then subsequently encode it again. The patch we made was as follows:
1086: - return nil, start + 4
1086: + return options.null, start + 4
Is there any chance that this patch could be surfaced for the next release?
Cheers,
Max
Oops, yeah, that was mentioned four years ago… I completely missed it, sorry. Thanks for the bump. I’ve just pushed the update. —Jeffrey
I’m trying to call JSON:decode() from inside an a startAsyncTask() call. From time to time the string passed might not contain valid JSON and decode() will raise an error. Lightroom, though, buries errors inside tasks which means I have no way to catch the error. Checking the string for valid JSON before passing to decode isn’t trivial. I wondered if you had anything to suggest.
many thanks.
You can trap such errors by using LrFunctionContext.postAsyncTaskWithContext() and immediately adding a cleanup or failure handler to the context passed to the new task. —Jeffrey
The encode/decode process is causing loss of precision for floating point numbers. I changed the tostring() call in encode _value to string.format(‘%20.16g’,value) to fix the problem.
Could you give an example where the original code was losing precision? —Jeffrey
Thank you very much for perfectly solving the problem of loss of precision caused by big number deserialization; Nginx performance loss is also small. Example of use:
JSON = require(“JSON”)
JSON.decodeNumbersAsObjects = true
lua_value = JSON:decode(‘{“num”:10201348204923842304}’)
raw_json_text = JSON:encode(lua_value)
print(raw_json_text)
It is amazing work. Thank you for the sharing!
Hi Jeffrey,
I will report an error when parsing. I think utf8 coding is not supported
“[string “chunk”]:688: [string “chunk”]:194: incomplete utf8 sequence at byte 16 of: {“materielid”:”塑胶原”,”materielcode”:”610.43420″}”
PS: I’m from China.
It works for me when I try your example, but it’s possible that the encoding was changed by WordPress. Could you email a ZIP file of some sample data that you encounter the error with? jfriedl@yahoo.com Thanks. —Jeffrey
Hi Jeffrey,
Great library, thanks for sharing. I’m curious is there a way to make encoding happen in the order I create a table in Lua so keys are not sorted in alphabetic order? Let say I have a table t = {z = “some value for z”, a = “some value for a”} if I’ll encode it I’m going to get {“a”:”some value for a”, “z”: “some value for z”}
Thank you.
No, I don’t think that’s possible. As far as I know, there is no temporal information available in Lua tables to indicate any kind of added/modified order. —Jeffrey