Simple JSON Encode/Decode in Pure Lua
I've coded up some simple JSON encode/decode routines in pure Lua and thought I'd share them in case anyone else would find them useful. I use them in Adobe Lightroom, but they're pure Lua 5, so can be used anywhere Lua is.
JSON Encode/Decode in Pure LUA by Jeffrey Friedl is licensed under a Creative Commons Attribution 3.0 Unported License.
Full docs and changelog are in the code itself, but basic use is:
JSON = (loadfile "JSON.lua")() -- one-time load of the routines local lua_value = JSON:decode(raw_json_text) -- decode example local raw_json_text = JSON:encode(lua_table_or_value) -- encode example local pretty_json_text = JSON:encode_pretty(lua_table_or_value) -- "pretty printed" version
Enjoy.
(You might also be interested in this comparison of Lua JSON packages.)
Leave a comment...
When I add this line to a plugin I am writing I get an error from Lightroom
MyPluginApi:28 cannot open JSON:lua: No such file or directory
The thing that makes no sense to me is that the JSON.lua file is located in the same directory as MyPluginApi.lua
Do you have an example plugin that might show the usage of this JSON library working?
Thanks
It looks as if you have a colon before ‘lua’ rather than a dot. —Jeffrey
Hi, I’ve recently noticed your module and included it in my comparison of JSON modules on the Lua wiki. Your module is one of the better pure-Lua JSON implementations and I guess I wouldn’t have needed to write my own if I had found yours earlier. I hope you don’t mind the inclusion in the list.
Thanks for the link, and for the tests. I’ve pushed a new version that addresses some of the problems you found in your tests. Thanks! —Jeffrey
Hi,
thanks for this. I’ve just checked out the file, but I didn’t find any info regarding the license type, just your copyright notice.
From a legal perspective, if the code comes without a license everyone can have a look but no one can use it, for any type of project. So are there any strings attached?
Cheers,
Michael
Feel free to use it as you like. —Jeffrey
I really like to use your module, but I can’t unless there is a license that clearly grants me permission to use it. For example MIT, like lua license itself, or BSD 2-clause.
Best,
Maik
I made it and I said you can use it. That should be enough. —Jeffrey
Err… something seems to be up with the comments in the file:
Lines 14 and 123 have
-- JSON = (loadfile "JSON.lua")() -- one-time load of the routines
While lines 24 and 111 have
-- JSON = (loadfile "JSON:lua")() -- one-time load of the routines
That JSON.lua / JSON:lua is a bit inconsistent… Is that intentional? (Considering that Lightroom just gave me the error “attempt to call a nil value” when I tried loading JSON:lua, I doubt it…)
Also, THANK YOU for publishing this. It’s making my first foray into Lightroom plugin development (and Lua) so much easier!
(While I’m at it, any pointers on using relative paths in loadfile(), or, alternatively, finding out with directory is Lightroom’s working directory so I can build a relative path from that? Doing loadfile(JSON.lua) falls over with
File not found
, but putting the absolute path in works fine, but… won’t work for anyone but me, which is not an optimal solution.And Lightroom’s file access is too fast to show up in Process Explorer, so I can’t get anything from there.)
Oops, thanks, I thought I fixed that once, but apparently not in my master source, so it reared its ugly head again. It should be dot-“lua” in all situations, so I’ve fixed it (again), thanks. As for the path stuff, try loadfile(LrPathUtils.child(_PLUGIN.path, “JSON.lua”)) or the like. —Jeffrey
Hi, i want to use this library in my project. Could you please add some example how to use it. I have a json string. First i want to extract it to an array then to an object. and then get a string with the key “text”
I’m new to lua, some exapmles would be neat
I just included your JSON.lua file in my project: https://github.com/etu/i3luastatus. I hope it’s fine.
Sure, it’s fine, you’re not the only one these days. I realize, though, that there was no link back to this page, so it’s harder for folks to check for updates, so I’ve added a link in the code directly back to this page. —Jeffrey
Hey,
It says “Version 20130120.6 (version 6, Jan 20, 2013)”, but in the changelog at the bottom of the linked file, it stops at version 5. Is it really the 6th version ? And if yes, what’s different from version 5 ?
Thank you.
The only difference is in a comment, so there’s no real difference. Not sure why it didn’t update before, but it’s updated now; thanks for the heads up. —Jeffrey
Jeffrey,
Thanks – great module.
Would you mind putting the ‘Feel free to use this any way you like” comment in the code (or just say you are placing in the public domain) rather than just in the blog. That would make your intentions clear to anyone auditing code later, which sadly gets done.
Alex
I’ve updated the code and the web page to reflect a Creative Commons “CC-BY” license, which is what I really should have done from the beginning. —Jeffrey
Hi Jeffrey,
I’m new to lua (since yesterday) and I was playing around with your nice module.
I’m runnin into a few issues / unclarities though and hope to get them clarified.
The pretty_encode does not generate the same output as the encode when it comes to ‘arrays’.
Consider the following:
local Test = {1, 2, 3, [5]=5} -- would be a table with Test[4] = nil
-- encode output: [1, 2, 3, null, 5]
-- pretty_encode output: [1, 2, 3]
Secondly, in the comments you put:
-- We need to inspect all the keys... if there are any strings, we'll convert to a JSON
-- object. If there are only numbers, it's a JSON array.
Which would indicate that
Test = { [1] = "a", [2] = "b", ["3"] = "c" }
results in a JSON object (ther is ‘any strings’), instead i get a ‘mixed keys’ error.
Hope you can help me out,
Mattie (Netherlands)
Thanks for the heads up on these… I’ve pushed a new version that addresses both issues. —Jeffrey
Hey there from Spain!
I just wanted to thank you for releasing this, it literally saved my life, as I am using Lua in a closed environment for a project, and the functions they provided me to work with JSON were pretty lame.
And I must also say thanks for your pure lua sha1 implementation! ; )
Regards.
Hi Jeffrey
Thanks for your JSON module. I stumbled across the following issue (using version 20131118.9):
JSON:decode(“[1,null,3]”) silently skips the “null” value and result in a table with two values only: {1, 3}.
This seems to be because of the problem described here: http://lua-users.org/wiki/StoringNilsInTables
The following one-line change would allow an application to define a nilPlaceholder in your module:
— JSON.lua.old 2013-11-18 13:07:36.000000000 +0100
+++ JSON.lua 2013-12-20 14:47:28.636625937 +0100
@@ -506,7 +506,7 @@
return false, start + 5
elseif text:find(‘^null’, start) then
– return nil, start + 4
+ return JSON.nilPlaceholder, start + 4
else
self:onDecodeError(“can’t parse JSON”, text, start, etc)
For example:
local NIL = {}
JSON.nilPlaceholder = NIL
The calling application could then compare the received values with NIL. If nilPlaceholder is not defined, it is simply “nil”.
Thanks and keep up the good work!
Oliver (Switzerland)
Yikes, I just (April 2014) noticed this comment sitting in moderation. Sorry. It turns out that I did notice a comment of someone else reporting the bug many months later, and have just (April 2014) pushed a fix. The fix doesn’t require any placeholder thing… it just works. Sorry for the delay(!) —Jeffrey
Hello Jeffrey,
Thank you very much for making your code available.
When trying to decode a malformed JSON:
your code gets to line 550:
and fails. Although it says it should not get there, it’s there.
Anyway I think the right code should be:
To be able to catch the exception. Do you agree?
Best regards.
Good catch… the “should never get here” comment is true only if the user’s JSON.assert actually breaks out of the pcall, as it does when I use it in Adobe Lightroom, so it’s not an appropriate comment for the general package, so I’ve updated it, and the code as you suggest (with s/message/value/). Thanks for the heads up. —Jeffrey
Hi Jeffrey,
When decoding [“1″,null,null,null,null,null,”marathon”], it returns an array of two elements ({1,marathon}).
It may be a normal behavior, but it would be helpful if it returns an array of 7 elements with nil values (something like {1,nil,nil,nil,nil,nil,marathon}).
Thank you for your work, it saved me a lot of time, here in Canada 🙂
Good catch, thanks. I’ve pushed a fix. —Jeffrey
Nice job: thank you!
HI Jeffrey,
When decoding a 900K json file, json lib will cost 14M+ memory. I review json.lua, and find use many stringcat .. operation at grok_string function.
so I change VALUE = VALUE .. c to tmp_stream:insert(c). diff code is here: https://gist.github.com/lvzixun/80e5b900b82059ebf5d7/revisions. using table.concat would be faster and reduce string memory allocation.
you can review code : https://gist.github.com/lvzixun/80e5b900b82059ebf5d7 😉
Unfortunately, whatever this “insert” function is, it’s not available in the version of Lua that I use. —Jeffrey
Forked this repo for LuaRocks. Do you have GitHub account for this?
https://github.com/jiyinyiyong/json-lua
No, sorry, no GitHub. —Jeffrey
Hi Jeffrey,
thanks for the code. This is exactly what I was looking for. Works like a charme – in a Lightroom plugin!
So far, only simple JSON responses are decoded (did this with a regexp before), but it will probably become more important when using more sophisticated WebAPIs of the Synology PhotoStation.
Greetings from Berlin,
Martin
Hi Jeffrey,
Thanks a lot for this code ! Excellent work ! We are using this serilizer together with Zabbix JSON-RPC API in a large distributed monitoring network. This code provided us with an excellent jumpstart at first when we made a prototype of the gateway from Zabbix to internal ESB. After a while we have decided not to touch anything as it works perfectly and performance is sufficient for this project.
Thanks a lot again,
Greetings from Kazakhstan,
Pavel.
Hello
I am new to Lightroom plugin development: I can’t find the correct syntax to load your JSON.lua file. I have put the JSON.lua file in the plugin folder. I have tried some code found on this page:
JSON = (loadfile “JSON.lua”)()
JSON = assert(loadfile “JSON.lua”)()
JSON = loadfile(LrPathUtils.child(_PLUGIN.path, “JSON.lua”))
without success.
Could you please give the correct syntax to use (Lightroom 6 Plugin SDK ).
Thanks a lot
Thierry
In Lightroom you can use JSON = require "JSON.lua", but note that Lightroom won’t recognize any file added to a plugin folder after it starts, so after having added the JSON.lua file, you have to restart Lightroom before anything will work. —Jeffrey
During encode(), numbers larger than 1e14 were being changed to scientific notation.
Here is some lua code to show how tostring() handles large numbers
> x=100000000000001
> print(tonumber(x));
1e+14
> print(string.format(“%.0f”,x))
100000000000001
I changes the encoding to use the “%.0f” format -> works great
Here is the GIT DIFF
@@ -837,7 +837,10 @@ function encode_value(self, value, parents, etc, options, indent)
—
return “-1e+9999”
else
– return tostring(value)
+ return string.format(“%.0f”, value);
end
Ah, good catch, but unfortunately it’s not quite that simple. Try with your x value set to 10000000000054321, 10000000000054322, and 10000000000054321+1 and you’ll see (if your system is the same as mine) that at some point the string.format() version is wrong, due, I’m sure, to precision being lost in encoding big numbers in too-small a number of bits. I’d rather see that loss of precision represented, instead of a wrong number presented precisely.
There’s a range of decimals for which string.format() is indeed better, as your example shows; if I can determine for sure such cases, I’ll incorporate a change for them. I just tried the obvious way and it was lacking, but I’ll dig further.
Of course, we still need to make sure that a number like 3.14159 is output correctly.—Jeffrey
I am not able to do loadfile with LUA called from HAProxy config
JSON = (loadfile “JSON.lua”)()
The program exists for loadfile call.
The caller lua and JSON.lua are placed in same folder.
Is there specific configuration needed for HAProxy?
Any thoughts or ways around having decode properly handling nil values from a table?
For instance:
{
“somevalue”: null
}
will create a empty output since lua doesn’t support nil values as a key in a table element.
I’ve locally (and horribly) solved this by patching if nil to return a function that returns nil — but looking at the table I have to account for that too — then on the encode I just say if it’s a function then it is nil — good enough for me, but not for a general stance.
local function grok_object(self, text, start, etc)
[…]
VALUE[key] = new_val or function() return nil end
function encode_value(self, value, parents, etc, options, indent)
if type(value) == ‘function’ then value = nil end
if value == nil then
return ‘null’
Other than this small hiccup, I really love how the rest works… Efficient and stable!
As of version 20160526.15 I’ve added a “null” field to the encoding-options table. If your Lua table has a string value with the same value as the “null” option, it gets encoded into JSON as a raw null. Using null="\0" might be appropriate, for example. —Jeffrey
Thanks a lot for this! I’m using to create and load save files for a game I’m working on, and it works great. One small suggestion would be to ignore functions while parsing, rather than throwing an error. This is only my opinion, of course.
Anyway, I am quite happy with this library. It was the first result when I searched for “lua json parser.”
As an aside, what do you use this library for in Lightroom?
Since you asked where I’m from: Portland, Oregon.
I’m not sure what you mean by “ignore functions while parsing”; JSON has no concept of “functions”. I use this in my Lightroom plugins to handle responses from a variety of photo-hosting-site’s APIs. —Jeffrey
I have found a bug when decoding an empty object, it is interpreted as an empty array, at least that is what shows on encode().
look at the field: ‘attributes’, it is an object in the string and then an array on decode()
lua
> JSON = (loadfile “./JSON.lua”)();– one-time load routines
> x='{“id”:”UserInfo_BILL”,”actions”:[],”attributes”:{}}’
> y=JSON:decode(x)
> print(JSON:encode(y));
{“actions”:[],”attributes”:[],”id”:”UserInfo_BILL”}
As mentioned in the docs, it’s handled correctly if you “
JSON.strictTypes = true
“. It’s not the default because supporting it requires adding metatables to the data, which can break some systems into which Lua is embedded. —JeffreyThank you for making this module!
I think I have spotted an unintended global. On line 516, you have:
> local isNumber = {
> __index = isNumber,
> — <snip>
> }
However, when __index = isNumber on line 517 is parsed, nothing has been assigned to the local isNumber variable on line 516 yet, so at that point isNumber is treated as an uninitialised global, and __index is assigned a value of nil. (At least, this is how it works in Lua 5.1.)
I haven’t followed all the paths through the code to see if this causes any problems, but I guess that this isn’t what was intended.
Holy cow, you’re right. What a nubie mistake on my part. Good catch, thanks. Fixed. —Jeffrey
Hi!
This is an awesome module, thanks a lot!
Wanted to know, sadly in the project I’m testing, ‘the order’ of some keys in the json file needs to be preserved, my question is, is it possible when reading a json to keep the original order in which they were read?
decoding/encoding the same file changes the order of key-value pairs.
It’s not currently possible, sorry. Lua tables have no order for non-numeric keys, so to preserve the order you’d have to use metatables somehow, which just opens up a can of worms for handling that metadata (e.g. what to do when new keys are added in Lua). —Jeffrey
Hello again, and thank you for the updates to the module dealing with trailing garbage. I’ve spotted another nil global bug in the new code, so I thought I would let you know.
On line 1065, you have this variable declaration:
> local error_message
This declaration is in the scope of the if block starting at line 1060.
> if next_i ~= #text + 1 then
The problem is that on line 1070 you try and use error_message:
> return value, error_message
But at that point we have left the scope of the if block, so error_message is a nil global.
If you could update the code to fix this, it would be much appreciated.
Oops, sorry, I fixed this earlier in the month but goofed up pushing out the fixed version here. It (version 20161109.21) is here now. —Jeffrey
Hey, do you have the old versions 1-20 somewhere online?
It would be cool to have a look at them (push them to git for proper version history/changes etc).
Cheers
You can append the version string to the download link to get old ones, e.g. http://regex.info/code/JSON.lua-20140920.13 —Jeffrey
Wow Jeffrey,
Thank you a million times for this library.
Jon
local stock= ‘{“open”:332.44,”close”:329.87,”high”:332.5,”low”:328.15,”volume”:11038600.0,”actual_close”:331.29}’
local s = JSON:decode(stock)
return s[“high”]
This gives me a result of 332
or if I add JSON.decodeNumbersAsObjects = true I get: (empty list or set)
What do I need to get the right answer, 332.5?
Are you sure you’re not using it in some kind of integer context once you get the value back from whatever function this is? I changed your ‘return’ line to a ‘print’ line, and I got the proper 332.5. —Jeffrey
You’re right, it turns out Redis turns floats into integers. Not sure about the behavior of JSON.decodeNumbersAsObjects = true.
Thanks for the library, I’m new to lua so still asking stupid questions.
hi, Jeffrey
I add a ‘array_newline’ option, which show array
[11,22,[33,44]]
as
[
| 11,
| 22,
| [
| | 33,
| | 44
| ]
]
https://github.com/yurenchen000/json-lua/commit/e870c6e702deacd3b9c5c2b1135893791fc51e16
thank you~
Seems like a good idea… I’ve added it, thanks! —Jeffrey
Hi Jeffrey,
Thanks for your module! I am running into a problem and I cannot figure out how to solve it. I would appreciate some help, please.
I have a json object being returned from a server, which seems to be properly decoded into a LUA table as I can reference it’s properties. One of the properties contains another json like string and I am trying to convert it to a table but keep getting “JSON.decode must be called in method format.” Here is the complete JSON string:
{“Success”:true,”GalleryId”:20094,”postURL”:”https://upload.blah.net/upload/processUpload/?websiteId=5140\u0026eventId=6500″,”FileUploadId”:”b5ymbczi”,”UMIs”:”{\”IMG_3271.JPG\”:\”109678\”,\”IMG_3272.JPG\”:\”109679\”,\”IMG_3273.JPG\”:\”109680\”,\”IMG_3274.JPG\”:\”109681\”}”}
I can isolate the last property called response.UMIs and print it out as the following string.
2017-06-05 18:59:19 +0000, TRACE UMIs as string: {“IMG_3271.JPG”:”109678″,”IMG_3272.JPG”:”109679″,”IMG_3273.JPG”:”109680″,”IMG_3274.JPG”:”109681″}
UMIS = {}
When I try UMIs = JSON.decode(response.UMIs) is when I get the error.
The JSON string is verified by a JSON linter so I am not sure what I am doing wrong.
Thanks!
Use “JSON:decode” rather than “JSON.decode” and it should work. —Jeffrey
Thanks man, this is exactly what I needed. I was just messing around with modding a game that I am playing at the moment. I was looking to export some data as JSON, so that I persist state between gaming sessions. This did the trick perfectly. You’ve saved me a bunch of time, and I really appreciate that you make this free to use.
Thank you!
I’d like to request support for boolean keys, right now it says “can’t encode table with a key of type boolean”. A simple fix is to have
…
if type(key) == ‘boolean’ then
table.insert(string_keys, tostring(key))
elseif type(key) == ‘string’ then
…
Makes sense. Done. —Jeffrey
Using VERSION = ‘20170416.23’
At line 1325:
local encode_value – – must predeclare because it calls itself
function encode_value
Actually, you don’t need to predeclare a recursive function in Lua. The interpreter includes the function in its own _ENV before calling the function. You only need to predeclare pairs of recursive functions, like this:
local a
local function b()
a()
end
function a()
b()
end
I know about that property of Lua and use it elsewhere in the file, so I suppose there was something esoteric happening at one point during development that made me feel the need to do it explicitly. The Lua docs are pretty clear that the two expressions are identical, so I’m at a loss to guess what it might have been, but in any case, you’re right and I’ve tidied it up. Thanks. —Jeffrey
Using VERSION = ‘20170416.23’
I’m using you package for debugging code, so I need to encode table which includes functions. For my purposes, I just added this to encode_value():
elseif type(value) == ‘function’ then
return tostring(value)
However, as a suggestion, I think it would be nice if you either:
1- included some kind of types whitelist as an option for encoodin; or
2- made JSON:onEncodeError() interactive, as is JSON:onDecodeError(); or
3- both of the above.
Anyways, I’m quite happy with this package. Very nice implementation, quick, efficient and very useful.
Good idea, thanks. Check out unsupportedTypeEncoder() in the new version I just posted. —Jeffrey
Hi Jeffrey,
Great package! We’ve been using it for a while now and it works pretty well. This seems to have been mentioned before but as of Version 20170823.25 (version 25: August 23, 2017) we haven’t been able to pass a null option for *decoding* JSON strings. This is useful to us because we need to retain keys in the lua table (since there is no distinction between keys that don’t exist and keys assigned as nil values). Having null configurable when encoding a JSON string is not sufficient because we decode, modify, and then subsequently encode it again. The patch we made was as follows:
1086: - return nil, start + 4
1086: + return options.null, start + 4
Is there any chance that this patch could be surfaced for the next release?
Cheers,
Max
Oops, yeah, that was mentioned four years ago… I completely missed it, sorry. Thanks for the bump. I’ve just pushed the update. —Jeffrey
I’m trying to call JSON:decode() from inside an a startAsyncTask() call. From time to time the string passed might not contain valid JSON and decode() will raise an error. Lightroom, though, buries errors inside tasks which means I have no way to catch the error. Checking the string for valid JSON before passing to decode isn’t trivial. I wondered if you had anything to suggest.
many thanks.
You can trap such errors by using LrFunctionContext.postAsyncTaskWithContext() and immediately adding a cleanup or failure handler to the context passed to the new task. —Jeffrey
The encode/decode process is causing loss of precision for floating point numbers. I changed the tostring() call in encode _value to string.format(‘%20.16g’,value) to fix the problem.
Could you give an example where the original code was losing precision? —Jeffrey
Thank you very much for perfectly solving the problem of loss of precision caused by big number deserialization; Nginx performance loss is also small. Example of use:
JSON = require(“JSON”)
JSON.decodeNumbersAsObjects = true
lua_value = JSON:decode(‘{“num”:10201348204923842304}’)
raw_json_text = JSON:encode(lua_value)
print(raw_json_text)
It is amazing work. Thank you for the sharing!
Hi Jeffrey,
I will report an error when parsing. I think utf8 coding is not supported
“[string “chunk”]:688: [string “chunk”]:194: incomplete utf8 sequence at byte 16 of: {“materielid”:”塑胶原”,”materielcode”:”610.43420″}”
PS: I’m from China.
It works for me when I try your example, but it’s possible that the encoding was changed by WordPress. Could you email a ZIP file of some sample data that you encounter the error with? jfriedl@yahoo.com Thanks. —Jeffrey
Hi Jeffrey,
Great library, thanks for sharing. I’m curious is there a way to make encoding happen in the order I create a table in Lua so keys are not sorted in alphabetic order? Let say I have a table t = {z = “some value for z”, a = “some value for a”} if I’ll encode it I’m going to get {“a”:”some value for a”, “z”: “some value for z”}
Thank you.
No, I don’t think that’s possible. As far as I know, there is no temporal information available in Lua tables to indicate any kind of added/modified order. —Jeffrey