The Joy of a Fast Photo Proximity Search in Lightroom
Boring Sunset Over Kyoto the Shogunzuka overlook (将軍塚) — Today — -- Shogunzuka Overlook (将軍塚) -- Kyoto, Japan -- Copyright 2013 Jeffrey Friedl, http://regex.info/blog/ -- This photo is licensed to the public under the Creative Commons Attribution-NonCommercial 3.0 Unported License http://creativecommons.org/licenses/by-nc/3.0/ (non-commercial use is freely allowed if proper attribution is given, including a link back to this page on http://regex.info/ when used online)
Nikon D4 + Nikkor 24-70mm f/2.8 @ 31mm — 1/60 sec, f/3.5, ISO 6400 — map & image datanearby photos
Boring Sunset Over Kyoto
the Shogunzuka overlook (将軍塚)
— Today —

I popped up to Kyoto's Shogunzuka Overlook (将軍塚) today for the first time since March, hoping for a nice sunset.

As you can see above, I didn't get it.

However, due to my blog's proximity search feature, we can follow the nearby photos link under the photo to see other shots from Shogunzuka that I've posted over the years, including:

June 2012 -- Shogunzuka (将軍塚) -- Kyoto, Japan -- Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Nikkor 70-200mm f/2.8 @ 70mm — 1/320 sec, f/6.3, ISO 5000 — map & image datanearby photos
June 2012
November 2011 -- Shogunzuka (将軍塚) -- Kyoto, Japan -- Copyright 2011 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/320 sec, f/11, ISO 320 — map & image datanearby photos
November 2011
July 2007 -- Kyoto, Japan -- Copyright 2007 Jeffrey Eric Francis Friedl, http://regex.info/blog/
Nikon D200 + Nikkor 70-200mm f/2.8 @ 70mm — 1/1500 sec, f/3.2, ISO 640 — map & image datanearby photos
July 2007
November 2011 -- Shogunzuka (将軍塚) -- Kyoto, Japan -- Copyright 2011 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/8000 sec, f/2.5, ISO 200 — map & image datanearby photos
November 2011
November 2008 -- Shogunzuka (将軍家) -- Copyright 2008 Jeffrey Friedl, http://regex.info/blog/ -- This photo is licensed to the public under the Creative Commons Attribution-NonCommercial 3.0 Unported License http://creativecommons.org/licenses/by-nc/3.0/ (non-commercial use is freely allowed if proper attribution is given, including a link back to this page on http://regex.info/ when used online)
Nikon D700 + Nikkor 24-70 mm f/2.8 @ 70mm — 1/200 sec, f/2.8, ISO 3200 — map & image datanearby photos
November 2008

I added the proximity-search feature to my blog several years ago, and I use it often in various ways because proximity search can be extremely useful, but my blog proximity search is obviously limited to photos I've actually published on my blog. My full catalog of photos in Adobe Lightroom is much, much bigger.

Over the years I've written a bazillion (~45) plugins for Lightroom that supports a photographer in all kinds of helpful ways, but one of the first (already five years ago!) was my Lightroom catalog Proximity Search plugin.

To take advantage of proximity search, your photos have to be geoencoded to begin with (that is, each photo must be associated with latitude/longitude coordinates of where it was shot). At the time I first released the plugin, there was no way to geoencode photos within Lightroom, so one had to somehow take care of it before importing photos into your catalog. It was a pain.

But, while working on that plugin I figured out a tricky way to build another plugin that allows you to geoencode within Lightroom, and my Geoencoding Support plugin was released a couple of weeks later; it remained the only way to geoencode within Lightroom until Adobe introduced the Map Module last year. (My Geoencoding Support plugin is all the more useful now that Lightroom supports location editing, because it extends the usefulness of location information far beyond what Lightroom provides.)

Anyway, I was happy that I could do a proximity search on my photos, but the unfortunate reality was that Lightroom's catalog interface for plugins was simply too slow, creating a high barrier to use: I'd use it sparingly, only when it was worth the several-minute wait for a result.

Still, even today it's better than what Lightroom itself now supplies, at least on my machine. If I set Lightroom's Map Module to an area I'm interested in, then switch to Library and select All Photographs, then go back to Map to see which ones show up, Lightroom completely locks up for eight minutes. Locks up. Eight minutes. Painful.

(I hear that Lightroom is faster in this respect on Windows; someone who tested my catalog said the lockup was only two minutes there.)

I don't understand why plugin access to the Lightroom catalog is so slow, but to gain some insight I tried accessing the Lightroom-catalog SQLite database directly. The results? The same search that locked up Lightroom for eight minutes took 0.6 seconds.

Even though the very first Lightroom-related article on my blog was a post in 2006 about accessing the Lightroom database directly, I've purposefully stayed away from doing so within my plugins as a matter of principle, keeping instead to the official plugin infrastructure.

But come on, an 800-fold speedup is just too much to pass up, especially for a feature that blooms in usefulness when you can use it on the spur of the moment with little friction. So for the first time in my plugin-development life, I did an end around Lightroom's interface, adding a Fast Full-Catalog Proximity Search plugin-extra feature to my Proximity Search and Geoencoding Support plugins last week. This is the fast search that I mentioned on my previous post.

The search is nominally invoked from the File > Plugin Extras > Geoencoding Support > Fast Full-Catalog Proximity Search menu, but on my system I mapped it to a keyboard shortcut, so while looking at an image that's geoencoded, a quick tap brings up this dialog:

Upon activation, the plugin goes outside of Lightroom to grab the data, then import just those results back in, creating a collection with them. It took just a few seconds to isolate the 548 photos from that general area over the years.

(Update: in modern versions, it works on Windows as well.)
One bummer about the workaround that achieves this: it doesn't work on Windows, nor on Lr4 or earlier, so in those situations the Fast Full-Catalog Proximity Search plugin-extra item reverts to the slower, official, much-less-compelling method. )-:

Still, if you're using Lr5 on a Mac and find this useful, please let Adobe know; perhaps they'll add this kind of thing directly into Lightroom.


All 3 comments so far, oldest first...

Hi Jeffrey,

sqlite databases can be opened in WAL mode (Write-Ahead Logging).
see http://www.sqlite.org/wal.html
With “PRAGMA journal_mode=wal;”, executed from an sqlite shell, this mode can be made persitent for a databse. This would allow access to the Lightroom catalog from more than one process concurrently.
However, if Lightroom is started with a WAL-mode catalog which is NOT already opened by an other process, it resets the journal_mode to “delete” and prevents access to the catalog from other processes. To get around this “limitation” I am using the following cmd script to start Lightroom:

—script starts here—
start sqlite3.exe -init initcmds LightroomCatalog5.lrcat
start “” LightroomCatalog5.lrcat
—script ends here—

The file initcmds as a parameter for the sqlite3.exe init option has the following contents:
—initcmds starts here—
PRAGMA journal_mode=wal;
.table
—initcmds ends here—

This script opens the catalog in WAL mode, executes the sqlite table command, which creates the files LightroomCatalog5.lrcat-wal and LightroomCatalog5.lrcat-shm, and then starts Lightroom.

This time Lightroom seems to honor the already opened catalog in WAL mode and allows access to the catalog from other processes.

I know, that this is not a solution for the majority of Lightroom users, but I am really wondering if you can use it to implement your “Fast Full-Catalog Proximity Search” on Windows. So please let me know

Greetings from Berlin
Jacques

Good info to know, thank you, but yeah, not something for the masses. I suppose I can try to detect whether read mode is supported and enable it if so… I’ll give it some thought… —Jeffrey

— comment by Jacques on October 12th, 2013 at 9:51pm JST (10 years, 6 months ago) comment permalink

Can you think of a rational reason Adobe implemented the search the way they did?

My guess is that full-catalog proximity search was not their main consideration when creating the various features that, when used together, let you do full-catalog proximity search. —Jeffrey

— comment by Zak on October 13th, 2013 at 9:18am JST (10 years, 6 months ago) comment permalink

Yeah, but is there an actual benefit to doing it their way? Or are you just diplomatically saying they just didn’t care?

I’m not so privy to the internals to know whether optimizing for this particular need would have detrimental effects on other things; more likely, this particular whole-catalog need would benefit from separate API hooks, that one can diplomatically say haven’t yet made the priority cut. —Jeffrey

— comment by Zak on October 15th, 2013 at 11:23am JST (10 years, 6 months ago) comment permalink
Leave a comment...


All comments are invisible to others until Jeffrey approves them.

Please mention what part of the world you're writing from, if you don't mind. It's always interesting to see where people are visiting from.

IMPORTANT:I'm mostly retired, so I don't check comments often anymore, sorry.


You can use basic HTML; be sure to close tags properly.

Subscribe without commenting