Hidden Gems in Lightroom 4.1; Hidden Gems in Lightroom Plugin Development
NOTE: Images with an icon next to them have been artificially shrunk to better fit your screen; click the icon to restore them, in place, to their regular size.
desktop background image of a peaceful temple garden view at the Koumyou-in Temple (光明院), Kyoto Japan  --  Be My Guest... Koumyou-in Temple (光明院), Kyoto Japan  --  Koumyou-in Temple (光明院)  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/400 sec, f/4, ISO 800 — map & image datanearby photos
Be My Guest...
Koumyou-in Temple (光明院), Kyoto Japan
Vertical Desktop-Background Versions
1050×1680  ·  1200×1920  ·  1600×2560

So, Adobe has released the first official update for Lightroom 4, Lightroom 4.1. The release announcement on Adobe's Lightroom Journal blog lists the major changes, bug fixes, and new camera support (including recent heavy hitters like the Nikon's D4 and D800, and Canon's 1D X and 5D Mark III). However, the release notes don't even mention two changes that will be welcome and important for many users, so I want to tell you a bit about them here.

(The photos on this post, from my first visit earlier today to the small but elegant Koumyou-in Temple (光明院) in eastern Kyoto, have nothing to do with the prose of the post. They're just pretty pictures.)

desktop background image of a peaceful temple garden view at the Koumyou-in Temple (光明院), Kyoto Japan  --  Japanese Aesthetic  --  Koumyou-in Temple (光明院)  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Nikkor 24mm f/1.4 — 1/400 sec, f/10, ISO 2000 — map & image datanearby photos
Japanese Aesthetic
Desktop-Background Versions
1280×800  ·  1680×1050  ·  1920×1200  ·  2560×1600

The Problem

Upload Progress in Lr4.0 and earlier

Historically in Lightroom, when uploading one's photos to an online service such as Facebook, Flickr, etc., one sees a little progress bar in the upper left that chunks along with each picture, such as the one shown at right. With my plugins, the progress bar makes two steps for each image, one after Lightroom has rendered the copy to be uploaded, and another after the copy has actually been uploaded. However, there's no movement to the progress bar while the image is actually being rendered, and while the image is actually being uploaded. That's just how Lightroom has always worked.

So, during these potentially time-consuming processes, there's no indication that anything is happening. This is all the more regrettable when working with large videos, as they can take quite a long time to render and upload.

But even worse, the way Lightroom worked internally, large uploads were extremely inefficient. This was perhaps fine for most folks' images, but became a real problem with videos with sizes stretching into the hundreds of megabytes. On my machine, Lightroom would simply crash if I tried to upload a video weighing in at more than 30 or 40 megabytes.

Built-in Navigation this guy's socks reminded me of this photo on my recent post about my wife's ballet shoes  --  Koumyou-in Temple (光明院)  --  Kyoto, Japan  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/125 sec, f/2.5, ISO 6400 — map & image datanearby photos
Built-in Navigation
this guy's socks reminded me of this photo on my recent post about my wife's ballet shoes
(Update some years later: the guy I met that day with the socks turned out to become a good friend)

The Solution

I made a suggestion to Adobe about how to allow plugin uploads to be more efficient, and they added this support in Lightroom 4.1. The result is extremely efficient, and plugins that make use of this support can now upload without file-size limit, at least as far as Lightroom is concerned. If your network and upload destination can handle a terabyte video, so can Lightroom.

While testing this stuff with a sneak-peek version of Lightroom that Adobe kindly provided to me last week, I realized that one of the by-products of using this new internal support was that I could have the plugin update the progress bar in real time while the upload progressed.

Upload Progress in Lr4.1

In the progress bar at right, notice the tiny little “7%” that's been added? That's the only difference from before, but it's huge from a user-interaction point of view. Now while the upload is happening, you can watch that percent-complete value march up, and you can see the progress bar creep toward completion. It's such a small thing that makes a surprisingly big difference: you no longer have to wonder whether the thing is stuck or not, waiting for the progress bar to jump once the upload has completed.

desktop background image of a peaceful temple garden view at the Koumyou-in Temple (光明院), Kyoto Japan  --  Contemplation Koumyou-in Temple (光明院), Kyoto Japan  --  Koumyou-in Temple (光明院)  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Nikkor 50mm f/1.4 — 1/400 sec, f/1.4, ISO 200 — map & image datanearby photos
Koumyou-in Temple (光明院), Kyoto Japan
Desktop-Background Versions
1280×800  ·  1680×1050  ·  1920×1200  ·  2560×1600

The Joy

I spent the majority of the weekend working on this. It required writing a bunch of new code to use the new internal upload-handling stuff, and then retooling the plugins to work it in. It wasn't always smooth: I filed four separate bugs with Adobe on things related to this.

Sometimes I make murmurs about what I don't like about plugin development, most recently in “Ongoing Pitfalls of My Lightroom Plugin Development”, but also over the years in posts like “What To Do When a Hobby Becomes Work” and the most-unfortunate “Breakdown: When Good Intentions Don’t Scale”. But the days I spent wrestling with this were a joy because I was back getting my hands dirty with complex code that ended up actually working (I was able to work around most of the bugs), and the first time I tested a large upload and actually watched the progress bar slowly progress from 1% to 100% was positively giddy.

Positively giddy.

I don't expect many folks to understand that, but it was gratifying both as a user of Lightroom (the whole “no longer need to wonder” thing), and as a developer because I overcame obstacles to solve the real problem of handling large uploads. Before, some users would have trouble at 10-megabyte uploads, but now, 10 gigabytes would be a breeze.

desktop background image of a dainty purple propeller-shaped flower, in Kyoto Japan  --  Dainty Propeller  --  Koumyou-in Temple (光明院)  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/400 sec, f/2.5, ISO 1800 — map & image datanearby photos
Dainty Propeller
Desktop-Background Versions
1280×800  ·  1680×1050  ·  1920×1200  ·  2560×1600

So far I've pushed this change only in my uploader for PicasaWeb, which is also used a lot by folks for G+.

(UPDATE: I've now pushed this update for my Flickr, SmugMug, Zenfolio, Facebook, and Photobucket plugins as well.)

Google currently allows videos up to 100 megabytes, and I tested right up to the limit without problems. (Note: PicasaWeb specs note a maximum video of 1GB, but that's only when uploaded via Google's own Picasa desktop software; third-party apps are limited to 100MB.)

There is one caveat about this new support in the PicasaWeb plugin: due to one bug that I was not able to work around, the new support is used only for initial uploads, and not for image replacements (such as when republishing changes). This is unfortunate, but not as dire as it would be if Google allowed videos to be replaced. Google does not allow videos to be replaced, so the plugin continues to quietly ignore video updates.

I'll be pushing support in my other plugins soon, but I wanted to write this post right away to share a small slice of my geeky photographer plugin-developer joy.

desktop background image of rock and moss in a peaceful temple garden scene at the Koumyou-in Temple (光明院), Kyoto Japan  --  Alone Together  --  Koumyou-in Temple (光明院)  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/400 sec, f/2.5, ISO 320 — map & image datanearby photos
Alone Together
Vertical Desktop-Background Versions
1050×1680  ·  1200×1920  ·  1600×2560

Another Important Lr4.1 Update: Respecting “Private” Locations

Lightroom 4's new map module lets you put a circle around an area and assign it a name and, optionally, mark that this location is “private”. When you export images located within that private location, the geoencoded coordinates are removed from the exported copy, thus, in theory, preserving your privacy.

However, with some photo-hosting sites (PicasaWeb, SmugMug, and Flickr come to mind), the uploading app explicitly tells the site the image's map coordinates while uploading, and herein lies the problem: in Lr4.0, a plugin could always see a photo's location, but had no idea whether the user had marked it as “private”. Thus, private locations were removed from the copy uploaded, but the location was still sent along with the image to be displayed to the world. So much for “private”.

Lr4.1 includes infrastructure support that allows a plugin to inspect whether a photo's location is “private”, so, my plugins will be updated soon to support this. The PicasaWeb plugin already does, sporting new options to control if/when location data is sent to Google.

Teamwork  --  Koumyou-in Temple (光明院)  --  Kyoto, Japan  --  Copyright 2012 Jeffrey Friedl, http://regex.info/blog/
Nikon D700 + Voigtländer 125mm f/2.5 — 1/400 sec, f/2.5, ISO 640 — map & image datanearby photos

Photos continued here...

All 10 comments so far, oldest first...

The percentage indicator is very welcome! I’ve recently purchased a used D3s and the first (and only) time I uploaded a video on Picasa I thought your plugin did not supported it… I was (strangely) patient enough to understand that it was more a rendering/upload problem rather than a fault in your plugin. I’ll test the new indicator right away 😉

The render step is unchanged… just like the user, the plugin must just sit and wait, so no progress bar there. It’s the upload (which could potentially take much longer than the render) that is now reported in smooth, real-time detail. —Jeffrey

— comment by Gianluca on May 30th, 2012 at 11:20pm JST (12 years ago) comment permalink

For what it’s worth I, at least, totally understand that giddyness. Sometimes I think the most depressing thing about doing software development is that the most gratifying things you do are somewhere between invisible and taken for granted by the end user. “You made it as efficient as possible in doing that thing? Well sure, why wouldn’t you?” Nobody knows our pain!

Okay maybe that last bit was a bit over the top. But I get you; those little touches are what really makes stuff like your plugins a pleasure and they’re well appreciated.

Now where’s your 500px plugin? 🙂

— comment by Don on May 31st, 2012 at 1:17am JST (12 years ago) comment permalink

Wow, you are fast to adapt when you ask for comments:) I see the option for the size of the pictures. Thanks for sharing this, I know how you must be proud of you. I use technology to help people with disability and when I can get around a problem I’m pretty proud of the finding. I wish I had your knowledge of technology to get around stuff (but I’m not willing to learn and put all the hard work you must have put in it).
It’s a real pleasure to read your blog. I wish you a wonderful day.
I’m from Québec, Canada.

— comment by Pesc on May 31st, 2012 at 2:01am JST (12 years ago) comment permalink

Dear Jeffrey Friedl,

I want to say thank you for this post and for all of your hard work. There are two things here that really please me:

1. I love the way that you push Lightroom and the engineering team further. Your personal quest to make this software better and better plus your willingness to share the results is awesome. I haven’t tried using Lightroom to upload large video files but I have encountered the “frozen progress bar” trouble when uploading large blocks of images to Flickr, Picasssa, etc. So a personal thanks for tackling this challenge and helping me out.
2. I think that you and the Adobe engineering team deserve a big thanks from all of us for your willingness to pour endless hours into making our lives easier. It’s great to know that some of your API requests and suggestions do get filled eventually! Better uploads via Lightroom plugins probably won’t help Adobe’s bottom line this quarter but its great to know that there are folks like you who care and who try!

Best regards,

David Marx

— comment by David Marx on May 31st, 2012 at 2:52am JST (12 years ago) comment permalink

What a fun post. Nerd excitement is some of the best there is to witness. I’ve always enjoyed NASA and other engineers and scientists’ moments of success. Thanks.

— comment by Andrew Webb on May 31st, 2012 at 3:52am JST (12 years ago) comment permalink

I’m currently living in Colorado, just North of Boulder, but I was born in Tokyo in 1967.

— comment by Andrew Webb on May 31st, 2012 at 3:53am JST (12 years ago) comment permalink

Congrats! Geeky joy is perfectly fine with me… 😉


— comment by Mike Nelson Pedde on May 31st, 2012 at 1:31pm JST (12 years ago) comment permalink

Hi Jeffrey,

Nice blog post and thank you for keeping us posted on the behind the scene work!

Just wondering about the private data handling. It is kind of cute if a flag indicates that GPS data is private, but if a plugin doesn’t respect this flag… Shouldn’t Lr asked the user if a plugin may access the private location data like it is done in a smart phone? Just wondering… Anyway, this is something for Adobe to think about…

Keep up the good work!

– Daniel

I understand what you’re saying, but since the plugin has complete reign to inspect/delete any file on your entire system (any file, at least, that the user running Lightroom can access), little gates like you suggest aren’t really very meaningful. Plugins are full partners with Lightroom, limited only by what hooks Adobe has had the resources to build. (Those limitations are large, unfortunately, but they’re borne from a lack of manpower at Adobe). —Jeffrey

— comment by Daniel on June 1st, 2012 at 4:14am JST (12 years ago) comment permalink

No giddyness over the release of the SDK? That’s my source of giddyness tonight. 😉

Also, with regards to the changes to the location support, I’m inclined to agree with Daniel – If a location is private, Lightroom shouldn’t really be giving the plugin a private location by default. I’d want plugin developers to have to explicitly request it (and implicitly acknowledge that the user’s requested that the location is private) rather than blindly take all the info possible.

Of course, it’s entirely possible to get around as you mention (extract data from the XMP sidecar, and so on), but that would be kludgey, and likely prone to breakage, which given that I’d want to make it hard for a plugin to get at private location data, is possibly a good thing.

By all means, submit a request to Adobe about it. Privacy issues are always given high priority, so they may well listen. I’ll also mention the concern to my contacts. —Jeffrey

— comment by Kyle on June 1st, 2012 at 9:55pm JST (12 years ago) comment permalink

Wonderful work Jeff. I am wondering if I missed it or not. do any of your Plugins work for
Google +?



There’s no specific G+ API yet, so the best we can do at this point is to upload to Google’s PicasaWeb (sometimes referred to by Google as “Google Photos”). David Marx made a series of videos detailing how to do this with my PicasaWeb plugin. —Jeffrey

— comment by John Daly on July 22nd, 2012 at 5:21am JST (11 years, 10 months ago) comment permalink
Leave a comment...

All comments are invisible to others until Jeffrey approves them.

Please mention what part of the world you're writing from, if you don't mind. It's always interesting to see where people are visiting from.

IMPORTANT:I'm mostly retired, so I don't check comments often anymore, sorry.

You can use basic HTML; be sure to close tags properly.

Subscribe without commenting