Accessing Lightroom’s SQLite DB Directly

(This post is mostly for the search engines, to help others looking for the solution that this post provides)

Note that some of the details presented in this post, such as the database filename and the actual schema itself, are for the “beta 3” version of Lightroom, and don't apply to Lightroom Version 1.0 or later.

Lightroom, Adobe's high-volume photo workflow application, stores all kinds of user-preference and per-image data in an SQLite database. As of the Beta 3 release, the database is stored in one file, the Mac filename being /Users/username/Pictures/Lightroom/Lightroom B3 Library.aglib

If you're technically-minded and want to peek at the database directly, you might try to access it with the sqlite3 command, which comes standard with OSX, and be disappointed to find that it doesn't work:

$ sqlite3 'Lightroom B3 Library.aglib'
SQLite version 3.1.3
Enter ".help" for instructions
sqlite> .schema
Error: unsupported file format

Eric Scouten provided the answer in this thread on the Adobe's Lightroom Beta discussion forum. It turns out that Lightroom uses features of SQLite introduced in its version 3.2. OSX (as of 10.4.7) ships with SQLite version 3.1.3, and so it can't read Lightroom's database.

If you have Apple's XCode tools installed, you can download the source from sqlite.org and build a more recent version (3.3.6 is the latest as of this writing), and then it all works:

sqlite3 'Lightroom B3 Library.aglib'
SQLite version 3.3.6
Enter “.help” for instructions
sqlite> .schema
CREATE TABLE Adobe_AdditionalMetadata (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    image INTEGER,
    metadataPresetID,
    xmp
);
CREATE TABLE Adobe_FTPPresets (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    passive,
    password,
    path,
    port,
    protocol,
    server,
    title,
    username
);
CREATE TABLE Adobe_cameraRawSettings (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    name,
    rawSettings UNIQUE
);
CREATE TABLE Adobe_imageDevelopSettings (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    croppedHeight,
    croppedWidth,
    digest,
    fileHeight,
    fileWidth,
    grayscale INTEGER,
    historySettingsID,
    image INTEGER,
    settingsID,
    text,
    whiteBalance
);
CREATE TABLE Adobe_imageFiles (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    absolutePath NOT NULL DEFAULT '',
    idx_filename NOT NULL DEFAULT '',
    image INTEGER NOT NULL DEFAULT 0,
    importHash,
    lc_idx_filename NOT NULL DEFAULT '',
    markedMissing,
    md5,
    modTime,
    relativePath NOT NULL DEFAULT '',
    robustRepresentation
);
CREATE TABLE Adobe_imageProperties (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    image INTEGER,
    propertiesString
);
CREATE TABLE Adobe_images (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    aspectRatioCache,
    captureTime,
    developSettingsIDCache,
    fileHeight,
    fileWidth,
    hidden INTEGER NOT NULL DEFAULT 0,
    importedTime,
    orientation,
    originalCaptureTime,
    originalRootEntity INTEGER,
    panningDistanceH,
    panningDistanceV,
    photoBinNominators INTEGER,
    photoBinPosition,
    printSharpening,
    propertiesCache,
    pyramidIDCache,
    rating,
    rawSettings INTEGER,
    rootFile INTEGER,
    sidecarStatus
);
CREATE TABLE Adobe_libraryImageSavedSettings3 (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    dateCreated INTEGER,
    image INTEGER,
    isUserCheckpoint INTEGER NOT NULL DEFAULT 0,
    name,
    text,
    uuid
);
CREATE TABLE Adobe_namedIdentityPlate (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    description,
    identityPlate,
    identityPlateHash,
    moduleFont,
    moduleSelectedTextColor,
    moduleTextColor
);
CREATE TABLE Adobe_oldMetadataCache (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    image INTEGER,
    xmp
);
CREATE TABLE Adobe_variables (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    name,
    value
);
CREATE TABLE Adobe_variablesTable (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    name UNIQUE,
    value NOT NULL DEFAULT ''
);
CREATE TABLE AgLibraryContent (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    containingTag INTEGER,
    content,
    owningModule
);
CREATE TABLE AgLibraryImageSearchIndex (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    dirty INTEGER NOT NULL DEFAULT 0,
    image INTEGER NOT NULL DEFAULT 0,
    searchIndex
);
CREATE TABLE AgLibrarySpecialCollectionInfo (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    kind UNIQUE NOT NULL DEFAULT '',
    sortDirection,
    sortOrder
);
CREATE TABLE AgLibraryTag (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    dateCreated NOT NULL DEFAULT '',
    flattenedParams,
    genealogy NOT NULL DEFAULT '',
    imageCountFlag INTEGER NOT NULL DEFAULT 1,
    kindName NOT NULL DEFAULT '',
    lastApplied,
    name,
    parent INTEGER,
    sortDirection,
    sortOrder
);
CREATE TABLE AgLibraryTagImage (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    flagged,
    flattenedParams,
    image INTEGER NOT NULL DEFAULT 0,
    positionInCollection NOT NULL DEFAULT 'z',
    tag INTEGER NOT NULL DEFAULT 0,
    tagKind NOT NULL DEFAULT ''
);
CREATE TABLE AgLibraryTagSearchWord (
    id_local INTEGER PRIMARY KEY,
    id_global UNIQUE NOT NULL,
    implied INTEGER NOT NULL DEFAULT 0,
    tag INTEGER NOT NULL DEFAULT 0,
    word NOT NULL DEFAULT ''
);

sqlite> .quit
$

It would be prudent at this point to quote directly from Eric's post:

Of course, any mention of accessing the database file directly should come with the standard disclaimer that you're doing so at your own risk and you should make backup copies.

Upgrading my MacBook Disk Drive

Today it's much cooler, only 79F (26C) in the shade at the moment. The last few had been very hot, surpassing 100F (38C) and causing the threat of rolling blackouts. (Some places did have blackouts because local transformers couldn't handle the load.)

When bought my new MacBook, I opted for the 80 gig drive, rather than the standard 60 gig, to give myself plenty of space. Thus, it is with great shock that I find it's already getting full.

Part of it could be the 20GB of videos for Anthony, the 21GB of images from my brother's wedding last week, or the 12GB of unprocessed photos that piled up over the last few months while I concentrated on finishing my book. I had to delete some of the videos to make room for all the wedding pictures.

So, after less than three weeks, I've decided to upgrade my disk to a 160GB drive. My Apple-employee friend Jason suggested just getting an external drive case and a new drive, copying my data over, then swapping the drives. That way, I'd be able to use my old 80GB drive externally, as a backup or for rarely-needed data.

Putzing around on the web, I found an article about a new 2.5” SATA external drive box, Trans International's minixpress 825. It's tiny, doesn't require external power when connected via Firewire, and can handle USB as well.

It's $100 without a drive, but since I need a drive too, I thought I'd pay the extra $228 to have it come with a 160GB drive (a Hitachi TravelStar).

Knowing that they were planning the first shipment just last week, I called Trans International to check availability. It turns out that they're expecting more on Friday or Monday, so that's fine. I did get some great advice from the guy at Trans International: don't buy the unit with a disk installed, but rather buy them separately: that way, you get the full disk warranty, separate from the unit's shorter warranty.

The cost differential was $1, so it was certainly worth it, and I placed my order accordingly.

Thanks Trans International!


hot Hot HOT!

It was HOT today, hitting 103°F (39.5°C) here in Silicon Valley, staying there much of the afternoon. Opening the door to the outside felt like what I'd imagine opening the door to a blast furnace might feel like. At least the humidity was low, and, as an added bonus, there were no rolling blackouts, as we had a few years ago.

Today's heat set a new record high temperature for the date. The previous high was a relatively chilly 96°F (35.5°C).


Mastering Regular Expressions, Third Edition

I've finally finished the writing and production of the Third Edition of my book, Mastering Regular Expressions (published by O'Reilly Media). I'd been working on it since the early fall, and finished the day before my travels started earlier this month.

This third edition is 58 pages longer than the second edition, and now reaches 542 pages in length. The main changes from the second edition are a new, 48-page chapter on PHP, and a rewritten/expanded Java chapter taking into account the many java.util.regex changes between Java 1.4 and Java 1.5/Java 1.6.

I've not yet updated the book's web site to reflect MRE3 changes, but I'll get around to it before the book hits the stores (which should be around the end of next month).


Frustration with TracFone Prepaid Wireless

(wow, my third post of the day — rare for me)

My parents and sister have used TracFone prepaid wireless phones for years, so I thought I'd give them a try to fill my cell phone needs while in America. We'll be here for six weeks, and then from time to time in years to come.

There are three cost components to using TracFone: you must buy a phone, buy service time, and buy minutes. The latter two are combined into minutes that expire some months or years after you buy them.

Last weekend, while still in Ohio, I stopped by a Radio Shack and bought two Motorola V176 phones (they seemed small and were the flip-type, which I wanted), and two 60-minute/two-month airtime cards (if I don't add more minutes before the minutes or months run out, I lose the use of the phone until I buy more minutes). The total was about $110.

I charged the batteries and activated the phones via TracFone's web site, which involved entering some long ID numbers into the web form, then entering long strings of digits (encrypted commands) into the phones. Everything went smoothly, and my phone showed that I had 80 minutes of airtime (the 10 that comes with the phone, 10 extra for activating online, and the 60 extra that I'd bought). According to the web site, they should be ready to use any time between immediately and one business day.

On Sunday it wasn't working yet. The “one business day” hadn't passed, but I was in a fairly tech friendly area, so I thought it should probably be working, and called up tech support. After going through 5 levels of phone menu, I was immediately connected to a person. She was very polite, and after working with her and entering what seemed like endless numbers into the “code entry” screen of the phone, the phones knew their phone number.

She confirmed something that the guy at Radio Shack had told me (but the TracFone website neglected to mention), that the first area in which you use them becomes your “home area”. Since I'm going to be spending most of the summer in California, I wanted that to be my home area, so I had to wait until later that day, when I got to California, before I could use them.

Unfortunately, when I got to the airport in San Jose and needed them, they still didn't work.

To make this way too long story shorter, I called TracFone customer support at least twice a day since then, each time being told (in very polite but oddly scripted English) that “they should work within 3 to 4 business hours, and if not, call us back.” And each time, they didn't work, and each time I'd call them back, half the time finding that they've closed for the day.

This seems to be a great business model: sell a service people need, take their money, then don't provide the service.

I bought the phones and service because I need them, and it's been very inconvenient not having them. It's been all the more frustrating because the customer care people can't or won't deviate from their well scripted interaction. When I say “well scripted”, I mean exactly that – scripted. At times, they clearly don't even understand what they're saying. For example, one of the (very polite) ladies I talked to yesterday kept saying “I'm updating the following information on my computer” but didn't continue to say what that information was – after about the 9th time, I realized that she was meaning to say “please hold on while I update information on the computer.” Anyway, it only added to the frustration, and it happened over and over.

So I decided that if it wasn't fixed by today, I'd just bite the cost and buy something else. But today's customer-support guy actually had a new idea: turn on the phone, take out the battery while the phone is still on, remove the SIM card, and after 30 seconds, put it all back together. That did it! It finally worked.

Geez, why couldn't they have put that idea on their web site? Why couldn't they have suggested it earlier? It seems to be a generic kind of solution, not in any way particular to me. It was extremely frustrating. If you do a search on the web, you'll find many others with frustrating TracFone activation experiences — I wonder how many would have their problem solved if they tried this?

So, our phones are working. Frankly, though, I wouldn't choose the Motorola v176 again. Here's what the top part of the keypad looks like:

What in the heck do those buttons mean?

The intuitive use is that the center button is the “select” button, after using the arrow buttons to select from among menu items. But the center button is not the select button, it's the upper-left “dot” button (as opposed to the identically labeled upper-right “dot” button).

Motorola gets an “F” for usability.