At first, I thought that the main data wrangling task in personal geotagging was going to be cleaning the GPS data. Sometimes the GPS is too coarse, sometimes it’s off because not enough satellites were in view, sometimes the GPS was just not turned on when I took the photo.

But, in the course of resetting my cameras’ clocks to standard time today, I realized that I’m going to have to wrangle the timestamps on the photos as well. I had thought that one-hour increments due to daylight savings and time-zone travel would be the main problem, but now I see that the minutes matter, too. Camera clocks don’t sync with the network or with GPS (at least not on my relatively ancient cameras), and I can move a significant distance in the three or seven minutes by which the clock has drifted away from GPS time.

I’m going to have to be able to apply short time offsets to the timestamps of all photos taken by a given camera on a given day. Setting the correction is going to require precisely recognizing at least one key location in each batch of photos in order to compute the offset.

That’s for past photos. For the future, I can get in the habit of either correcting the clock frequently or of snapping a reference shot with my cellphone along with each set of camera shots.

Space-time is tricky.

Since I’m starting with photos that are already timestamped, I want to be able to look up their timestamps in some sort of GPS database to see whether there’s a recorded time/location event near that time. Unfortunately, my initial perusal of a couple of geo-extensions to well known databases (SpatialLIte, PostGIS), show a generous helping of geometry-oriented query capabilities but no obvious time-query capabilities.

It could be that time-indexing is so simple compared with spatial indexing that what I’m looking for is just swamped in the documentation, or it could be that I’ll have to dig some more.

In a perfect world, this would be a relatively simple matter of looking up each photo’s creation date and time, finding that date and time in my years of GPS logs, and associating my GPS location for that date and time with the photo.

The world isn’t perfect. I took a lot of photos before I got a GPS device. Sometimes the device’s batteries give out or I forget to turn it on. There may be time zone and daylight-savings issues that make nonsense of the date-and-time link between the photo and the location, and I need to find and fix them.

Even if I manage to bring perfection to my existing data sets, I had better design a smooth-enough workflow for my future photos that I’ll actually keep up, and of course I’ll need to correct future mistakes and omissions.

Further, tagging the photos really becomes useful if I put them on maps, so I need a tool to make located map icons that point to the photos. I already tag my photos with broad categories, and I’d like to be able to vary the icon according to the category, or to sort the categories into separate map layers.

These are the basic goals. I’ll be blogging more as I design and build this thing.

The human genome has just over 3 billion base pairs in about 25 thousand genes. This is a large enough data set that it gets algorithms of its own.

Chris Anderson of The Long Tail has an important post about how the metadata used in some music-listening applications doesn’t satisfy the listeners needs:

[...] classical is a genre that the one-size-fits-all music aggregators such as iTunes don’t handle particularly well. They’re oriented around pop music, with its artist, album, track data format. Meanwhile classical music organizes around composer, conductor, performer, soloist

He also voices my exact peeve about how jazz is treated:

However, neither of them does a very good job with Jazz, where the individual musicians are often more meaningful than the band.

Yup. No reasonable cataloguer of jazz recordings separates “Thelonious Monk Trio” from “Thelonious Monk Quartet” from “Thelonious Monk”. At the same time, it’s important to be able to locate all appearances of Thelonious Monk, regardless of whether he was the leader of the session (note that “leader” and “session” are appropriate terms in jazz discography, but not for pop or classical).

When your only tool is a hammer…

I can’t help but wonder if the problems Chris calls out in iTunes come from the poor selection of data tools in most applications programmers’ toolkits. Relational databases, the current orthodox storage technique, favor using one or more tables, each consisting of records having the same selection of attributes. There are hacks you can use to simulate having, say, jazz tracks and pop tracks in the same Tracks relation, but hacks and simulations tend to twist one’s code, so most programmers resist going there.

An XML database in every toolbox!

We don’t really have to live this way anymore. With the popularity of XML for data interchange, the tools ecology has given us a variety of XML database systems. The XML data model has the flexibility to represent varying record structures: in fact, it has much more flexibility than we need for the purpose!

Heretical as it may seem to put the cart of an interchange format before the horse of data abstraction, the XML situation is very useful in practice, at least for databases of moderate size. The W3C has come up with the XPath and XML Query specifications that provide excellent query mechanisms for data represented in the XML model. XML Query in particular is designed to look somewhat familiar to the hardened SQL user. There’s data typing taken from the XML Schema datatype recommendataion as well.

Better nails

Anyhow, let’s learn to design with a more flexible hammer, and maybe we’ll be able to hit a wider class of nails, rather than our users’ thumbs!

March is International Runaway Metaphor Month.

OpenStreetMap is a wiki-like project to build a world map using contributed GPS tracks [OpenGeoData pointed me there]. Their map of Baghdad is here.

This project is truly a product of the early 21st century: it requires GPS satellites, cheap but accurate GPS receivers, the World Wide Web, inexpensive computers with fast color graphics, and so forth.

And like all modern geographic applications, it also exploits a special property of GPS’s information domain: everyone agrees on the meaning of geographical location; only dates and times have a similar level of standardization. In relational-database terminology, this means that any table with a date or location column has a meaningful join with any other.

This doesn’t work with most data. I’ve had driver’s licenses in four U.S. states, but you can’t aggregate my driving record from the state records because they all use different ID numbering schemes (nice for my privacy in this case).

Also noteworthy is the fact that GPS information can be used to put a time dimension into maps, since we can tell when the street is used as well as where it is. There are some very pretty examples at Cabspotting.