It’s a tough problem: if you do things (like scanning entire libraries) at Google-scale, you just can’t pay attention to the details. One partial way out (which Geoff mentions) is to allow users to submit corrections, as Google Maps does for positions of placemarks.

The article addresses a number of important points about the provenance and usefulness of metadata, and Google employees provide some great comments and discussion.

(Via Brad DeLong).

He names the two approaches “structure-and-browse” and “collect-and-query”. I feel like I’ve been groping for these terse descriptions for years!

Chris Anderson of The Long Tail has an important post about how the metadata used in some music-listening applications doesn’t satisfy the listeners needs:

[...] classical is a genre that the one-size-fits-all music aggregators such as iTunes don’t handle particularly well. They’re oriented around pop music, with its artist, album, track data format. Meanwhile classical music organizes around composer, conductor, performer, soloist

He also voices my exact peeve about how jazz is treated:

However, neither of them does a very good job with Jazz, where the individual musicians are often more meaningful than the band.

Yup. No reasonable cataloguer of jazz recordings separates “Thelonious Monk Trio” from “Thelonious Monk Quartet” from “Thelonious Monk”. At the same time, it’s important to be able to locate all appearances of Thelonious Monk, regardless of whether he was the leader of the session (note that “leader” and “session” are appropriate terms in jazz discography, but not for pop or classical).

When your only tool is a hammer…

I can’t help but wonder if the problems Chris calls out in iTunes come from the poor selection of data tools in most applications programmers’ toolkits. Relational databases, the current orthodox storage technique, favor using one or more tables, each consisting of records having the same selection of attributes. There are hacks you can use to simulate having, say, jazz tracks and pop tracks in the same Tracks relation, but hacks and simulations tend to twist one’s code, so most programmers resist going there.

An XML database in every toolbox!

We don’t really have to live this way anymore. With the popularity of XML for data interchange, the tools ecology has given us a variety of XML database systems. The XML data model has the flexibility to represent varying record structures: in fact, it has much more flexibility than we need for the purpose!

Heretical as it may seem to put the cart of an interchange format before the horse of data abstraction, the XML situation is very useful in practice, at least for databases of moderate size. The W3C has come up with the XPath and XML Query specifications that provide excellent query mechanisms for data represented in the XML model. XML Query in particular is designed to look somewhat familiar to the hardened SQL user. There’s data typing taken from the XML Schema datatype recommendataion as well.

Better nails

Anyhow, let’s learn to design with a more flexible hammer, and maybe we’ll be able to hit a wider class of nails, rather than our users’ thumbs!

March is International Runaway Metaphor Month.

OpenStreetMap is a wiki-like project to build a world map using contributed GPS tracks [OpenGeoData pointed me there]. Their map of Baghdad is here.

This project is truly a product of the early 21st century: it requires GPS satellites, cheap but accurate GPS receivers, the World Wide Web, inexpensive computers with fast color graphics, and so forth.

And like all modern geographic applications, it also exploits a special property of GPS’s information domain: everyone agrees on the meaning of geographical location; only dates and times have a similar level of standardization. In relational-database terminology, this means that any table with a date or location column has a meaningful join with any other.

This doesn’t work with most data. I’ve had driver’s licenses in four U.S. states, but you can’t aggregate my driving record from the state records because they all use different ID numbering schemes (nice for my privacy in this case).

Also noteworthy is the fact that GPS information can be used to put a time dimension into maps, since we can tell when the street is used as well as where it is. There are some very pretty examples at Cabspotting.

There’s a great discussion about the relative advantages of the two approaches on the blog. Basically, storing the tags in the file makes the association harder to lose as you move the file around, and storing the tags separately avoids modifying the user’s data file. Neither one is obviously in accord with the user’s intention in all cases.

I think the issue has whole extra layers of subtlety. We perceive metadata that is stored within a data file as being what Jon Udell calls “file truth”. Since there’s only one set of metadata stored in the file, it becomes the One True Metadata. On the other hand, metadata stored in a separate database reads as the opinion of the maintainer of the database. This is exactly what social bookmarking systems such as del.icio.usdo: each attribution of a tag to a URL is also associated with a user making that attribution.

A pluralistic society requires a separate metadatabase!

This isn’t just another engineering tradeoff, though. The truth about “file truth” is that it’s still an opinion—the opinion of the last agent to modify the metadata within the file. When there’s One True Metadata, we can only represent disagreements by obliterating the last guy’s assertion.

Imagine trying to tag a scan of a photo taken at your parents’ wedding of someone you don’t recognize. You think it’s Dad’s college roommate, but your sister thinks it’s Mom’s second cousin. You have one “person depicted” slot: do you fight over it? Do you leave it blank and explain the situation in a semantically bland catch-all description field? Or do you each tag it as you will in your respective databases?

Not only is it unrealistic to allow for only one true description of a file, it’s also time we stopped regarding metadata as lost forever just because it’s not stored in the file. We could set up a distributed database that works like Gracenote‘s CD identification database, but for all files instead of just music files. As with CDs, the lookup key for a file can be generated by anyone who possesses the file (by applying a secure hash), but the particular metadata obtained depends on which tagger’s part of the repository is consulted. It’s all doable, and it would eliminate blogstorms about how evil application X erases user metadata.

Such documents are often enormous. They’re written in standardese (often badly). They’re usually written by committees. They go through a maze of twisty little revisions, all different.

But worst of all, they often bury their novelty in a sea of details that resemble those in the last spec I reviewed.

I’d like to do for data formats and other information representations what the Gang of Four book does for programs: call out and label the patterns that come up over and over again so that I can classify details into bigger chunks for mental processing.

You can expect to see several different kinds of post in this series:

• Case studies. I have to look at lots of actual data formats in order to discern the patterns!
• Data format patterns. Most posts will be about patterns I find in data formats…
• Information usage patterns. …but some posts will be about how information is generated, stored, and used.
• Other. I’ll probably think of some other topics as well.

I expect to look at simple cases, such as comma-separated values, as well as fiendishly complex cases, such as PDF. Programming-language syntaxes are fair game; database index disk structures are right out. In between, I’ll draw the boundary as interest dictates.

This series will be open-ended as long as people keep inventing data formats faster than I can look at them.

Lots of books will teach you how to process data with particular technologies, but Kent’s book goes deeper. He shows in chapter after chapter how database practice fails to match the way humans actually use information.

Data and Reality is almost thirty years old, but the issues haven’t really changed: if anything, they’re much more in our collective faces.

This book may be for you if:

• you feel strongly that you and your Social Security number (U.S. tax identification) are not the same thing,
• you wonder whether Mark Twain and Samuel Clemens were the same person for all purposes,
• you don’t know what to put down for Homer’s year of birth in that author/title cataloguing app you downloaded,
• you wonder about people who think that something doesn’t exist if it’s not in the expected database (or if it’s not on the Web).

Information philosophy

If these issues sound a lot like the first ten minutes of a college philosophy course, that’s intended. Philosophy is all about seeking answers to questions we don’t often pause to ask.

Pause.

Pause.

We run into questions like these all the time in building software, especially now that we’ve woven the Web and woven ourselves and our lives into it.

On Information in Rotation I’m going to call this category “Information Philosophy”; I think it will get woven in with the more orthodox techy blog-fodder as we go along.

Anyhow, I strongly recommend Data and Reality, which is available as print-on-demand or (inexpensive) eBook from the publisher at the link I gave above (as of 2006-12-28).

[The paperback has ISBN 9781585009701 and the eBook 9781420898880. Does this make them different books?]