Everybody loves to bash XML namespaces, but this essay is the most careful and dispassionate I have seen to date. You should read the whole thing if you deal with XML (or even if you just like to read the product of clear thinking).

Here’s a key quote:

I would claim that the aspect of XML Namespaces that causes pain is the URI/prefix duality: the thing that occurs in the document (the prefix + local name) is not the same as the thing that is semantically significant (the namespace URI + local name). As soon as you accept this duality, I believe you are doomed to a significant extra layer of complexity.

Nothing new here, but its being in the Times means my mom reads about it, and yours might too.

Recalling that my Android G1 phone has GPS, I installed MyTracks from the online Android apps store.

Yesterday I took both devices for a spin together, and they recorded the same paths (within a few feet), with more differences while walking or indoors than while driving. Today I tried a comparison bike ride, which the Android app flunked by only seeing satellites intermittently from the pocket where I was carrying it. I’ll have to try again with the phone away from my body.

Assuming that works, the real difference between the two devices is cultural.

The Garmin is a classic device from a hardware company. It’s marketed to runners, bicyclists, boaters, hikers, and other outdoorists. The battery compartment, display, and data ports are protected by rubber fittings: you can use it in the rain. On the other hand, the software sucks, and it’s hooked to a business model that makes maps from Garmin the only ones you can install.

The G1 is an open smartphone with a GPS receiver in it. The MyTracks software is pretty good, and the mapping comes from Google Maps on the device. This is the classic computer platform style: the platform vendor stands back and lets third-parties compete to make the best app in each niche. On the other hand, the hardware is Just a Phone: as suggested above, I have my doubts about the antenna, and there’s certainly no special provision for the elements.

Historical evidence from analogous market situations strongly suggests that the open platform model will win. Hardware-company culture will suck up energy by trying to segment the market with too many SKUs (have a look through Garmin’s web site to see what I mean), and software improvement will always be an afterthought as they try to move boxes through channels as diverse as Best Buy, Crutchfield, REI, and auto parts stores.

Android GPS software vendors, on the other hand, only have one channel to worry about: the app store. They only have to worry about being able to clearly state which devices their apps run on. This frees them to concentrate on acquiring and making interesting use of GPS data. They have no physical boxes to move (the hardware vendors take that risk).

On the hardware side, the niche for rain-resistance can be satisfied by accessory makers making rubber sleeves for entire devices or by hardware vendors wrapping rugged housings around electronics designed and built by another company that knows how. In neither case does the hardware outfit need to know that the customer wants to do GPS: it’s sufficient just to know that they anticipate getting rained on for whatever reason.

The same arguments apply to iPhone, of course, but I don’t have one of those to write about.

Brad Fitzpatrick says:

What does product liberation look like? Said simply, a liberated product is one which has built-in features that make it easy (and free) to remove your data from the product in the event that you’d like to take it elsewhere.

Note the emphasis on free and easy. Online services tend to stop at possible, which falls short of easy by the proverbial Simple Matter of Programming.

Facebook, for example, has a programming interface for each kind of information you store that lets a Facebook application extract it into a file. It’s possible for me to write an application that does so for everything I have on Facebook, but that isn’t the same as having a predefined procedure that archives my entire Facebook presence into a standard file format that most social networking sites can import from.

Good luck to Google in this effort! I myself would go further in asking that the entire behavior (as well as data) of my social-network presence be standardizable and portable, as Ramón Cáceres [disclosure: personal friend] and his collaborators are proposing with their work on virtual individual servers.

The human genome has just over 3 billion base pairs in about 25 thousand genes. This is a large enough data set that it gets algorithms of its own.

It’s a tough problem: if you do things (like scanning entire libraries) at Google-scale, you just can’t pay attention to the details. One partial way out (which Geoff mentions) is to allow users to submit corrections, as Google Maps does for positions of placemarks.

The article addresses a number of important points about the provenance and usefulness of metadata, and Google employees provide some great comments and discussion.

(Via Brad DeLong).

Udell points out, quite rightly, that

“Give us the data” is an easy slogan to chant. And there’s no doubt that much good will come from poking through what we are given. But we also need to have ideas about what we want the data for, and communicate those ideas to the providers who are gathering it on our behalf.

In what is planned to become an annual competition known as NYC Big Apps, the city will make available about 80 data sets from 32 city agencies and commissions. The winners of the competition will get a cash prize, recognition at a dinner with the mayor, and marketing opportunities.

One has to be wary of competitions: they can be a way of trying to get some work for free, or a sign that the project doesn’t have realistic funding behind it. On the other hand, it shows that the sponsor wants to tap a wider range of imagination than it would get with the usual contracting process.

Dinner with the mayor!

Hat tip: "tootie" at reddit.

Jon Elerath’s article on hard disk reliability in the June 2009 Communications of the ACM (may require ACM login) gives a lot of detail about the different kinds of disk storage error and appropriate countermeasures. He gets down to the level of things that scratch vs. things that smear (kind of like H.P. Lovecraft for the archivally-minded).

The big takeaway is that there are big obvious crash-failures like the bearing getting wobbly or servo tracks being trashed: these make the drive stop working, and you rebuild from something you still trust. And then there are insidious quiet read/write failures that you can only counteract with a policy of "scrubbing" drives proactively.

Now that I buy storage by the 1.5 terabytes/spindle, I really should do something to dissuade Those Whose Names Are Random from assimilating my data to the Outer Abyss of Maximum Entropy. If you should happen to find some of your goats missing, better not to ask…

He names the two approaches “structure-and-browse” and “collect-and-query”. I feel like I’ve been groping for these terse descriptions for years!

Chris Anderson of The Long Tail has an important post about how the metadata used in some music-listening applications doesn’t satisfy the listeners needs:

[...] classical is a genre that the one-size-fits-all music aggregators such as iTunes don’t handle particularly well. They’re oriented around pop music, with its artist, album, track data format. Meanwhile classical music organizes around composer, conductor, performer, soloist

He also voices my exact peeve about how jazz is treated:

However, neither of them does a very good job with Jazz, where the individual musicians are often more meaningful than the band.

Yup. No reasonable cataloguer of jazz recordings separates “Thelonious Monk Trio” from “Thelonious Monk Quartet” from “Thelonious Monk”. At the same time, it’s important to be able to locate all appearances of Thelonious Monk, regardless of whether he was the leader of the session (note that “leader” and “session” are appropriate terms in jazz discography, but not for pop or classical).

When your only tool is a hammer…

I can’t help but wonder if the problems Chris calls out in iTunes come from the poor selection of data tools in most applications programmers’ toolkits. Relational databases, the current orthodox storage technique, favor using one or more tables, each consisting of records having the same selection of attributes. There are hacks you can use to simulate having, say, jazz tracks and pop tracks in the same Tracks relation, but hacks and simulations tend to twist one’s code, so most programmers resist going there.

An XML database in every toolbox!

We don’t really have to live this way anymore. With the popularity of XML for data interchange, the tools ecology has given us a variety of XML database systems. The XML data model has the flexibility to represent varying record structures: in fact, it has much more flexibility than we need for the purpose!

Heretical as it may seem to put the cart of an interchange format before the horse of data abstraction, the XML situation is very useful in practice, at least for databases of moderate size. The W3C has come up with the XPath and XML Query specifications that provide excellent query mechanisms for data represented in the XML model. XML Query in particular is designed to look somewhat familiar to the hardened SQL user. There’s data typing taken from the XML Schema datatype recommendataion as well.

Better nails

Anyhow, let’s learn to design with a more flexible hammer, and maybe we’ll be able to hit a wider class of nails, rather than our users’ thumbs!

March is International Runaway Metaphor Month.

OpenStreetMap is a wiki-like project to build a world map using contributed GPS tracks [OpenGeoData pointed me there]. Their map of Baghdad is here.

This project is truly a product of the early 21st century: it requires GPS satellites, cheap but accurate GPS receivers, the World Wide Web, inexpensive computers with fast color graphics, and so forth.

And like all modern geographic applications, it also exploits a special property of GPS’s information domain: everyone agrees on the meaning of geographical location; only dates and times have a similar level of standardization. In relational-database terminology, this means that any table with a date or location column has a meaningful join with any other.

This doesn’t work with most data. I’ve had driver’s licenses in four U.S. states, but you can’t aggregate my driving record from the state records because they all use different ID numbering schemes (nice for my privacy in this case).

Also noteworthy is the fact that GPS information can be used to put a time dimension into maps, since we can tell when the street is used as well as where it is. There are some very pretty examples at Cabspotting.

There’s a great discussion about the relative advantages of the two approaches on the blog. Basically, storing the tags in the file makes the association harder to lose as you move the file around, and storing the tags separately avoids modifying the user’s data file. Neither one is obviously in accord with the user’s intention in all cases.

I think the issue has whole extra layers of subtlety. We perceive metadata that is stored within a data file as being what Jon Udell calls “file truth”. Since there’s only one set of metadata stored in the file, it becomes the One True Metadata. On the other hand, metadata stored in a separate database reads as the opinion of the maintainer of the database. This is exactly what social bookmarking systems such as del.icio.usdo: each attribution of a tag to a URL is also associated with a user making that attribution.

A pluralistic society requires a separate metadatabase!

This isn’t just another engineering tradeoff, though. The truth about “file truth” is that it’s still an opinion—the opinion of the last agent to modify the metadata within the file. When there’s One True Metadata, we can only represent disagreements by obliterating the last guy’s assertion.

Imagine trying to tag a scan of a photo taken at your parents’ wedding of someone you don’t recognize. You think it’s Dad’s college roommate, but your sister thinks it’s Mom’s second cousin. You have one “person depicted” slot: do you fight over it? Do you leave it blank and explain the situation in a semantically bland catch-all description field? Or do you each tag it as you will in your respective databases?

Not only is it unrealistic to allow for only one true description of a file, it’s also time we stopped regarding metadata as lost forever just because it’s not stored in the file. We could set up a distributed database that works like Gracenote’s CD identification database, but for all files instead of just music files. As with CDs, the lookup key for a file can be generated by anyone who possesses the file (by applying a secure hash), but the particular metadata obtained depends on which tagger’s part of the repository is consulted. It’s all doable, and it would eliminate blogstorms about how evil application X erases user metadata.