Information in Rotation » Storing data

Disk drive reliability in detail

Dan Rabin — Mon, 29 Jun 2009 17:56:15 +0000

I tend to get abstract and philosophical about data here, and it’s good to have have an occasional splash in the cold water of how the stuff gets stored.

Jon Elerath’s article on hard disk reliability in the June 2009 Communications of the ACM (may require ACM login) gives a lot of detail about the different kinds of disk storage error and appropriate countermeasures. He gets down to the level of things that scratch vs. things that smear (kind of like H.P. Lovecraft for the archivally-minded).

The big takeaway is that there are big obvious crash-failures like the bearing getting wobbly or servo tracks being trashed: these make the drive stop working, and you rebuild from something you still trust. And then there are insidious quiet read/write failures that you can only counteract with a policy of "scrubbing" drives proactively.

Now that I buy storage by the 1.5 terabytes/spindle, I really should do something to dissuade Those Whose Names Are Random from assimilating my data to the Outer Abyss of Maximum Entropy. If you should happen to find some of your goats missing, better not to ask…

Quick addendum about the Chandler Repository

Dan Rabin — Wed, 31 Jan 2007 07:40:53 +0000

If my brief article about the Chandler Repository caught your interest, you might want to check out Andi’s blog, in which he discusses some of the design and implementation issues.
Accounts of software design from the time of implementation are incredibly valuable. Maybe someday we’ll have a search tool that lets you enter your implementation ideas and gives you back the accounts of people who have tried them before.

The Chandler Repository

Dan Rabin — Tue, 30 Jan 2007 23:35:14 +0000

I spent a few hours yesterday in the company of Andi Vajda, lead developer of the data repository component of the Open Source Applications Foundation’s Chandler project. We talked about the technical details of the repository.

The Chandler repository is an object database with some interesting design features:

All links between objects are bidirectional. Andi’s main motivation for this choice was to be able to guarantee to clients that references aren’t dangling without having to implement a garbage collector.
All objects have universally unique identifiers (UUIDs) to make equality testing trivial.
Repositories are versioned. Aside from the usual rollback capabilities this enables, client code can inspect a particular object as it was in any extant version.
In addition to a conventional notion of class inheritance, the Chandler repository supports a notion of “cloud inheritence” for merging data schemas when copying objects from one database to another.

The repository layer is cleanly separated from the layer of Chandler that knows the semantics of calendar items, tasks, and so forth. That layer is responsible for mapping its items to repository items. Interchange of personal information is carried out at this higher layer, not by the repository. There’s an interesting bit of Python metaprogramming by Philip Eby that hides some of the mapping complexity from the application-level programmer.

I’m going to have a look at the source to see how this works. There are popular application-building tools that carry out object/relational mappings (moving between the objects that are convenient to use in code and the representations that are convenient in the usual relational database), but this is the first time I’ve come face-to-face with an object/object mapping tool. I’ll report back when I’m better informed.

[Thanks to Andi for fact-checking this post!]