Paul’s Journal

If you care about distributed systems, you need to read the paper about Amazon’s Dynamo.

Comments:

• Making node joining/leaving an administrative command is not something most academics consider, but it significantly reduces complexity.  We made a similar decision with the PodServer system for Bloglines.  I believe this is the right decision, since a node changing membership on the long term is a rare event.  Even with our growing blog index, we only add new nodes once every 6 months or so. (Plan ahead )
• Shout out to BerkeleyDB.  Glad to see other people pushing it hard. Combined with the older white-paper about Google using BerkeleyDB for their Google Accounts system, it just validates my positive feelings on continuing to use it as a core part of the Bloglines architecture.
• The configurability of N/R/W is a great idea.  Most systems make N configurable, but skimp out on giving full flexibility to the people using the system.
• I’m convinced I need to read more about Vector Clocks.  For the Bloglines PodServer, we are blessed with only have a single writer per record due to how our crawlers work, so we just ‘cheat’ on versioning, but this has caused us pain a few times.
• I wish Amazon would Open Source Dynamo. I can understand the difficulties in doing that, but its a nice thing to dream about.
• I think I will propose an Apache Labs project to start something like Dynamo.  For a basic key/value storage system on a constant hashing ring, without all of the High Availability concerns, you could get something working pretty quickly.   Adding all of the high end features could take time of course…..

More discussion over Sam Ruby’s Blog: Key + Data.

This all ties in nicely with the GeekSessions 1.2 topic  of “Designing beyond the database”, where I presented last night.

The Amazon’s MP3 store is almost perfect.

The downloader works great on OSX.

The music is DRM free.

It’s fast. (Faster than ITunes Music Store!)

Only downside so far is the lack of good recommendations. But I trust Amazon to fix that with a little time and data.

Bought 5 Albums so far.

Thank you Amazon for making an awesome Music Store. It’s about time someone did it right.

Wouldn’t it be cool if Inline CSS worked inside a feed reader?

I think that would be cool.

PS: Thanks to everyone who helped with the Sanitation Rules Wiki.

EDIT: Like Bloglines does…..

I’ve migrated my Journal to Wordpress. Typo just wasn’t doing it for me anymore.

If you are seeing this, the DNS has propagated.

Trying to stream Flash Video?

Tired of using PHP?

I wrote mod_flvx to do all of the streaming stuff via an apache module.

To configure, add AddHandler flv-stream .flv to your httpd config.

thats all.

EDIT: Wiki page explaining all of the encoding and flash player needs for using mod_flvx.

From Alpha-Geek.com:

I was even able to find a blog of one of their workers – Paul’s Journal – though he doesn’t seem to talk about his $job too much.

It is hard to post about $job, because very little of Bloglines backend architecture is considered public information. For example, from the companies Blogging Policy Document:

“It is the Company’s policy that all employees, agents and contractors must keep confidential all information about IAC Search & Media or any IAC company’s operations, products and business activities that has not been made public …”

Which I believe pretty much includes how our custom distributed filesystem works, or how Bloglines is scaling (other than saying that memcached is awesomr, which is public), or even the details behind our new Atom Parser. It isn’t something I am too concerned about really, you don’t see anyone from Google posting the technical details behind Google Reader. It is just a standard in so many coporations to keep information close.

As Requested in the $job Blogging Policy Document: “The views expressed in this blog are my own and do no necessarily reflect the views of my employer.”