On Friday, the first actual day of SXSW:

• Grabbed lunch at Annies, and met with Jerry Chen who has been contributing to the libcloud project.
• Opened CASSANDRA-885 bug report.  OOMing after the node crashed, as its unable to replay its commit log.
• Went to a showing of “The Girl with the Dragon Tattoo“.  The movie was very intense, but I still liked it.  Showing was at the Alamo, and having a Margarita while watching was even better.
• Had dinner with the entire crew at Stubb’s.  Loved the brisket.
• After some bar hopping, we ended up back at the pad… playing Rock Band.

HipHop is definitely some very cool technology, built by an enthusiastic team, solving real world performance issues in large scale websites, and I have no doubt other companies using PHP (Hello Yahoo!) will find it invaluable, and hopefully help turn it into a successful open source project.

What I find most interesting and encouraging about Facebook’s most recent open sourcing efforts, HipHop today and Tornado last year, is how they are taking a dramatically different approach to their earlier open source projects like Thrift or Cassandra.

Thrift, purely as an example, was one of their first projects built internally, and later open sourced.  It originally open sourced on April 1, 2007, but it had a difficult time building a community around the code.  The approach was a blog post, and code basically ‘tossed over the wall’.  External developers did try to contribute, but I believe the interactions were less than optimal, as the original forum for discussion was a Facebook Group — they learned from this quickly,  programmers didn’t like web forums for submitting patches, and later proper mailing lists were setup.

Cassandra was another project that was essentially thrown over the wall, code was available, but there was no imitative to build a community around it.

Today, both Thrift and Cassandra found their way to the Apache Software Foundation, via independent paths.  Apache Cassandra is turning into a very healthy community, having made many releases, and is in the process of graduating to a top level project.  Apache Thrift, made their first release in December 2009, has been slowly gathering more external contributors and an open community built around the code.

What I see happening with both HipHop and Tornado is completely different, and that is what is most encouraging.  From the start, they are doing everything right to encourage an open community be built around these projects.  Open Communities are what create successful projects, and give companies creating the open source projects the most rewards.

When you create an open source project, you gain almost nothing but a PR hit if there isn’t a community built around it.  For infrastructure projects, like HipHop, Cassandra, Thrift, Scribe, and Tornado, the most important thing that gives you the most rewards from open sourcing it, is having other people hack on the code — but more than that, to use the code in their own company.

Just look at the massive community that has exploded around Apache Lucene and Apache Hadoop — Yahoo could of kept this infrastructure project internal, and sure, it might of fulfilled their original goals, but they wouldn’t of ever received the thousands of external contributions, which has turned the Lucene/Hadoop world into one of the most diverse and thriving open source communities of late, giving Yahoo a thousand times return on their investment in Hadoop.

Thank you Facebook for getting it — community is just as important as the code that you are open sourcing, and I would like to wish the HipHop for PHP developers the best of luck with their new open community project!

I’ve been working along with the entire Cloudkick Team on a few parts of our launch:

• Integration with Apache libcloud, so now Cloudkick supports EC2, Rackspace, Slicehost, RimuHosting, Linode, VPS.net and GoGrid.
• Our new monitoring Agent, Cloudkick Agent. Extremely light weight, written in C & Lua.  Hopefully I’l have some time to blog more about some of the cool technology we did here, but we were tired of seeing monitoring agents written in High Level languages using up tons of memory on a Cloud Server.
• Our Cloudkick Changelog Tool, aka “ckl”.  This tool lets you keep track of a large admin team and what everyone is doing.  The ASF Infrastructure team has already started using it outside of Cloudkick.  Of course, the Cloudkick UI is a little nicer than the demo one with the open source code.
• Our new Graphing and long term trending system, built on top of Reconnoiter and Apache Cassandra.
• Learned more about Django and JQuery than ever before.

Now that our big launch is out, hopefully I’ll find a little more time to post on this journal more.

Lets say you want to cache just your root page, aka ‘/’, for your website, just in case you get hit by a Slashdot Effect.

For Apache httpd 2.2.12 or newer, you can do this by first enabling Caching on All pages, then setting the no-cache enviroment variable globally, and then unsetting it for a specific path:

CacheDirLevels 2
CacheDirLength 1
CacheEnable disk /
CacheRoot /var/cache/apache2/mod_disk_cache
CacheIgnoreHeaders Set-Cookie
CacheIgnoreNoLastMod On
CacheMaxExpire 600
SetEnv no-cache
<LocationMatch “^/$”>
UnsetEnv no-cache
</LocationMatch>

For Apache httpd before 2.2.12, you need a different method of disabling caching globally, and then re-enabling it.  The easiest way is using mod_headers, to muck with Vary header

Header set Vary *
<LocationMatch “^/$”>
Header unset Vary
</LocationMatch>

Strictly speaking, doing this to the Vary header is an RFC violation, and you best bet is to upgrade to a newer httpd version. 

This works because mod_cache will refuse to cache any HTTP resource with a Vary value of “*”, because this is saying that every response form the origin will be different.

Normal anti-hot linking recipes, like the one on the HTTPD Wiki are all about disabling access to the image completely.

If you have lots of people hot linking, these users can use up valuable Keep Alive sessions, so the easiest way to solve this problem is to disable Keep Alive for just those clients viewing a hot linked image.

This is possible by using mod_setenvif and the nokeepalive environment variable:

SetEnvIfNoCase Referer (.+) nokeepalive

SetEnvIfNoCase Referer (.*)example.com(.*) !nokeepalive

What this does is first disable KeepAlive for all users that have a Referer set, and then re-enable keepalive for those users who are coming from ‘exmaple.com’, which should be replaced with your site.

now its just down to execution.

The coolest thing coming out of it is some motivation to finish SvnPubSub.  You can try it out right now, it is running on our svn-master:

curl -i http://svn-master.apache.org:2069/dirs-changed/xml

(data formats, URLs, etc will change, don’t use it for anything special yet)

The whole apache.org incident though has killed my time for hacking on serf, but I’ll hopefully find some time to finish the server support soon.

Other misc stuff:

• Moving up to San Francisco in the next month, will do another post with more details later.
• Cloudkick is going well, doing fun stuff with Python and Twisted, hopefully have more to blog about there soon.

The easiest way to do this for all your URLs is something like this using mod_asis:

       # Bind mod_asis to files ending in .asis
       AddHandler send-as-is asis
       # Add other Aliaes/AliasMatches for any other resources needed (logos, css, etc)
       Alias /logo.png /opt/mysite/maint/logo.png
       # The magic line, pulling all matching URLs into one file
       AliasMatch /(.*) /opt/mysite/maint/index.html.asis

Your index.html.asis would contain something like this:

Status: 503
Cache-Control: no-cache
Content-type: text/html

<html><h1>My site is down, be back soon!</h1></html>

I can’t help you make a fail whale / plumber image though, that is up to you

/**
 * List of all C-T-E Types found on httpd-dev and FreeBSD-current:
 *
 * Content-Transfer-Encoding:      8bit
 * Content-Transfer-Encoding:  7bit
 * Content-Transfer-Encoding: 7BIT
 * Content-Transfer-Encoding: 7Bit
 * Content-Transfer-Encoding: 7bit
 * Content-Transfer-Encoding: 8-bit
 * Content-Transfer-Encoding: 8BIT
 * Content-Transfer-Encoding: 8Bit
 * Content-Transfer-Encoding: 8bit
 * Content-Transfer-Encoding: BASE64
 * Content-Transfer-Encoding: BINARY
 * Content-Transfer-Encoding: Base64
 * Content-Transfer-Encoding: QUOTED-PRINTABLE
 * Content-Transfer-Encoding: Quoted-Printable
 * Content-Transfer-Encoding: base64
 * Content-Transfer-Encoding: binary
 * Content-Transfer-Encoding: none
 * Content-Transfer-Encoding: quoted-printable
 * Content-Transfer-Encoding: x-uuencode
 * Content-Transfer-Encoding:7bit
 * Content-Transfer-Encoding:quoted-printable
 *
 * This is why we have RFCs.
 */

minotaur# uname -a
FreeBSD minotaur.apache.org 7.2-STABLE FreeBSD 7.2-STABLE #0: Wed Aug  5 01:05:27 UTC 2009     root@loki.apache.org:/usr/obj/usr/src/sys/MINOTAUR  amd64
minotaur# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h1m, 0.01% done, 266h36m to go
config:

        NAME           STATE     READ WRITE CKSUM
        tank           DEGRADED     0     0     0
          raidz2       DEGRADED     0     0     0
            replacing  DEGRADED     0     0     0
              da14     UNAVAIL      3   570     0  experienced I/O failures
              da0      ONLINE       0     0     0  6.24M resilvered
            da1        ONLINE       0     0     0  4.06M resilvered
            da2        ONLINE       0     0     0  4.15M resilvered
            da3        ONLINE       0     0     0  4.09M resilvered
            da4        ONLINE       0     0     0  4.14M resilvered
            da5        ONLINE       0     0     0  4.10M resilvered
            da6        ONLINE       0     0     0  4.15M resilvered
            da7        ONLINE       0     0     0  4.11M resilvered
            da8        ONLINE       0     0     0  4.17M resilvered
            da9        ONLINE       0     0     0  4.08M resilvered
            da10       ONLINE       0     0     0  4.13M resilvered
            da11       ONLINE       0     0     0  4.14M resilvered
            da12       ONLINE       0     0     0  4.14M resilvered
            da13       ONLINE       0     0     0  4.08M resilvered
        spares
          da14         AVAIL   

errors: No known data errors

da14 failed. we had da0, not in the array yet, so we just did:

zpool replace tank da14 da0

But now it is stuck.

It never makes progress on the Resliver.

It sure sounds like this bug:
http://bugs.opensolaris.org/view_bug.do?bug_id=6655927

But this is FreeBSD 7-STABLE from earlier this month, and it really shouldn’t be affected by that bug.

Sigh.

Yesterday I got a call from Comcast.  I didn’t think much of it.  At least how I remember it, the lady said the Auto-bill pay had made a mistake and over charged me for July.  $240 instead of $155 or something.  And they noticed this mistake, and had corrected it. Great so.. I should just be getting a credit. Woohoo money back I never noticed leaving.

Today when I woke up, the internet wasn’t working.

It was showing a stupid comcast activation screen.  To activate…. it wanted me to download some comcastic activation software.  Screw it, I was feeling particularly dumb this morning and I did on my older Macbook Pro.  It completely FUBARed up my network settings.  Glad I didn’t do it on my primary work machine.

Okay, well, one dead laptop later, I decided to call Comcast.

I will give Comcast some credit here, I was speaking to a Real Human Being in less than 2 minutes.  I remember being on hold for USWest/Qwest for hours when dealing with any DSL issues.

I just try to play dumb, ‘my internet has stopped working’, and the lady says it was turned off for non-payment.

Non-payment? I’ve had auto-bill pay on for 4 years, since I moved in.  I have auto-bill-pay with Comcast because I never ever ever want to deal with their crap or be on the phone with them. I WANT FAST INTERNET AND LEAVE ME ALONE.

It turns out that $242 was from someone ELSE who had paid onto MY account in June.

What happened yesterday is that Comcast reversed that payment, and since the auto-bill pay thought I had already paid for June and most of July, and now I had a delinquent charge of $242, overdue for 2 months.

So they turned off my service.

Well, turning off my service for when they fucked up sure is one way to get my attention.

So, I paid $403 to keep the internet flowing.  It isn’t like I had a choice in high speed internet.

In my rage I did change my TV plan from Digitial Rare-Metal to the basic standard plan, but I wish I could of just canceled my entire account.

If I had a choice, I would choose any other service that can offer me 20 megabit/second downloads.

AT&T in theory can offer DSL service in my area, but I don’t have a landline, and they don’t offer anywhere near 20mbit/second downloads.

Sigh.
Edit: While writing this post, Safari crashed. Twice.  :rage:

similar malloc debugging for linux:

MALLOC_TRACE=/tmp/out.log

related: memcheck.
and more info on the suse wiki.

I am happy to announce that I am joining Cloudkick next week.

It should be a different experience to compared to Joost; Very small company vs Medium, everyone is local vs everyone is distributed, and languages I enjoy (Python, C) vs Ones I don’t (Java, Javascript).

“Fear not for the future, weep not for the past. ” – Percy Bysshe Shelley

Friday July 31 will be my last day at Joost.

It was a fun ride for the last 19 months.  Starting out working on Joost’s P2P library, later on our Content Delivery over HTTP, and more recently writing Server Side Javascript inside Rhino, along with countless trips to Leiden and New York City.  Thanks to all of my colleagues who I worked with, Joost was a great experience.

As announced a few weeks ago, Joost is changing its focus to white label services.  I took a long look at this, but in the end I decided the new Joost just wasn’t for me right now in my life.

Good luck to everyone who is staying on at Joost.

My new job is with Cloudkick.

Apache Orthrus is a C library and user interfaces for RFC 2289,
“A One-Time Password System (OTP)”, also known as OPIE or S/Key.

SVN: https://svn.apache.org/repos/asf/labs/orthrus/trunk/

If you have ever tried to compile OPIE on OSX, you might understand why I started this, its just painful and full of silly things.

Most of the Apache Software Foundation’s FreeBSD machines use OPIE for our sudo accounts, and I’ve been using SkeyCalc, which was last released as a PPC Binary for 10.1 in 2002.  Right now Apache Orthrus can do the client side of OTP, but I want to finish the project to include a PAM module and full verification support.

Patches welcome though, if anyone else uses OTP out there……

In a cramped AA md-80 on the tarmac for a 2 hours so far, next update in an hour. Sigh.
.

For an example of why it is cool, lets look at replacing a common task with mod_rewrite: Blocking Image Theft.

The HTTPD wiki even has an example of how to do this for us:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !=""
RewriteCond %{HTTP_REFERER} !example\.com [NC]
RewriteRule \.(jpe?g|gif|png)$ - [F,NC]

With the new mod_lua, you can do this using real if statements and functions.

For example:

<LuaHookTranslateName imagetheft>
function is_image(path)
    -- You could put complicated regular expressions here.
    if path:match("%a+.png") then
        return true
    end
    return false
end

function imagetheft(r)
    if not is_image(r.uri) then
        return apache2.DECLINED
    end

    referer = r:headers_in("Referer")
    if referer then
        if referer:find('example.com') then
            return apache2.DECLINED
        else
            r:err("Forbidden for Image Theft! uri=".. r.uri)
            return 403
        end
    end
    return apache2.DECLINED
end
</LuaHookTranslateName>

While this example comes out signifigantly longer, I do believe in the long run it will let people write more maintainable configurations, espcially for things like complicated RewriteRules — since basic things that have long be obsecured into RewriteCond can now be easily accessed and evaluated with an If statement in Lua.

Much of the API is still in flux, but hopefully we will come up with some shortcut builtins that make many common taskes easier and shorter.

However, I don’t really like being tied into the whole Java world because of it.

When Google released their v8 Javascript Engine earlier this year, I always wanted to build an Apache Module for it.

This afternoon I had some time, and so I created mod_v8.

It doesn’t do much beyond a Hello World right now, but it is as simple as this:

ap.write("Hello World!");

I’m not sure if I will spend time making it a proper project, I really want to spend more time on making httpd 2.4 before getting too distracted with shiny things….

• Improve the Simple MPM.
• Integrate Lua and mod_wombat into httpd trunk.
• Improve the FastCGI Support in httpd turnk.
• Finish my little infrastructure stats & graphs project.

Well, I did get something working on my Infrastructure Stats Project.  You can see the work in progress here.  It uses flot and jquery to draw graphs of the mail and code commits at the ASF since the Apache Project started in 1996.

I didn’t get around to most of my httpd goals, since every time I work on the Simple MPM, the fact that KQueue causes a kernel panic on OSX kinda discourages me.  I was hoping that the latest kernels from Apple would fix the problem, but they don’t.

I did manage to commit 4 new modules to httpd trunk however:

• mod_heartbeat – Generates Multicast heartbeat messages containing the status of the server.
• mod_heartmonitor – Collects these Multicast heartbeats for other modules to use.
• mod_ratelimit – Bandwidth Rate Limiting for Clients.
• mod_lbmethod_heartbeat – Module to Load Balance mod_proxy_balancer workers using the data from the heartbeats.

mod_heartbeat, mod_heartmonitor, and mod_ratelimit were all originally written at Joost, and my employer was nice enough to let us contribute them back to Apache.

I’ve also announced the intent to roll the first 2.3.0-alpha release next weekend.  Hopefully this means within a few months, a new stable 2.4.x branch will come out.

Yesterday, Apple approved the Joost Application for the IPhone.

Apple Store Link.

The Joost Application is something I helped prototype, and wrote most of the server side support, although I was busy with other things and haven’t worked on it recently, and I’m glad it is now released to the public.

It isn’t perfect, as some of the reviews have said, we are doing our best with the APIs that Apple provides, but the Video Player has some pretty big bugs we do our best to work around, but I hope Apple significantly improves MPMoviePlayerController with another firmware release.

Radar Bugs: (what are rdar links?)

• 6352281 – Problems with the Headphone getting inserted causing video to pause
• 6299534 – Problems with the Next Button on the default Apple UI.
• 5888718 – Video player does not detect when it is upside down.
• 6407138 – Audio Sync Issues

• Improve the Simple MPM.
• Integrate Lua and mod_wombat into httpd trunk.
• Improve the FastCGI Support in httpd turnk.
• Finish my little infrastructure stats & graphs project.

I’ll be happy if I make progress on any one of them…..

Yay!

ps, New Orleans and ApacheCon is fun.

pss. Last day is Today.

It told me that my WP version was too old, so using pterm I sshed into my server, and upgraded from 2.3 to 2.6.

And now I’m writing a blog post on my iPhone.

The iphone is not perfect, but nothing else is even close.

More info on the mailing lists.

I should write more about the Simple MPM, but ApacheCon New Orleans is  coming up next week, so I want to try to get more of the code in before then, since traditionally coding during the conference never works out, and I’ll try to write up a blog post about why its cool sometime next week during the conference.

One of the new features I don’t think anyone has mentioned much is that we now have built in support for chroot.  Just add ChrootDir “/srv/my_root”, and all IO in Apache after the initial startup will be inside the chroot.

However, the SEVEN HUNDRED BILLION DOLLAR buyout deal that Henry Paulson, the Secretary of the United States Treasury, is trying to get done, is insane.

This isn’t Capitalism.

If I didn’t believe in Capitalism, I wouldn’t be working at a startup.

If Joost fails, I don’t expect anyone to help me.  I expect it to hurt. But it is my risk to take, why do giant financial companies, who have done apparently stupid things, get treated differently?

If Joost is successful, I expect rewards, and this is the basis of any capitalistic system,  but these buyouts are creating a system in which stupid companies can do badly — and their employees still get rewarded.

I didn’t live during the Great Depression.  I don’t know how bad it can get. I have lived an ‘easy’ life in modern America.

But I would rather see another depression than this massive buyout.

Whatever is going on, it is not capitalism, and I am sad to see America going in this direction.

Forgive me

Anyways, figured some people might be interested in what I commited last night for the Apache HTTP Server….

First is the ability to ’suspend’ an HTTP Request from a Module.

Normally in Apache, there is a one to one relationship between the running thread, and a client connection that it is servicing.  In the Event MPM previously, we had broken it out so between different HTTP requests in the same TCP connection, aka Keep Alives, could be serviced by different threads.

Now with r697357 in httpd trunk, you can return SUSPENDED from a content handler.  The core  will assume you have taken over handling of the request, and get out of your way.

This is insanely important, because in the long run it lets you write very cool modules, like a full async HTTP Proxy, aka something like Varnish, where you could easily handle tens of thousands of client connections, using only a hundred odd threads, where today you would need a huge amount of RAM, and a single thread or process for every client to do this with Apache.

Secondly, as a demonstration that it works, I wrote mod_dialup. It is a module that sends static content at a bandwidth rate limit, defined by the various old modem standards. So, you can browse your site with a 56k V.92 modem, by adding something like this:

<Location /mysite>

ModemStandard V.92

</Location>

Previously to do bandwidth rate limiting modules would have to block an entire thread, for each client, and insert sleeps to slow the bandwidth down.  Using the new suspend feature, a handler can get callback N milliseconds in the future, and it will be invoked by the Event MPM on a different thread, once the timer hits.  From there the handler can continue to send data to the client.

The last week has seen many new features and improvments made to httpd. Many of them have been accelerated by people at the ApacheCon EU Hackathon this week.

• mod_session

  On Friday Graham Leggett introduced a series of modules to support generation of sessions from HTTPD. Included is mod_session_crytpo, which encrypts the data using AES. This is the first time ‘form based’ authentication has had real support in the Apache Core.
  [docs: mod_session]
  [thread: Apache support for form authentication]
  

• mod_socache

  On Tuesday Joe Orton commited the new Small Object Cache modules, which have been under discussions for a couple months now. The mod_sslsession cache has been changed to use this. Currently suported cache backends are DBM, memcached, and Shared Memory. I expect many other modules will changed to use this cache API as time goes on.
  [svn: ap_socache.h]
  [thread: [PATCH] ap_socache.h & mod_socache_*]
  

• If/Else blocks added

  Nick Kew ported the expression parser from mod_includes, and has used this to add If and Else blocks to the core.This provides a viable alternative to mod_rewrite and RewriteCond, and letsyou set any modules configuration values.
  [docs: if]
  [thread: Dynamic configuration for the hackathon?]
  [commit: r644253]
  

• Turkish Documentation

  Nilgün Belma Bugüner contributed a complete translation of the Apache HTTP Server documentation in Turkish.
  [docs: Turkish]
  [thread: New Turkish Documents]
  [commit: r645667]
  

• Serf Bucket Discussions

  Discussion at the Hackathon covered how Serf Buckets use a “pull” method, for both input and output, unlike the current filter stack in httpd, which is Pull for input filters, but push for output filters. There was general agreement that the expieriment of mod_serf should be expanded up the filter stack.
  [svn: mod_serf.c]
  

• Simple MPM created

  Paul Querna started work on a new MPM at the Hackathon. The MPM hopes to run on both Unix and Win32 platforms, and keep the same behavoirs on both.
  [svn: SIMPLE.README]
  

Not giving any talks this year.

I will also be at Joost’s Leiden office the week following.  (April 12-19) 

Returning to San Jose, was not so fun.

Yesterday, American Airlines cancelled a couple hundred flights due to problems with their MD-80s.

I was originally going LGA -> ORD -> SJC.

The ORD -> SJC leg got cancelled.

They re-routed me LGA -> DFW -> SJC.

When I landed in DFW, the flight to SJC had been cancelled.

There were no more flights to SJC.

The flights to SFO and OAK were all fully booked.

I luckly got on the top of the standby list to OAK, and got on that flight.

My checked bag however, is somewhere between New York and California, and American Airlines doesn’t know where it is yet. Sigh.