Paul's Journal

Adoption of TLS Extensions

2012-09-07T17:17:17-07:00

TLS extensions expand the SSL/TLS protocols. The extensions have many uses, like adding more features, supporting more scalable patterns or making the protocol more secure. However, adoption has been disappointingly slow until the last few years as the reignited browser wars have kicked client vendors into action. I was unable to find recent statistics about the adoption of TLS extensions, so I went about figuring it out.

There are about 15-20 TLS extensions in specifications. Many however are rarely used, some of the most common and important extensions are:

Server Name Indication (SNI): Standardized in 2003, this extension enables browsers to send the hostname they intend to connect to, which solves the Virtual Host Problem in SSL. Without this extension every SSL certificate needs its own IP address to work. In 2005 I implemented SNI support in mod_gnutls, and in 2007 mod_ssl added support for SNI. Browser support lagged behind the servers, but is now thought to be widespread, excluding clients running on Windows XP. More: Wikipedia: Server Name Indication, RFC 6066
Session Tickets: The base SSL/TLS protocol includes Session Caching to reduce the number of expensive cryptographic operations a server needs to do if a client has previously visited it. This session caching relies upon the client sending a session ID, and the server storing data about that session. This model however is difficult to implement in a large scale server environment with many endpoints terminating SSL, as they would all need a shared caching infrastructure. Session tickets solve this by having the server give the client an encrypted ‘ticket’, which contains all of the information needed to resume the session, without the server needing an additional shared cache. Client support is widespread in Chrome and Firefox, but realistic server deployments still appear to be rare. In late 2011 I patched mod_ssl trunk to support configuration of session tickets, but the feature hasn’t been back ported to a release branch. More: Vincent Bernat: Speeding up SSL: enabling session reuse, RFC 5077
Next Protocol Negotiation (NPN): Used by the SPDY protocol to reduce round trips, this lets both the client and server agree on the protocol to run inside the encrypted connection. Without this extension, SPDY would require the use of additional round trips to upgrade from HTTP to SPDY. The extension is not yet an RFC, but has seen widespread adoption as Chrome and Firefox have both implemented support. Apache HTTP server added native support just 4 months ago. More: IETF Draft: Next Protocol Negotiation Extension
Renegotiation Indication: A TLS protocol bug was discovered in 2009 that allowed attackers to inject data into the stream read by the client. This extension was crafted to prevent this attack. A common mitigation was to disable all renegotiation once a connection was established, so the lack of this extension doesn’t necessarily indicate that a client is vulnerable, but it is a good indication of the age of the SSL/TLS stack being used by the Client. More: RFC 5746

User Agents Drive TLS Adoption

The adoption of TLS features and extensions is directly tied to the User Agent. While consumer websites and web browsers are important, I believe there has not been significant enough attention focused to Web Service API User Agents. Consumer browsers are now on much faster upgrade cycles, but many servers are not going to follow the same upgrade curves.

For this reason, I’ve collected samples from 3 different data sources:

monitoring.api.rackspacecloud.com: The API endpoint for the Rackspace Cloud Monitoring product. Most traffic is from Python, Java, and Ruby based API clients.
svn.apache.org: Primary version control site for the ASF. The majority of the clients are using Subversion Clients, but there is a smaller mix of browsers and other agents.
issues.apache.org: The most browser focused site that I could easily sample. It hosts the ASF JIRA and Bugzilla which are primarily used by consumer browsers.

If someone out there could sample a popular consumer site (Google? Facebook? Yahoo?) and post the results I would be very interested in seeing them.

Collecting Samples

It seemed too difficult to modify the existing server software to log all of the information that I wanted, and because in some cases the TLS termination is done in devices like a load balancer, I decided to build a tool to decode the information from a packet capture. All of the extensions I am interested in are sent by the Client in its ClientHello message. This means I didn’t need to do any cryptographic operations to decode it, just parse the TLS packet.

I started by using the excellent dpkt library to dissect my packet captures, but quickly figured out it didn’t actually parse any of the TLS extensions. A little patching later and I had it parsing TLS extensions.

The script I wrote handles the common issues I’ve seen, but still could be improved to do TCP stream re-assembly, but in practice with all the captures I made, the TLS Client Hello messages were in a single TCP packet.

If you want to try collecting and analyzing your own samples:

git clone git://github.com/pquerna/tls-client-hello-stats.git
cd tls-client-hello-stats
# Let tcpdump run for awhile, press ctrl+c to stop capturing.
sudo tcpdump -i eth0 -s 0 -w port443.cap port 443
python parser.py port443.cap

Observations

SSL/TLS Versions

TLS 1.0 is the version advertised by most clients and servers. The version spread for issues and monitoring are about what I would expect, but I was surprised to see that svn.apache.org was still seeing over 23% of its clients reporting SSLv3 as their highest supported version.

Deflate Support

While not an extension, deflate compression has to be advertised by both sides in order to support it. If used, it also imposes increased memory usage requirements on both the client and server, so I was interested in seeing if clients are advertising support for it.

OpenSSL enables the deflate compression by default, and until recent versions it was difficult to disable. I suspect that most of the monitoring traffic is using a default OpenSSL client library, and the more sophisticated browser user agents are explicitly disabling it. Since HTTP and SPDY both support compression inside their protocols, enabling deflate at the TLS layer would commonly lead to content being double compressed.

Number of Total Extensions Sent

It is interesting that most of the API centric clients send so few extensions. This seems to indicate potentially both the age of the TLS software stack being used, and the complexity of how it is configured by the developer.

SNI Support

I was disappointed to find a massive gap between consumer browsers and API consumers for SNI. This can be traced to common libraries not setting the SNI extension until recently. For example, only Python 3.2 or newer sends the SNI extension, and because ”Python 2 only receives bug fixes”, it will never be back ported for the most commonly deployed versions of the Python language.

Session Tickets

Session Tickets seem to have a more reasonable usage by non-browser user agents, but the consumer browsers are again leading adoption.

NPN Support

NPN support has been driven by the adoption of SPDY in Chrome and Firefox, so it isn’t surprising that for monitoring we see almost no support from clients.

Renegotiation Indication Support

While the Renegotiation Indication extension is sent by a significant number of clients on issues, its use is extremely low both svn and monitoring. This again shows how Browsers are leading the charge in upgrading, but also since the Renegotiation attacks require a man-in-the-middle, it would generally be a lower priority for server-to-server software.

Raw Data

I’ve posted a gist with the raw data for my three samples, if you wanted to look at the information for a more rarely seen extensions.

Conclusion

I think the data I’ve seen so far says a few things:

Server Name Indication: I was hopeful that SNI could soon be used in prime time, but I believe this is still years away with the usage numbers I’ve seen.
Session Tickets: I believe it is reasonable to put effort into using Session Tickets if you have more than one server doing SSL/TLS termination. Reducing the need to use distributed Session Caching eases implementation, and should result in a generally faster user experience.

I think it is great that Browser vendors like Chrome and Firefox are driving the use of newer features and extensions in TLS. It is obvious however that because API clients are commonly built by a more diverse set of developers, and those developers are less specialized in SSL/TLS security issues, that their adoption of the newest extensions is lagging. I hope this could change quickly if HTTP/2.0 and SPDY start driving the need to use NPN, and I hope that this would get developers to upgrade their SSL/TLS stacks.

Upgrades: SPDY, IPv6, FreeBSD, Jekyll

2012-09-05T17:17:17-07:00

Upgrades:

FreeBSD on Cloud Servers: This site is now being served from a Rackspace Open Cloud Server running FreeBSD 9. This includes using a PF firewall, ZFS root, etc.
HTTPS: The site now supports HTTPS. Sensitive business here on the blog ya know.
SPDY: Powered by node-spdy, the site is now available over HTTPS with the SPDY protocol.
IPv6: All new Rackspace Cloud servers include IPv6, so I’ve went ahead an added an AAAA record.
100% Static: I migrated a few months ago to a Jekyll based blogging system.
Monitoring: I’m checking if the site is up using Rackspace Cloud Monitoring, both over IPv4 and IPv6.

Details:

/etc/pf.conf: Allow inbound ports 22, 80 and 443, allow all outgoing.
/etc/sysctl.conf: Sets net.inet.ip.portrange.reservedhigh to 0, letting non-root users bind to ports bellow 1024. This lets me run my Node.js server without root, and without needing to figure out dropping privileges later, mostly because I’m being lazy and its my blog.
Node.js Server: Binds to both IPv4 and IPv6, HTTPS/SPDY and HTTP, and a few simple redirects. I’m logging to stdout, and using runit to keep it up.
Node.js from Ports: At first I was going to compile Node.js from scratch, but then I noticed that the FreeBSD ports collection provides it, and was pleasantly surprised to see it is well maintained – so I went with using it.
ZFS Root: I haven’t setup anything cool with ZFS yet, but I’m thinking about how to do a ZFS Send to Cloud Files.

Retaliatory Only Patents

2012-03-13T00:30:30-07:00

Today Yahoo launched a patent lawsuit against Facebook.

Yahoo has always said they collect patents for defensive purposes only. Then Yahoo’s newest CEO, Scott Thompson, is brought in, and gives choice quotes like this one, just 3 months ago:

“We’ll be back to innovation, we’ll be back to disruptive concepts,” he added. “I wouldn’t be here if I didn’t believe that was possible.”

Suing Facebook with patents is a disruptive concept. Yahoo just broke the patent mutually assured destruction stalemate in the valley. This also signals to all engineers that Yahoo is not interested in building disruptive products, and is instead a sinking ship.

I believe the patent system as it exists today is broken. Software patents have major issues. There are many things I would like to change, but I cannot. I also believe reform of the system as a whole is unlikely. Previously, I have chosen to try to ignore patents as much as possible. The trouble is if you ignore the patents your own company is at significant risk. Other companies don’t have the same moral beliefs about patents, and will use patents against you.

Retaliatory Only Patents

Many companies say they have a defensive only patent policy. But control of companies changes. Policies change and patents are granted for up to 20 years. Most technology companies also provide some sort of cash or other incentive to employees for filing patents of behalf of the company. Just imagine if you were an engineer at Yahoo in 2005. You made a cool new patentable idea, and went down the path of getting it patented. In 2010, you left Yahoo, as most sane people did, and could of even joined Facebook. Then 2 years after you left Yahoo, your patent, which you thought was going to be used for defensive purposes only, is used in an offensive suit against Facebook.

This kind of situation is exactly why I’ve tried to ignore patents for so long.

I think there is a better approach to motivating engineers, besides a bonus for patents.

If during the patent filing process, a company created a binding legal agreement to only use the new patent for defensive or retaliatory purposes, I would personally find this highly motivating. I am sure that the legal definition of “defensive or retaliatory” would take 20 pages of text to define, but I trust that lawyers can figure out the details. This kind of policy would make me feel much better about putting effort into filing patents for a company. If the company later changes control, they could change this policy, but it would only apply to new patents after that change in control.

I am not a lawyer, but what is stopping something like this from happening?

March 2012

2012-03-01T10:31:38-08:00

Vacation:

* March 2 to 15: Japan

* March 16 to 19: San Francisco

* March 19 to 30: Chile and Argentina

I don’t expect to be reading much email.

I’m still undecided about where/how to post pictures.

Designing Network Protocols

2012-02-22T18:24:10-08:00

Hacker News user peterwwillis started a discussion about a new network protocol introduced by the mod_heartbeat module in Apache 2.4:

It frustrates me when people use ASCII instead of packed bitmaps for things like this (packet transmitted once a second from potentially hundreds or thousands of nodes, that each frontend proxy has to parse into a binary form anyway before using it). Maybe it’s a really small amount of CPU but it’s just one of many things which could easily be more efficient.

This thread on HN continued with dozens of other posts from many authors, with peterwwillis holding his ground on his original point.

I disagree with the belief that a binary format should have been used and will attempt to show why the chosen network protocol for mod_heartbeat was both reasonable and correct.

Background

Apache 2.4 was released this week, 6 years after 2.2 was released. Compared to the 2.2 development cycle, where I was the Release Manager, I have not been as active in 2.4. However, one of the few features I did write for 2.4 was the mod_heartbeat module. mod_heartbeat is a method for distributing server load information via multicast. While I wrote mod_heartbeat 3 years ago, many other Apache HTTP Server developers have added features and bug fixes since then.

The primary use case is for use by the mod_lbmethod_heartbeat module, to direct traffic to the least loaded server in a reverse proxy pool.

The mod_heartbeat code and design was derived from a project at Joost. After stopping development of our thick client and peer to peer systems, we were moving to a HTTP based distribution of video content. We had a pool of super cheap storage nodes, which liked to die far too often. We built a system to have the storage nodes heartbeat with what content they had available, and a reverse proxy that would send clients to the correct storage server.

This enabled a low operational overhead around configuration of both our storage nodes and of the reverse proxy. Operations would just bring on a new storage node, put content on it, and it would automatically begin serving traffic. If the storage node died, traffic would be directed to other nodes still online.

Understand your goals

mod_heartbeat’s primary goal is: Enable flexible load balancing for reverse proxy servers.

For Joost we had good switches since we were previously setup for high packet rate peer to peer traffic. We also had previously used multicast for other projects. We choose to use a simple UDP multicast heartbeat as our server communication medium.

When designing the content of this heartbeat packet, I was thinking about the following issues:

* 10 to 200 servers: If you only have 10 nodes, you can do everything by hand. If you have hundreds of nodes, you are most likely building a hierarchical distribution of load. In my experience it is not a common configuration to have 10,000 application servers behind a single load balancer. I believe the sweet spot for this automatic configuration via multicast is pools between 10 and 200 servers.

* Multiple Implementers: The Apache HTTP server is all about being the flexible centerpiece of internet architectures, with many diverse producers, consumers, and interfaces. We must have a network protocol that is easily implemented in any programing language or enviroment, without adding additional dependencies.

* Extensibility: At Joost we embedded the available video content catalogs into the heartbeat advertisements. We needed a protocol that would be open to proprietary extensions without causing pain.

* Limited Network Impact: In a clustered systems you do not want the overhead of the cluster communication to negatively affect your application. It is important here to understand that many systems will actually hit packet-per-second limits before raw bandwidth limits. We also assumed at this point in time all systems have gigabit internal networking. In my experience the difference between a 20 byte packet and an 8 byte packet that is being multicasted once a second is not a relevant issue on modern LANs. Even with 1000 servers emitting packets, this is 19.53 KB/s of bandwidth. How efficient this network flow is will depend on your exact multicast configuration and your specific switches, but in most configurations it is a non-issue.

* Operability / Debug-ability: Wireshark and packet dumps are the best friend of a Network Admin. When people are doing packet dumps, they are looking for problems. A simple ASCII encoding of data will be easy for these people to see when they are in times of stress. Decoding a more complex binary encoding might get added as a feature to Wireshark someday, but it is yet another barrier

* Design for the long term: Design all public network protocols to be around for 10 years or longer. Include a versioning scheme. Don’t assume that 10 years from now your encoding system will still be around. I love msgpack for internal applications, but on these time scales for a public protocol, nothing beats straight up ASCII bytes.

What I did in 2007

Given the above considerations in 2007 at Joost, I started sketching out the possible formats for the multicast packet.

I considered using a binary format, but the immediate problem was having extendable fields. This meant we would need more than a few simple bytes. To create an extensible binary format, I started looking at serialization frameworks like Apache Thrift. At this time in 2007 Thrift had only been open sourced a few months, and it really wasn’t a stable project. It also didn’t have a pure C implementation, and instead would have added a C++ dependency to Apache HTTP server, which is unacceptable. Since 2007 the number of binary object formats like BSON, Google Protocol Buffers, Apache Avro, and Msgpack have exploded, but just 4 years ago there really weren’t any good standardized choices or formats for a pure-C project. The only existing choice would be to use ASN.1 DER, which would of implied a large external dependency, in addition to just being too complex. I decided that because of this and the other goals around debug-ability to peruse an ASCII based encoding of the content.

The choices for non-binary formats were:

XML: While XML is everywhere, and almost all languages have good bindings, it would be the most verbose choice. I also felt that it is too extendable. Someone later would add namespaces and other features that would make implementing a consumer much more difficult.
JSON: Easier to consume, and today there are libraries for all languages. A major problem was that in 2007, there were no good JSON parsers in pure C. I know this because at the same time I was working on libjsox, a pure C JSON parser with Rici Lake, and it was incomplete. (As an aside, YAJL is an excellent JSON parsing library for C that you should use now days). Like XML, JSON would also mean consumers would potentially have to handle more complex objects, rather than a simple key value pair.
Query parameters: RFC 3986 defined URLs, including the structure of query parameters. This format is understood by every component in a web server stack, and Apache already included examples of parsing this type of format. The format is also easy to build without external libraries, meaning reimplementation in any language is very easy. The use of a key and value system also means implementers can use simple data structures like a linked list or hash for interacting with their representation.

I made the decision to use query string style parameters as the best compromise for the content of the multicast packet’s content.

In the open source version of mod_heartbeat, there are two fields that are exposed today:

ready: The number of worker processes that are ready to accept new connections.
busy: The number of worker processes that currently servicing requests.

Adding the version string v=1, and then encoding the fields above we get something like this:

v=1&ready;=75&busy;=0

What would I change today?

If I were to need to implement the same system today, there are a few things I might change, but I don’t think any of them are critical mistakes given the original design constraints:

Consider using Gossip: Gossip based systems are more complex, but with more and more systems moving to Cloud based infrastructure, multicast communication is not a viable choice. Additionally, in some infrastructures, multicast can be problematic if not well configured, or if you have too many hosts joining and leaving the multicast group.
Consider using JSON: JSON is a more verbose format, but the availability of parsers in all languages, including C, has significantly improved. I still do not think Thrift or Protocol Buffers are ubiquitous enough to anoint one of them as the only way Apache HTTP Server transports data.

Conclusion

Binary encodings of information can be both smaller and faster, but sometimes a simple ASCII encoding is sufficient, and should not be overlooked. The decision should consider the real world impact of the choice. In the last few years we have seen the emergence of Thrift or Protocol Buffers which are great for internal systems communication, but are still questionable when considering protocols implemented by many producers and consumers. For products like the Apache HTTP server, we also do not want to be encumbered by large dependencies, which rules out many of these projects. I believe that the choice of ASCII strings, using query string encoded keys and values is an excellent balance for mod_heartbeat’s needs, and will stand the test of time.

Rackspace Open Sources Dreadnot, a Continuous Deployment tool

2012-01-05T13:55:40-08:00

Today we open sourced Dreadnot, our take on a Continuous Deployment tool. Details are posted over on the Rackspace Cloud Blog

Source is up on github.com/racker/dreadnot.

2011 Timecards

2011-12-31T05:11:20-08:00

Work Project:

Hobby Project:

2012 Goal

Finish the hobby project.

Created using Dustin’s git-timecard.

Write Logs for Machines, use JSON

2011-12-26T15:10:15-08:00

Logging for Humans

A printf style format string is the de facto method of logging for almost all software written in the last 20 years. This style of logging crosses almost all programing language boundaries. Many libraries build upon this, adding log levels and various transports, but they are still centered around a formated string.

I believe the widespread use of format strings in logging is based on two presumptions:

The first level consumer of a log message is a human.
The programer knows what information is needed to debug an issue.

I believe these presumptions are no longer correct in server side software.

An example of the problem

An example is this classic error message inside the Apache HTTP Server. The following code is called any time a client hits a URL that doesn’t exist on the file system:

ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r,
  "File does not exist: %s", r->filename);

This would generate a log message like the following in your error.log:

[Mon Dec 26 09:14:46 2011] [info] [client 50.57.61.4] File does not exist: /var/www/no-such-file

This is fine for human consumption, and for decades people have been writing Perl scripts to munge it into fields for a computer to understand too. However, the first time you add a field, for example the HTTP User-Agent header, it would break most of those perl scripts. This is one example of where building a log format that is optimized for computer consumption starts to make sense.

Another problem is when you are writing these format string log messages, you don’t always know what information people will need to debug the issue. Since you are targeting them for human consumption you try to reduce the information overload, and you make a few guesses, like the path to the file, or the source IP address, but this process is error prone. From my experience in the Apache HTTP server this would mean opening GDB to trace what is happening. Once you figure out what information is relevant, you modify the log message to improve the output for future users with the relevant information.

What if we logged everything into JSON?

If we produced a JSON object which contained the same message, it might look something like this:

{
    "timestamp": 1324830675.076,
    "status": "404",
    "short_message": "File does not exist: /var/www/no-such-file",
    "host": "ord1.product.api0",
    "facility": "httpd",
    "errno": "ENOENT",
    "remote_host": "50.57.61.4",
    "remote_port": "40100",
    "path": "/var/www/no-such-file",
    "uri": "/no-such-file",
    "level": 4,
    "headers": {
        <strong>"user-agent": "BadAgent/1.0",</strong>
        "connection": "close",
        "accept": "*/*"
    },
    "method": "GET",
    "unique_id": ".rh-g2Tm.h-ord1.product.api0.r-axAIO3bO.c-9210.ts-1324830675.v-24e946e"
}

This example gives a much richer picture of information about the error. We now have data like the User-Agent in an easily consumable form, we could much more easily figure out that BadAgent/1.0 is the cause of our 404s. Other information like the source server and a mod_unique_id hash can be used to correlate multiple log entries across the lifetime of an request.

This information is also expandable. As the knowledge of what our product needs to log increases, it is easy to add more data, and we can safely do this without breaking our System Admins precious Perl scripts.

Why now?

This idea is not new, it has just never been so easily accessible. Windows has had “Event Logs” for a decade, but in the more recent versions it uses XML. The emergence of JSON as a relatively compact serialization format that can be generated and parsed from almost any programming languages means it makes a great light weight interchange format.

Paralleling the big data explosion, is a growth in machine and infrastructure size. This means logging and the ability to spot errors in a distributed system has become even more valuable.

Logging objects instead of a format string enables you to more easily index and trace operations across hundreds of different machines and different software systems. With traditional format strings it is too fail deadly for the programmer to not log all the necessary information for a later operator to trace an operation.

Generating JSON with Log Magic

Log Magic is a small and fast logging library for Node.js that I wrote early on for our needs at Rackspace. It only has a few features, and it is only about 300 lines of code.

Log Magic has the concept of a local logger instance, which is used by a single module for logging. A logger instance automatically populates information like the the facility in a log entry. Here is an example of creating a logger instance for a module named 'myapp.api.handler and using it:

var log = require('logmagic').local('myapp.api.handler');

exports.badApiHandler = function(req, res) {
  log.dbg("Something is wrong", {request: req});
  res.end();
};

The second feature that Log Magic provides is what I call a “Log Rewriter”. This enables the programmer to just consistently pass in the request object, and we will take care of picking out the fields we really want to log. In this example, we ensure the logged object always has an accountId and txnId fields set:

var logmagic = require('logmagic');
logmagic.addRewriter(function(modulename, level, msg, extra) {
  if (extra.request) {
    if (extra.request.account) {
      extra.accountId = extra.request.account.getKey();
    }
    else {
      /* unauthenticated user */
      extra.accountId = null;
    }
    extra.txnId = extra.request.txnId;
    delete extra.request;
  }
  return extra;
});

The final feature of Log Magic is dynamic routes and sinks. For the purposes of this article, we are mostly interested in the graylog2-stderr, which outputs a GELF JSON format message to stderr:

var logmagic = require('logmagic');
    logmagic.route('__root__', logmagic['DEBUG'], "graylog2-stderr");

With this configuration, if we ran that log.dbg example from above, we would get a message like the following:

{
    "version": "1.0",
    "host": "product-api0",
    "timestamp": 1324936418.221,
    "short_message": "Something is wrong",
    "full_message": null,
    "level": 7,
    "facility": "myapp.api.handler",
    "_accountId": "ac42",
    "_txnId": ".rh-3dT5.h-product-api0.r-pVDF7IRM.c-0.ts-1324936588828.v-062c3d0"
}

Other implementations

There are many other libraries that are starting to emerge that can output logs in a JSON or GELF format:

* winston: (Node.js) A more complete (or complex?) logging module compared to Log Magic, but the prolific crew at Nodejitsu have done a great job.

* graypy: (Python) A graylog2 logger that interacts with the standard Python logging module.

* gelf4j (Java) We use a modified version of this library that logs to stderr instead of using UDP.

The Transaction Id

One field we added very early on to our system was what we called the “Transaction Id” or txnId for short. In retrospect, we could of picked a better name, but this is essentially a unique identifier that follows a request across all our of services. When a User hits our API we generate a new txnId and attach it to our request object. Any requests to a backend service also include the txnId. This means you can clearly see how a web request is tied to multiple backend service requests, or what frontend request caused a specific Cassandra query.

We also send the txnId to our user’s in our 500 error messages and the X-Response-Id header, so if a user reports an issue, we can quickly see all of the related log entries.

While we treat the txnId as an opaque string, we do encode a few pieces of information into it. By putting the current time and the origin machine into the txnId, even if we can’t figure out what went wrong from searching for the txnId, we have a place to start deeper debugging.

Transporting Logs

Since our product spans multiple data centers, and we don’t trust our LAN networking, our primary goal is that all log entries hit disk on their origin machine first. Some people have been using UDP or HTTP for their first level logging, and I believe this is a mistake. I believe having a disk default that consistently works is critical in a logging system. Once our messages have been logged locally, we stream them to an aggregator which then back hauls the log entries to various collection and aggregation tools.

Since all of our services run under runit, our programs simply log their JSON to stderr, and svlogd takes care of getting the data into a local file. Then we use a custom tool written in Node.js that is like running a tail -F on the log file, sending this data to a local Scribe instance. The Scribe instance is then responsible for transporting the logs to our log analyzing services.

For locally examining the log files generated by svlogd, we also made a tool called gelf-chainsaw. Since JSON strings cannot contain a newline, the log format is easy to parse, you just split up the file by \n, and try to JSON.parse each line. This is useful for our systems engineers when they are on a single machine, trying to debug an issue.

Collecting, Indexing, Searching

Once the logs crossing machines, there are many options to process those logs. Some examples that can all accept JSON as their input format:

* Perl Scripts (Hah! Did you think Perl will ever go away?)

* Graylog2 (open source)

* LogStash (open source)

* Loggly (SaaS)

* Splunk (Proprietary Software, can do JSON with an extra tool)

For Rackspace Cloud Monitoring we are currently using Graylog2 with a patch to support Scribe as a transport written by @wirehead.

Bellow is an example of searching for specific txnId in our system in Graylog2:

While this example is simple, we have some situations where a single txnId spans multiple services, and the ability to trace all of them transparently is critical in a distributed system.

Conclusion

Write your logs for machines to process. Build tooling around those logs to transform them into something that is consumable by a human. Humans cannot process information in the massive flows that are created by concurrent and distributed systems. This means you should store the data from these systems in a format that enables innovative and creative ways for it to be processed. Right now, the best way to do that is to log in JSON. Stop logging with format strings.

The Switch: Python to Node.js

2011-12-18T01:33:06-08:00

In my previous post, I glossed over our team switching from Python to Node.js. I kept it brief because the switch wasn’t the focus of the post, but since I believe I am being misunderstood, I will explain it in depth:

Cloudkick was primarily written in Python. Most backend services were written in Twisted Python. The API endpoints and web server were written in Django, and used mod_wsgi. We felt that while we greatly value the asynchronous abilities of Twisted Python, and they matched many of our needs well, we were unhappy with our ability to maintain Twisted Python based services. Specifically, the deferred programming model is difficult for developers to quickly grasp and debug. It tended to be ‘fail’ deadly, in that if a developer didn’t fully understand Twisted Python, they would make many innocent mistakes. Django was mostly successful for our needs as an API endpoint, however we were unhappy with our use of the Django ORM. It created many dependencies between components that were difficult to unwind later. Cloud Monitoring is primarily written in Node.js. Our team still loves Python, and much of our secondary tooling in Cloud Monitoring uses Python.

This attracted a few tweets, accusing various things about our developers, but I want to explore the topic in depth, and 140 characters just isn’t going to cut it.

Just how much Python did Cloudkick have?

We had about 140,000 lines of Python in Cloudkick. We had 40 Twisted Plugins. Each Plugin roughly corresponds to a backend service. About 10 of them are random DevOps tools like IRC bots and the like, leaving about 30 backend services that dealt with things in production. We built most of this code in a 2.5 year experience, growing the team from the 3 founders to about a dozen different developers. I know there are larger Twisted Python code bases out there, but I do believe we had a large corpus of experiences to build our beliefs upon.

This wasn’t just a weekend hack project and a blog post about how I don’t like deferreds, this was 2.5 years of building real systems.

It worked.

We were acquired.

Our Python code got the job done. We built a product amazingly quickly, built our users up, and were able to iterate quickly. I meant it when I said our team still still loves Python.

What I didn’t mention in the original post, is that after the acquisition, the Cloudkick team was split into two major projects – Cloud Monitoring, which the previous post was about, and another unannounced product team. This other product is being built in Django and Twisted Python. Cloud Monitoring has very different requirements moving forward – our goals are to survive and keep working after a truck drives into our data centers, and this is very different from how the original Cloudkick product was built.

What happened to Python then?

Simply put, our requirements changed. These new requirements for Cloud Monitoring included:

* Multi-Region availability / durability

* Multiple order of magnitude increases in servers monitored

* Scalable system, that can still be used 5 year from now. (Remember Rackspace Cloud grew 89% year over year right now)

Cloudkick was built as a startup. We took shortcuts. It scaled pretty damn well, but even if we changed nothing in our technology stack, it was clear we needed to refresh our architecture and how we modeled data.

The mixing of both blocking-world Django, and Twisted Python also created complications. We would have utility code that could be called from both environments. This meant extensive use of deferToThread in order to not block Twisted’s reactor thread. This created an overhead for every programmer to understand both how Twisted worked, and how Django worked, even if your project in theory only involved the web application layer. Later on, we did build enough tooling with function decorators to reduce the impact of these multiple environments, but the damage was done.

I believe our single biggest mistake from a technical side was not reigning in our use Django ORM earlier in our applications life. We had Twisted services running huge Django ORM operations inside of the Twisted thread pool. It was very easy to get going, but as our services grew, not only was this not very performant, and it was extremely hard to debug. We had a series of memory leaks, places where we would reference a QuerySet, and hold on to it forever. The Django ORM also tended to have us accumulate large amounts of business logic on the model objects, which made building strong service contracts even harder.

These were our problems. We dug our own grave. We should’ve used SQLAlchemy. We should’ve built stronger service separations. But we didn’t. Blame us, blame Twisted, blame Django, blame whatever you like, but thats where we were.

We knew by April 2011 that the combination of new requirements and a legacy code base meant we needed to make some changes, but we also didn’t want to fall into a “Version 2.0” syndrome and over engineering every component.

Picking the Platform.

We wanted some science behind this kind of decision, but unfortunately this decision is about programming languages, and everyone had their own opinions.

We wanted to avoid “just playing with new things”, because at the time half our team was enamored with Go Lang. We were also very interested in Python Gevent, since OpenStack Nova had recently switched to it from Twisted Python.

We decided to make a spreadsheet of the possible environments we would consider using for our next generation product. The inputs were:

Community
Velocity
Correctness (aka, static typing-like things)
Debuggability/Tooling
Downtime/Compile Time
Libraries (Standard/External)
Testability
Team Experience
Performance
Production

We setup the spreadsheet so we could change the weight of each category. This let us play with our feelings, what if we only cared about developer velocity? What if we only cared about testability?

Our conclusion was, that it came down to was a choice between the JVM platform and Node.js. It is obvious that the JVM platform is one of the best ways to build large distributed systems right now. Look at everything Twitter, LinkedIn and others are doing. I personally have serious reservations about investing on top of the JVM, and Oracles recent behavior (here, here) isn’t encouraging.

After much humming and hawing, we picked Node.js.

After picking Node.js, other choices like using Apache Cassandra for all data storage were side effects – there was nothing like SQL Alchemy for Node.js at the time, so we were on our own either way, and Cassandra gave us definite improvements in operational overhead of compared to running a large number of MySQL servers in a master/slave configuration.

Node.js? It has nested callbacks everywhere, thats ugly!

I think this is one of the first complaints people lob at Node.js when they just start. It makes a regular occurrence on the users mailing list – people think they want coroutines, generators or fibers.

I believe they are wrong.

The zen of Node.js is its minimalist core, both in size and in features. You can read the core lib Javascript in a day, and one more day for the C++. Don’t venture into v8 itself, that is a rabbit hole, but you can pretty quickly understand how Node.js itself works.

Our experience was that we just needed to pick one good tool to contain callback flows, and use it everywhere.

We use @Caolan’s excellent Async library. Our code is not 5 level deep nested callbacks.

We currently have about 45,000 lines of Javascript in our main repository. In this code base, we have used the async library as our only flow control library. Our current use of the library in our code base:

async.waterfall: 74
async.forEach: 55
async.forEachSeries: 21
async.series: 8
async.parallel: 4
async.queue: 3

I highly suggest, that if you are unsure about Node.js and are going to do an experiment project, make sure you use Async, Step, or one of the other flow control modules for your experiment. It will help you better understand how most larger Node.js applications are built.

Closing

In the end, we had new requirements. We re-evaluated what platforms made sense for us to build a next generation product on. Node.js came out on top. We all have our biases, and our preferences, but I do believe we made a reasonable choice. Our goal in the end is still to move our product forward, and improve our business. Everything else is just a distraction, so pick your platform, and get real work done.

PS: If you haven’t already read it, read SubStack’s great the node.js aesthetic post.

Technology behind Rackspace Cloud Monitoring

2011-12-17T18:28:03-08:00

Earlier this week we announced a new product: Rackspace Cloud Monitoring. It is just starting as a (free) private beta, so if you want to try it out, be sure to sign up via the survey here.

Transition from Cloudkick Technology

Rackspace Cloud Monitoring is based on technology built originally for the Cloudkick product. Some core concepts and parts of the architecture originated from Cloudkick, but many changes were made to enable Rackspace’s scalability needs, improve operational support, and focus the Cloud Monitoring product as an API driven Monitoring as a Service, rather than all of Cloudkick’s Management and Cloud Server specific features.

For this purpose, Cloudkick’s product was successful in vetting many parts of the basic architecture, and serving as a basis on which to make a reasonable second generation system. We tried to make specific changes in technology and architecture that would get us to our goals, but without falling into an overengineering trap.

Cloudkick was primarily written in Python. Most backend services were written in Twisted Python. The API endpoints and web server were written in Django, and used mod_wsgi. We felt that while we greatly value the asynchronous abilities of Twisted Python, and they matched many of our needs well, we were unhappy with our ability to maintain Twisted Python based services. Specifically, the deferred programming model is difficult for developers to quickly grasp and debug. It tended to be ‘fail’ deadly, in that if a developer didn’t fully understand Twisted Python, they would make many innocent mistakes. Django was mostly successful for our needs as an API endpoint, however we were unhappy with our use of the Django ORM. It created many dependencies between components that were difficult to unwind later. Cloud Monitoring is primarily written in Node.js. Our team still loves Python, and much of our secondary tooling in Cloud Monitoring uses Python. [EDIT: See standalone post: The Switch: Python to Node.js]

Cloudkick was reliant upon a MySQL master and slaves for most of its configuration storage. This severely limited both scalability, performance and multi-region durability. These issues aren’t necessarily a property of MySQL, but Cloudkick’s use of the Django ORM made it very difficult to use MySQL radically differently. The use of MySQL was not continued in Cloud Monitoring, where metadata is stored in Apache Cassandra.

Cloudkick used Apache Cassandra primarily for metrics storage. This was a key element in keeping up with metrics processing, and providing a high quality user experience, with fast loading graphs. Cassandra’s role was expanded in Cloud Monitoring to include both configuration data and metrics storage.

Cloudkick used the ESPER engine and a small set of EPL queries for its Complex Event Processing. These were used to trigger alerts on a monitoring state change. ESPER’s use and scope was expanded in Cloud Monitoring.

Cloudkick used the Reconnoiter noitd program for its poller. We have contributed patches to the open source project as needed. Cloudkick borrowed some other parts of Reconnoiter early on, but over time replaced most of the Event Processing and data storage systems with customized solutions. Reconnoiter’s noitd poller is used by Cloud Monitoring.

Cloudkick used RabbitMQ extensively for inter-service communication and for parts of our Event Processing system. We have had mixed experiences with RabbitMQ. RabbitMQ has improved greatly in the last few years, but when it breaks we are at a severe debugging disadvantage, since it is written in Erlang. RabbitMQ itself also does not provide many primitives we felt we needed when going to a fully multi-region system, and we felt we would need to invest significantly in building systems and new services on top of RabbitMQ to fill this gap. RabbitMQ is not used by Cloud Monitoring. Its use cases are being filled by a combination of Apache Zookeeper, point to point REST or Thrift APIs, state storage in Cassandra and changes in architecture.

Cloudkick used an internal fork of Facebook’s Scribe for transporting certain types of high volume messages and data. Scribe’s simple configuration model and API made it easy to extend for our bulk messaging needs. Cloudkick extended Scribe to include a write ahead journal and other features to improve durability. Cloud Monitoring continues to use Scribe for some of our event processing flows.

Cloudkick used Apache Thrift for some RPC and cross-process serialization. Later in Cloudkick, we started using more JSON. Cloud Monitoring continues to use Thrift when we need strong contracts between services, or are crossing a programing language boundary. We use JSON however for many data types that are only used within Node.js based systems.

Node.js ecosystem

We have been very happy with our choice of using Node.js. When we started this project, I considered it one of our biggest risks to being successful – what if 6 months in we are just mired in a new language and platform, and regretting sticking with the known evil of Twisted Python. The exact opposite happened. Node.js has been an awesome platform to build our product on. This is in no small part to the many modules the community has produced.

Here it is, the following is the list of NPM modules we have used in Cloud Monitoring, straight from our package.json:

async (rackers patched it)
cassandra-client (rackers wrote it)
cloudfiles
command-parser (rackers wrote it)
elementtree (rackers wrote it)
express
ipv6 (rackers patched it)
jade
logmagic (rackers wrote it)
long-stack-traces (rackers patched it)
magic-templates (rackers wrote it)
metrics
node-dev
node-int64
node-uuid
nodelint
optimist
sax
showdown
simplesets
strtok
swiz (rackers wrote it)
terminal (rackers wrote it)
thrift (rackers patched it)
whiskey (rackers wrote it)
zookeeper (rackers patched it)

Now that our product is announced, I’m hoping to find a little more time for writing. I will try to do more posts about how we are using Node.js, and the internals of Rackspace Cloud Monitoring’s architecture.

PS: as always, we are hiring at our sweet new office in San Francisco, if you are interested, drop me a line.