ryah

New HTTP Parser

2009-11-21T15:42:58Z

I've implemented a new HTTP/1.1 request and response parser by hand. (My previous parser was written with the help of Ragel.) It requires 124 bytes per HTTP connection, makes zero allocations, has no dependencies, is nearly optimal in its use of CPU instructions, interruptible on any character, has extensive tests, and is MIT licensed.

README

http_parser.h

http_parser.c

(Only one user at the moment: I've just merged it into Node.)

UC tuition

2009-11-19T21:01:15Z

http://www.nytimes.com/2009/11/20/education/20tuition.html

The University of California Board of Regents was expected to approve a plan on Thursday to raise undergraduate fees — the equivalent of tuition — 32 percent by next fall, to help make up for steep cuts in state funding.

[...]

Since the school year began, thousands of students have protested both the budget cuts and the proposal for higher fees, which would bring in-state tuition to more than $10,000 a year. On Wednesday 14 protesters, including 12 students, were arrested at U.C.L.A., for disrupting the meeting of the Regents Finance Committee, which was eventually closed to visitors.

working on presentation

2009-11-01T08:18:15Z

i hate beamer, but what else is there?

anvil shooting

2009-10-25T22:08:32Z

Smart Vanpool

2009-10-25T20:02:57Z

Vans are energy efficent when filled with 6 people, more so than any form of rail or bus transportation. (link via easwaran.) It seems creating an intellgent network of vans could be a good means of public transport. There are many factors involved, but it is imaginable that such a vanpool could be made profitable for the operators (not require city subsidies) and affordable for average people (vastly cheaper than a private taxi). I imagine vans with drivers, routed by a UMTS-capable navigation system to drop and pick-up passengers. The passengers request pick-ups via a telephone operator or website. They give a destination, and are given various choices of pick-up location—the ones more convenient for the vans in rotation will cost less, off-route locations cost more, short wait-times are more expensive, longer wait-times are cheaper. Passengers can be made to make transfers to other vans going in their direction. Perhaps instructions are sent to the passenger via SMS ("Get out at the next stop, wait 5 minutes for van 45"). The implementation of this is difficult, but tractable. It's certainly not on the same order of cost as installing light-rail infrastructure. Probably, a smart-vanshare program could be implemented with several millions of dollars—which is nothing when compared to digging tunnels for subway systems. Furthermore, the software development is a one-time cost, the system can be extended to other cities by simply adding vans, drivers, and telephone operators.

nginx_http_push_module

2009-10-16T11:51:34Z

http://github.com/slact/nginx_http_push_module

Nginx HTTP push module - Turn nginx into a long-polling message queuing HTTP push server.

If you want a long-polling server but don't want to wait on idle connections via upstream proxies or make your applications totally asynchronous, use this module to have nginx accept and hold long-polling client connections. Send responses to those clients by sending an HTTP request to a different location.

(I wish I had thought of this!)

Pro-Köln

2009-10-08T19:56:28Z

I forgot to post about our own local nazis. These posters were seen depressingly often in Cologne during the recent election:

"The Mayor AGAINST the Mosque!"
"He's for you!"

Stop / Yes to Minaret Ban

2009-10-08T19:52:16Z

http://www.tagesschau.de/ausland/minarett102.html
http://www.minarette.ch/

previously

four @ 2009-09-19T23:59:00

2009-09-19T21:58:26Z

Nice benchmarks. Must copy. (Had no idea the TCP backlog had any effect.)

raphaeljs

2009-09-19T13:13:31Z

http://raphaeljs.com/
very impressive demos

i would love to write a nginx stats module which used this...

yabm

2009-09-11T14:59:58Z

I made a little benchmark this morning with Facebook's new Tornado web server. (I am the only person who enjoys looking at response time histograms? I never see them elsewhere.)

Combined benchmark

2009-09-06T16:28:27Z

node v0.1.9
thin v1.2.1, ruby 1.8.7, em 0.12.8
v8cgi v0.6.0, apache 2.2.8 (Ubuntu)
narwhal/Rhino d147c160f11fdfb7f3c0763acf352b2b0e2713f7

The setup is similar to the last two benchmarks.

The point I want to make with these benchmarks is that Node is on the same order of magnitude in speed as production systems like EventMachine/Thin. Furthermore that at least two of the other proposed javascript systems are not on that order.

I think more can be done to clean up these numbers for Node. I haven't tried to optimize anything much yet.

1 concurrent client
completed requests:
node 23782
thin 22606
v8cgi 1990
narwhal 1234

> summary(node1$ttime)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  0.0000   0.0000   1.0000   0.7437   1.0000 106.0000 

> summary(thin1$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.000   1.000   1.000   1.122   1.000  74.000 

> summary(narwhal1$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  15.00   22.00   23.00   23.74   24.00   88.00 

> summary(v8cgi1$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  12.00   13.00   13.00   14.49   18.00   39.00

5 concurrent clients
completed requests:
thin 36899
node 34303
narwhal 2553
(v8cgi falls apart with more than one concurrent client)

> summary(node5$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00    2.00    3.00    3.79    4.00  142.00 

> summary(thin5$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  2.000   3.000   3.000   3.633   3.000 179.000 

> summary(narwhal5$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  16.00   42.00   51.00   58.12   63.00 5079.00

30 concurrent clients
completed requests:
thin 40575
node 35928
narwhal 2438

> summary(node30$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2.00   13.00   22.00   24.48   32.00  169.00 

> summary(thin30$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  10.00   18.00   19.00   21.64   20.00  133.00 

> summary(narwhal30$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  215.0   303.0   362.0   366.1   422.0   674.0

300 concurrent clients

completed requests:
thin 36045
node 35668
narwhal 2921

> summary(node300$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   12.0    66.0   112.0   239.4   157.0 12200.0 

> summary(thin300$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   71.0    84.0    87.0   208.7   107.0 23950.0 

> summary(narwhal300$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    928    2837    2935    2921    3018    8759

(raw data, scripts)

node vs v8cgi benchmark

2009-09-06T14:10:23Z

A simple benchmark between node.js v0.1.9 and v8cgi v0.6.0 (via the apache module, apache version 2.2.8 (Ubuntu)). I used ab -t 30 -g data.csv -c X for X=1 and X=2.

The v8cgi script I used was

response.write("hello world\n");

The node script I used was

body = "hello world\n";
node.http.createServer(function (request, response) {
  response.sendHeader(200, {
    "Date": "Sun, 06 Sep 2009 13:13:02 GMT",
    "Server": "Server: Apache/2.2.8 (Ubuntu) mod_v8cgi",
    "Content-Type": "text/html", 
    "Content-Length": body.length
  });
  response.sendBody(body);
  response.finish();
}).listen(8000);
puts("Server running at http://127.0.0.1:8000/");

One client at a time (-c 1)

Node completed 23389 requests
v8cgi completed 1971.

Two clients at a time (-c 2)

Node completed 28622 requests
v8cgi completed 66.

(raw data, scripts, configuration)

four @ 2009-09-05T23:08:00

2009-09-05T21:09:55Z

node vs narwhal benchmark

2009-09-05T13:19:09Z

A simple benchmark against two popular "Server-Side Javascript" programs: node.js v0.1.9 and Narwhal version d147c160f11fdfb7f3c0763acf352b2b0e2713f7 using the Rhino backend.

The script I used for Node was this

body = "Hello, Web!";
node.http.createServer(function (request, response) {
  response.sendHeader(200, {
    "Content-Type": "text/plain", 
    "Content-Length": body.length, 
  });
  response.sendBody(body);
  response.finish();
}).listen(8000);
puts("Server running at http://127.0.0.1:8000/");

and the script I used for Narwhal was what I found on the quick start page:

var jack = require("jack");

exports.app = jack.ContentLength(function (env) {
    return [200, {"Content-type": "text/plain"}, ["Hello, Web!"]];
})

Then I ran apache bench on the two of them, each for 30 seconds (-t 30), each with the -c 30 option. The web servers ran on my 32bit Linux system, 512mb ram, 1200 mhz - apache bench was running on a nearby computer. I recorded the data with -g and analyzed the data with R. The scripts I used and the data I collected are in this tarball.

Node completed 37018 requests.
Narwhal completed 2716.

Time series (smaller is better):

Response time histogram (left is better):

> summary(node$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00   14.00   20.00   23.68   31.00  158.00 

> summary(narwhal$ttime)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    187     269     313     329     368     992

ttime = total response time in milliseconds.

music / future

2009-08-31T12:10:08Z

Devendra Banhart -- Dragonflys

2009-08-19T13:19:54Z

I don't
owe me any money
You don't
owe me a thing
When we drink beer
Dragonflies appear
Dragonflies appear

connect(2)

2009-08-12T13:10:55Z

Connecting a non-blocking socket is tricky . Here is the essential trick from connect(2):

EINPROGRESS
              The  socket  is non-blocking and the connection cannot be completed immediately.  It is possible
              to select(2) or poll(2) for completion by selecting the socket  for  writing.   After  select(2)
              indicates  writability,  use  getsockopt(2)  to  read the SO_ERROR option at level SOL_SOCKET to
              determine  whether  connect()  completed  successfully   (SO_ERROR  is  zero)  or  unsuccessfully
              (SO_ERROR is one of the usual error codes listed here, explaining the reason for the failure).

node v0.1.3

2009-08-06T12:27:24Z

node.js development is coming along nicely!

I've just made the 10th release, v0.1.3. Fixing bugs, adding features. The project is still very rough—I think the average time-to-encountering-a-bug-after-first-compile is about 20 minutes. We've got a little IRC channel with 6 or 7 regulars and an usually active mailing list (which doubles as a bug tracking system).

non-blocking stdio hack

2009-08-03T09:43:10Z

http://s3.amazonaws.com/four.livejournal/20090803/nonblocking_stdio.tar.gz

 The problem with stdout, stderr, and stdin is that they are necessarily
 blocking when associated with a file. Like file descriptors for normal
 files, they cannot be used with select() or poll().

 If one does 
 
   my_program > /mnt/nfs_filesystem/output

 even printf() system calls might block the entire process's execution.
 This is bad for high performance servers which juggle many sockets and
 must never block execution.

 This function can help with that. Now STDOUT_FILENO, STDERR_FILENO, and
 STDIN_FILENO can be used with select(). All are set non-blocking and will
 act like pipe end-points (they are actually).

 This is done by internally using a ring buffer, pipe, and child process
 to pump data to the blocking file descriptor. 

 Compile with -pthreads. Call nonblocking_stdio() as quickly as possible
 after your program starts. Expect strange behavior with using printf()
 and friends--those are buffered and expect blocking file descriptors.

dns.c

2009-07-25T12:48:12Z

http://25thandclement.com/~william/projects/dns.c.html

A recursive, reentrant, non-blocking DNS resolver library in a single .c file. By William Ahern.

Download and compile it. I like how he does testing. Also how he does private variables in public structs. There's a lot to learn here.

ascii doc

2009-06-28T21:18:30Z

http://www.methods.co.nz/asciidoc/

libev

2009-06-25T16:52:24Z

Have I mentioned how much I like libev? You must have noticed, since I haven't written a thing without it for the past year and half. It's such a small and simple API and yet I continue to discover new things. Just this week, I learned about ev_ref() and ev_unref(). (They allow the event loop to drop out naturally even when a watcher is still in use. This is useful to me because I have to always watch for updates from the thread pool (via a pipe). It's almost perfectly bug free, works on every computer imaginable, and has a nice embedding API.

I consider it a work of art.

popen() for node

2009-06-21T14:04:06Z

I just implemented a neat process launching thing in Node. (It's the first time I've ever used vfork()!) It's great because it allows one to stream data in and out of child process. Here's a simple example:

 var cat = new node.Process("cat");
 cat.onOutput = function (chunk) {
   puts("cat said: " + chunk);
 };
 cat.onExit = function (status) {
   puts("cat exited with status " + status);
 };
 cat.write("hello");
 cat.write(" ");
 cat.write("world");
 cat.close();

It will be easy to implement Web Workers on top of this in pure javascript.

Check out the docs

Why isn't there a bidirectional popen()?

2009-06-19T16:29:14Z

Good read: http://lua-users.org/lists/lua-l/2007-10/msg00189.html

In node I think it will look like this

var cat = popen("cat");

cat.onOutput = function (chunk) {
  puts("cat said: '" + chunk + "'");
};

cat.onExit = function (code) {
  puts("the cat program exited with code " + code);
};

cat.write("hello world");
cat.close();
cat.kill();