Home
ryah [entries|archive|friends|userinfo]
ryah

[ website | tiny clouds ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

ascii doc [Jun. 28th, 2009|11:19 pm]
[Tags|]

http://www.methods.co.nz/asciidoc/
LinkLeave a comment

PHP Rewrite [Jun. 28th, 2009|05:06 pm]
[Tags|]

I don't know anything about PHP except that it's heavily used around the world.

I've never looked at the source code but I think it's safe to assume that it's poorly engineered for network concurrency - almost everything is.

A PHP process runs a FastCGI server which accepts connections from the reverse proxy (apache). Almost certainly the PHP process (probably via the fastcgi library) starts a new pthread for each connection. Extension modules, most notably MySQL, will do I/O which blocks that thread.

With great effort, one could re-engineer the underbelly of PHP to run on an event loop. Obviously one would need to support the same API, so it would all look blocking at the PHP layer - but underneath it would run in a single thread and be non-blocking. That is, implement green threads for PHP. Each request is would no longer be OS thread, but one green thread.

Most extension libraries will need to be re-written - more or less from scratch. But I suspect that if one could support the base library and the MySQL extension (and perhaps the memcached extension) that 90% of people's PHP programs would run on it.

File I/O would be done as it is in node.js, through the libeio thread pool. There is no official non-blocking MySQL library but work is being done in that area by Mysql-Proxy and http://drizzle.org/. Postgres has such a non-blocking library - so that extension would not be difficult. libmemcached certainly has a non-blocking interface.



Great Effort.

This would take a substantial amount of effort. It's more or less reimplementing PHP, except the parser and other low-level language stuff. On the other hand, there is so much sitting on top of PHP that this would be justified: a million websites would suddenly use 10% of their previous CPU load and half the memory.


edit: i retract this idea.
Link5 comments|Leave a comment

libev [Jun. 25th, 2009|06:48 pm]
Have I mentioned how much I like libev? You must have noticed, since I haven't written a thing without it for the past year and half. It's such a small and simple API and yet I continue to discover new things. Just this week, I learned about ev_ref() and ev_unref(). (They allow the event loop to drop out naturally even when a watcher is still in use. This is useful to me because I have to always watch for updates from the thread pool (via a pipe). It's almost perfectly bug free, works on every computer imaginable, and has a nice embedding API.

I consider it a work of art.
LinkLeave a comment

popen() for node [Jun. 21st, 2009|03:58 pm]
[Tags|, ]

I just implemented a neat process launching thing in Node. (It's the first time I've ever used vfork()!) It's great because it allows one to stream data in and out of child process. Here's a simple example:
 var cat = new node.Process("cat");
 cat.onOutput = function (chunk) {
   puts("cat said: " + chunk);
 };
 cat.onExit = function (status) {
   puts("cat exited with status " + status);
 };
 cat.write("hello");
 cat.write(" ");
 cat.write("world");
 cat.close();
It will be easy to implement Web Workers on top of this in pure javascript.

Check out the docs
LinkLeave a comment

Why isn't there a bidirectional popen()? [Jun. 19th, 2009|06:29 pm]
[Tags|]

Good read: http://lua-users.org/lists/lua-l/2007-10/msg00189.html


In node I think it will look like this
var cat = popen("cat");

cat.onOutput = function (chunk) {
  puts("cat said: '" + chunk + "'");
};

cat.onExit = function (code) {
  puts("the cat program exited with code " + code);
};

cat.write("hello world");
cat.close();
cat.kill(); 
Link2 comments|Leave a comment

projects using node [Jun. 16th, 2009|07:12 pm]
[Tags|, ]

I think all are in the early stages of development:

redis-node-client, node-json-rpc, and node_http_dispatch by Brian Hammond

pubsub by Malte Ubl

frisbee, A clone of the Disqus blog commenting system, by Urban Hafner

hxV8, the haXe → javascript compiler with modifications to use node I/O. by blackdog66

node_chat by me (live demo at chat.tinyclouds.org)
LinkLeave a comment

tip [Jun. 16th, 2009|01:44 pm]
Software you work with is best installed in your home directory. I install them in ~/local/software_name-version. This is usually done with a command like

./configure --prefix=$HOME/local/ruby-1.9.1

The advantage of this is that one can have many versions installed at once. (For example, many versions of Ruby.) Also, the software can be uninstalled with a simple rm -rf ~/local/ruby-1.9.1. The disadvantage is that you have many bin/ and lib/ directories and it's a pain to maintain the $PATH. To handle this, do the following in your .zshenv (or .bashrc or whatever)
for i in $HOME/local/*; do
  [ -d $i/bin ] && PATH="${i}/bin:${PATH}"
  [ -d $i/include ] && CPATH="${i}/include:${CPATH}"
  [ -d $i/lib ] && LD_LIBRARY_PATH="${i}/lib:${LD_LIBRARY_PATH}"
  [ -d $i/lib/pkgconfig ] && PKG_CONFIG_PATH="${i}/lib/pkgconfig:${PKG_CONFIG_PATH}"
  [ -d $i/share/man ] && MANPATH="${i}/share/man:${MANPATH}"
done
Link3 comments|Leave a comment

node version 0.0.3 [Jun. 11th, 2009|02:26 pm]
[Tags|, ]

* Many bug fixes including a major problem with http.Client on macintosh
* Upgrades v8 to 1.2.7
* Add onExit hook
* Guard against buffer overflow in http parser
* require() and include() now need the ".js" extension
* http.Client uses identity transfer encoding by default.
* connection.remoteAddress
* request.uri.queryKey renamed to request.uri.params

website
API documentation
node-0.0.3.tar.gz
LinkLeave a comment

Node (Another Server-Side Javascript) [May. 31st, 2009|03:21 pm]
[Tags|]

Node is a new server-side javascript project. It provides a purely event-based interface to I/O:

* TCP server and client
* Standard setTimeout() setInterval() timers
* Asynchronous file I/O
* HTTP server and client

Node's main focus is on performance and efficiency. It is built on top of

* V8 javascript
* libev, event loop abstraction
* libeio, file I/O thread pool

Node has the goal of supporting most POSIX operating systems (including Windows/MinGW) but at the moment it is only being tested on Linux, Macintosh, and FreeBSD. Node has no external dependencies (V8 and other libraries are included in its source tree.)

Node strives to provide complete and good documentation. This goal is not yet met but there is enough material to allow one to get started.

Node is released under a MIT license.

Release: node-0.0.2.tar.gz

Website: http://tinyclouds.org/node/

API documentation: http://tinyclouds.org/node/api.html

Git Repository: http://github.com/ry/node
Link3 comments|Leave a comment

operating system prediction [May. 26th, 2009|01:59 pm]
[Tags|]

In 10 years there will be one operating system for desktop computers with 99% market share. It will be Linux. (But don't worry, it thankfully won't come packaged with Gnome, KDE, or OpenOffice.)

IA-64 will be the singular CPU arch.
Link4 comments|Leave a comment

analogy [May. 17th, 2009|02:44 pm]
process : computer
 thread : core
LinkLeave a comment

favorite song [May. 5th, 2009|07:01 pm]


The Mae Shi - Run to Your Grave

Run, run, run to your grave
Run, run, run to your grave

Run, oh, run, run to your grave
Run, oh, run, run to your grave

Cause they're coming for your brain but
They will leave with your head
And they've got money and science
And they will leave you for dead

Sleep, sleep, sleep in your tomb
Sleep, oh, sleep, sleep in your tomb

Don't bury your body with your diamonds
Cause you know they'll dig up your grave
And don't hold on to your riches
Cause when you die, you're a slave

And emotion is a simple test to the synapse
Don't let it fool you into thinking that you've got brains
And the more you feel, the more you will take with you
So cut the flesh and let your blood flow to the drain

You've got to
Tear, burn, soil the flesh
God will do the rest
Scream, cry, pray, confess
God will do the rest

Scream, cry, pray, confess
God will do the rest
Scream, cry, pray, confess
God will do the rest
Link1 comment|Leave a comment

big servers and 64-bit v8 [May. 4th, 2009|11:44 pm]
[info]evan writes:
[...] When you have that much memory it's difficult to actually make use of all of it from a language like JavaScript because it's single-threaded. That is, the time it takes to read in and out (or more importantly, process) that much data starts making apps that would use more data unuseful. He suggested that part of the reason 64-bit Java was important was that there were these heavyweight servers with multiple cores that wanted to run a bunch of Java threads in the same heap simultaneously.

There's an interesting parallel here to servers. Say you can stick 10 terabytes of disk in a single machine -- it ends up not being too useful as a server unless it's archival, as in most of that data isn't accessed, because you're ultimately limited by disk bandwidth. Just trying to stream the contents of the disks at a sequential 50mb/sec** would take two and a half days. So when you're trying to serve real-time data off a disk (like, say, a search engine might) you're better off having more machines with smaller disks.
LinkLeave a comment

next generation of web UI [May. 3rd, 2009|05:02 pm]
I can't wait for SVG controlled <audio> elements and custom fonts.
Link5 comments|Leave a comment

RFC 1122 (Requirements for Internet Hosts -- Communication), section 4.2.2.13: Closing a Connection [May. 2nd, 2009|09:34 pm]
[Tags|]

A TCP connection may terminate in two ways:
  1. the normal TCP close sequence using a FIN handshake, and
  2. an "abort" in which one or more RST segments are sent and the connection state is immediately discarded.
If a TCP connection is closed by the remote site, the local application MUST be informed whether it closed normally or was aborted.

The normal TCP close sequence delivers buffered data reliably in both directions. Since the two directions of a TCP connection are closed independently, it is possible for a connection to be "half closed," i.e., closed in only one direction, and a host is permitted to continue sending data in the open direction on a half-closed connection.

A host MAY implement a "half-duplex" TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.

When a connection is closed actively, it MUST linger in TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). However, it MAY accept a new SYN from the remote TCP to reopen the connection directly from TIME-WAIT state, if it:
  1. assigns its initial sequence number for the new connection to be larger than the largest sequence number it used on the previous connection incarnation, and
  2. returns to TIME-WAIT state if the SYN turns out to be an old duplicate.


DISCUSSION:

TCP's full-duplex data-preserving close is a feature that is not included in the analogous ISO transport protocol TP4.

Some systems have not implemented half-closed connections, presumably because they do not fit into the I/O model of their particular operating system. On these systems, once an application has called CLOSE, it can no longer read input data from the connection; this is referred to as a "half-duplex" TCP close sequence.

The graceful close algorithm of TCP requires that the connection state remain defined on (at least) one end of the connection, for a timeout period of 2xMSL, i.e., 4 minutes. During this period, the (remote socket, local socket) pair that defines the connection is busy and cannot be reused. To shorten the time that a given port pair is tied up, some TCPs allow a new SYN to be accepted in TIME-WAIT state.

Here is my guess of what half-duplex close sequence means in POSIX.
client-program   client-kernel   server-kernel  server-program
      |                |               |              |
      |--shutdown()--->|               |              |        client: shutdown(fd, SHUT_WR)
      |                |-----FIN------>|              |
      |                |               |--read EOF--->|        server: read() returns 0
      |                |<----ACK-------|              |
      |                |               |              |
      |                |               |              |
      |                |      .        |              |
      |                |      .        |              |
      |                |      .        |              |
      |                |               |              |
      |                |               |              |
      |                |               |<--write()----|        server: optionally still writes data
      |                |<--------------|              |
      |<--- read()-----|               |              |        client: still reads data as normal
      |                |-----ACK------>|              |
      |                |               |<--write()----|
      |                |<--------------|              |
      |<--- read()-----|               |              |        client: still reads data as normal
      |                |-----ACK------>|              |
      |                |               |              |
      |                |               |              |
      |                |      .        |              |
      |                |      .        |              |
      |                |      .        |              |
      |                |               |              |
      |                |               |              |
      |                |               |<--close()----|        server: close(fd) or shutdown(fd, SHUT_WR)
      |                |<-----FIN------|              |                both have the same effect 
      |<--read EOF ----|               |              |        client: read() returns 0
      |                |-----ACK------>|              |
      |---close()----->|               |              |        client: close(fd)


Is this accurate?
LinkLeave a comment

TCP [May. 2nd, 2009|06:12 pm]
[Tags|]

I think the proper abstraction of a TCP socket (both sides) is something like this

events:
- connected
- received data
- send buffer drained
- got FIN
- done

methods:
- connect (for clients only)
- send
- close (sends FIN, shutdown(fd, SHUT_WR))

this is rather tricky to get right. (posix's unholy mixture of sockets and files aids to the confusion.) am i missing anything?
LinkLeave a comment

HTTP Parser [Apr. 27th, 2009|05:21 pm]
[Tags|]

I extracted the HTTP parser from libebb and beefed it up. It now handles HTTP responses, and I gave it a bit of documentation, made various other little improvements.

    * No dependencies
    * Parses both requests and responses.
    * Handles keep-alive streams.
    * Decodes chunked encoding.
    * Extracts the following data from a message
          o header fields and values
          o content-length
          o request method
          o response status code
          o transfer-encoding
          o http version
          o request path, query string, fragment
          o message body


source repo: http://github.com/ry/http-parser/tree/master

version 0.1: http://s3.amazonaws.com/four.livejournal/20090427/http_parser-0.1.tar.gz
Link2 comments|Leave a comment

multi-line CPP macros in vim [Apr. 27th, 2009|03:26 pm]
[Tags|, ]

How do you edit multi-line cpp macros in vim?
LinkLeave a comment

tipping point [Apr. 27th, 2009|11:25 am]
More Atheists Shout It From the Rooftops
They are connecting on the Internet, holding meet-ups in bars, advertising on billboards and buses, volunteering at food pantries and picking up roadside trash, earning atheist groups recognition on adopt-a-highway signs.

They liken their strategy to that of the gay-rights movement, which lifted off when closeted members of a scorned minority decided to go public.

“It’s not about carrying banners or protesting,” said Herb Silverman, a math professor at the College of Charleston who founded the Secular Humanists of the Lowcountry, which has about 150 members on the coast of the Carolinas. “The most important thing is coming out of the closet.”

Polls show that the ranks of atheists are growing. The American Religious Identification Survey, a major study released last month, found that those who claimed “no religion” were the only demographic group that grew in all 50 states in the last 18 years.
Link6 comments|Leave a comment

It's better in Somalia! [Apr. 25th, 2009|01:25 pm]
Link3 comments|Leave a comment

navigation
[ viewing | most recent entries ]
[ go | earlier ]

Advertisement