In a previous post I showed how to write some custom firewall rules, and use a hack to get it to load automatically. With Leopard the hack to load it broke.

In steps a great new app for the system tweaker called Lingon (http://lingon.sourceforge.net). This handy little app lets you configure plist files and install them with just a few clicks.

Just select new, give it a name, give it a command to execute, select from a list of options how you want to to run and away you go. It even has an expert mode to dig deeper.

Just trying out the new iPhone WordPress application. I must say that it seems really slick.

The interface seems nice to work with, and if you’ve got the hang of the iPhone keyboard (ie. trusting the auto correction) it is a nice way to keep a blog going.

There is a very interesting link from Boing Boing about a piece by the New York Times on Robbie Cooper’s “Alter Ego: Avatars and their Creators” photography exhibition that shows portraits of people with their avatars from various games. It is interesting to see the particular features that they choose to include in their avatars, the improvements they choose to make and the escapes they create. The show is going to be at the Portland Art center this month. I just wish I was in Portland.

In general, I have found that network bandwidth requirements increase between 50-100% every year. With the prevalence of high speed broadband options to the home, user expectations have increased. Now that 6Mb cable modem service to the home is common, business class services shared among many users can at times seem inadequate.

There is a quantization that appears in network services. You can start with T1 or E1 circuits at 1.5Mb and bond them for more bandwidth. This works until you get to about 9Mb, at that point you generally need to start looking at options like a fractional DS3. The problem is that when you make that jump the cost increase significantly. To deal with the circuit you need a more expensive router, and the costs for the circuit and installation jump. Then to deliver more than 45Mb you need to jump up to OC3s which substantially increase the cost.

When you look at the consumer options like cable and FiOS and even DSL it is often hard for people to understand the different classes of service and the associated costs. There is a cost that comes with the reliability, low latency, committed bandwidth and service levels that come with business class service. The question remains, is that service and reliability commensurate with the cost.

When I look at the growth rate for bandwidth consumption, and what the options are for businesses, there really is no good option. We are already hitting that wall. The services are too expensive for a business to absorb, but the demand for bandwidth is far too great.

In many ways it is easier to run fiber out to the residential customers than it is to run it out to the businesses. In the suburbs you only need run a cable over a pole or dig a hole in someone’s yard. In the city you have to get building permits, redirect traffic, jackhammer through pavement, etc. So it remains to be seen whether business class service can keep up with residential offerings. The only thing I know is the providers have a long way to go, and the zombie mob of bandwidth hungry users are clawing at their door, so they better get started.

Oh that last mile. There are many providers of connectivity solutions out there, but getting them to your door can be a major challenge. There are initiatives to provide WiFi access, and some vendors are building their own facilities, but it doesn’t address the problem.

The Local Exchange Carrier (LEC)

The LECs are what we got when the Bell Telephone monopoly was broken up into smaller monopolies. They own the last mile into homes and businesses in their region. Connectivity either comes from them or rides the “last mile” over their lines. For example say you want to bring in a Sprint Internet circuit in Chicago. You would have to obtain a local loop from AT&T to complete the connection. This, of course, comes with an access charge, but it also complicates troubleshooting. You enter into a situation where multiple vendors are pointing fingers at each other.

All Carriers Are Not Created Equal

Not all of the LECs offer the same levels of service. If you look at the various Metro-IP options in the different markets, you can see this. Verizon has invested in building out their TLS network in New York, providing layer 2 networks across the city. AT&T picked up SBCs offerings which have been lagging behind other vendors. Their Opteman and Gigaman services compare poorly with other options. And if you look at what Qwest is offering in some markets or at Verizon’s FiOS service, you can see the difference.

The problem is that the LECs own the connectivity, so if the LEC is lacking there is no where to go. They are not really beholden to anyone. You can look at what the LECs have done to DSL providers in many areas. You hear stories all the time about a provider being told by the LEC they couldn’t deliver the service, only to have the LEC turn around and offer it directly. Or the many barriers placed to DSL installation.

Who Owns The Last Mile

In an ideal world, I would love to see the municipality take ownership of the cabling. The municipality could then subcontract the building and the management of the network out to the vendor, giving them specific performance goals. The contract would regularly be up for review, and the vendor would be replaced if they failed to meet the committed goals.

Of course this requires responsible government, and we all know how that goes. The idea is to remove the vendor lock in and give the user base a voice in their network, which is becoming increasingly critical. This also provides an opportunity to introduce desperately needed competition into the market.

Cities could also create incentives to bring multiple vendors in. By creating tax incentives, or just working with other the providers to do the construction necessary to lay the infrastructure. When infrastructure is put in, it should be opened to other vendors as well. Divergent fiber and competition, what more could we ask for.

My initial impression of VMWare Fusion is just, wow. While Parallels was the first out there, VMWare is now bringing their deep virtualization expertise to OS X now. The latest public beta really delivers on the Linux side. While Parallels seems to have their focus on the Windows market, VMWare has been building for the Linux market as well.

I started off by installing my favorite Linux desktop environment, Ubuntu. The first thing to note is the ease of installation. Creating a virtual machine was simple, quite similar to the Parallels experience. Feisty Fawn installed easily from an ISO image. I found the system to be very responsive.

Then I noticed the install VMWare tools option under the “Virtual Machine” menu. selecting this resulted in a vmware-tools image on my Ubuntu desktop with a gzipped file on it. After unpacking it, I opened a terminal and used sudo to launch a perl-based installer that asked a lot of questions. I would suggest using the -d option to use the default answers.

Once this was installed, I had many of the great options that I enjoyed with the Windows virtual machines under Parallels. I could move my mouse between OS X and Ubuntu without it getting locked into the virtual machine window. Now that I have this feature available, I could never go back.

The other thing that surprised me was that when I resized the window, Ubuntu adapted its resolution to what I set. I had enough trouble just getting X Windows to to work at all under Parallels, I never would have thought it could be this seamless.

It is not without its problems, and this is still a beta. I have had a few issues with my network interface dropping, but a quick /etc/init.d/networking restart got things going again. I do find that when operating in full screen mode there is a conflict between the Ubuntu menu bar and the OS X one that pops down.

Overall, I am impressed and look forward to spending more time with this. I would have to think that in the end, VMWare is going to have the edge over Parallels. They have a more established base, and the amount of pre-built virtual machines makes for a compelling offering.

The Omni Group has posted a demonstration video of their upcoming GTD (Getting Things Done) product OmniFocus. For those who are not familiar with the GTD methodology, it is based on a productivity book by David Allen called Getting Things Done: The Art of Stress-Free Productivity, and popularized by Merlin Mann over at 43 Folders. The basic goal is to get everything out of your head and into some trusted task management system so that you can stop stressing about it.

There are a bunch of systems out there based on this methodology, but this one looks like the right balance of organization, and not too much overhead. Hopefully it lives up to the promise and hype.

I can’t tell you how many times I have had to explain to someone who has just updated their connection to 10, 45, 100 Mb so they can quickly transfer there critical files to the other side of the planet only to find they aren’t getting any better throughput than they had before. This is a common misunderstanding about the relationship between bandwidth and throughput. The bottleneck is not the bandwidth, it’s the latency, and it is tough to argue with the speed of light.

The main problem comes down to our reliance on very effective and very old protocols that were developed for a different type of network. The problem here is TCP or Transmission Control Protocol. It is the widely used transport protocol that most of the internet relies on.

TCP provides the mechanisms to control the flow of packets across a network. It allows you to recognize when packets get lost and retransmit them. It can identify when the network begins to get congested and throttle back. It does a great many very useful things. The problem is that in order for all of this to happen the sender needs to receive an acknowledgment from the receiver or an ACK packet.

This is where the latency of a network comes in. As the round trip time between networks grows, the amount of data that can flow across a TCP stream goes down. TCP gets hung up waiting for the ACK packets and the transfer rates go down. This is why even though someone might put in 10Mb connections in Kansas and Jakarta they are disappointed when they get less than a megabit of throughput.

We are starting to see the limitations of the protocols the form the basis of network communication. This is why a whole new breed of network appliances are starting to emerge, the WAN accelerator. Without getting too complicated, something I’ll save for a later post, there are several ways in which you can make things go faster, and different products take advantage of different techniques.

Compression

If you compress the data going across your network there are fewer packets to send and you get greater throughput. This is of course assuming you can compress and decompress fast enough to make it worth your while and that your file is compressible. A lot of traffic will benefit from compression. Things like database log shipping or word documents will see the benefits of compression. Things that are already compressed or encrypted will see little or no benefit from compression, and in some cases it might be worse.

A lot of systems perform header compression, which reduces the amount of data being transmitted by compressing the packet’s header although this generally only results in minimal improvement.

Increase the window size

There is a certain amount of data that can be sent without needing to have an ACK packet sent. By significantly increasing this window, you can improve the throughput of a TCP stream. Typically most OSes have a 64K limit on their window size which is quickly exhausted on a high-bandwidth, high-latency network connection. By utilizing TCP extensions, this window size option can be increased to 1 Gigabyte.

This is a technique that is often used by acceleration appliances, and it can be very effective if there is not a lot of packet loss on the network. When there is a significant amount of packet loss this benefit can quickly disappear.

Caching

The fastest packet is one that doesn’t need to be transmitted. The biggest bang for your buck tends to come from Caching of data. Often when you think about caching, the idea of a file being stored and retrieved locally. Most of the accelerators use byte level caching. This means that they don’t look at files, but rather byte patterns. This gives the benefit of being able to cache portions of different documents that use the same chunks of data.

There is a huge amount of complexity around identifying and storing these byte patterns. This is a big differentiation point between the different vendors. Some do this very effectively and others not so much. Depending on the nature of your data, you might see certain vendors perform better than others.

Protocol specific acceleration

The other place that vendors seek to differentiate themselves is with protocol specific tricks that improve the throughput. This involves intelligent systems that can anticipate usage patterns and start pre-caching information in anticipation of it being used. If you open a file off of a server, the accelerator will know to start caching the rest of the file so it is there when you need it. There are tricks for decrypting and then compressing and caching portions of ssl traffic, and many more such optimizations. There is a wide variance in how the different vendors perform in different environments.

Convert TCP to UDP

UDP does not rely on ACK packets, it does not guarantee that packets make it to the other side. Because of this, UDP can traverse high latency networks with greater throughput than TCP. This means that the application or device that is doing the conversion needs to handle flow control and data integrity on its own, and do it more effectively than TCP. There are many products on the market that do this quite well.

The problem is that when you do these types of things you start to have problems with things having all the packets arrive in the correct order and prioritizing them across a network. So, depending on what you are send across the wire this may be a worthwhile technique. I see this approach most often used in software based acceleration options.

Break it into multiple TCP streams

Another technique is to break the single TCP stream into multiple parallel TCP streams. This way rather than waiting for the acknowledgments from a single stream you can have multiple ACK going across. You run into issues with reassembling the stream, packet order, etc., but you can saturate a circuit with TCP traffic this way.

Selective ACKs

Another TCP extension option is Selective Acknowledgments or SACKs that allow finer control over how packets are acknowledged, reducing the some of the penalties of packet loss and high latency network connections.

What’s next

So the question remains, if growth of bandwidth is outpacing TCPs ability to handle the throughput, where do we go. There really is a looming crisis in the protocols we use. They are deeply embedded in the applications we use every day, but at the same time, they are not performing at the level we now require. It remains to be seen if acceleration will be built into every network we build going forward or if there will be a shift away from some of the underlying protocols. I would have to believe there will be a limit to the benefits we will get out of acceleration, and that soon we will be talking about how to fix the network.

I like to use the blogging bundle from the great text editor TextMate to do my posting here. The other day I encountered an error that gave me some problems, and I just want to point out a workaround. After posting to this blog, I received and error that went something like this…

Received exception:Wrong size. Was 1275, should be <unknown>

The fix is to comment out a few lines in /usr/lib/ruby/1.8/xmlrpc/client.rb. The following lines are those you want to disable, they appear at line 547:

    expected = resp["Content-Length"] || "<unknown>"
    if data.nil? or data.size == 0
      raise "Wrong size. Was #{data.size}, should be #{expected}"
    elsif expected.to_i != data.size and resp["Transfer-Encoding"].nil?
      raise "Wrong size. Was #{data.size}, should be #{expected}"
    end

For those who are unfamiliar with how to do this, you can open Terminal.app and type…

    sudo vi /usr/lib/ruby/1.8/xmlrpc/client.rb

You will then be prompted for your password. You need to elevate your privileges because this file requires you to have administrator access to edit it.

Once in vi, you can use the command 547G to skip to line 547. You then need to put a “#” in front of the lines indicated above so it looks like this:

    #expected = resp["Content-Length"] || "<unknown>"
    #if data.nil? or data.size == 0
    #  raise "Wrong size. Was #{data.size}, should be #{expected}"
    #elsif expected.to_i != data.size and resp["Transfer-Encoding"].nil?
    #  raise "Wrong size. Was #{data.size}, should be #{expected}"
    #end

This will turn the code into a comment so that the error is no longer generated. When you are finished type :wq and hit enter to save the file and quit vi.

You can find more detail on this from the MacroMates site at http://macromates.com/wiki/Blogging/WordPress

There was an interesting quote I read today in Michael Shermer’s Skeptic column in Scientific American this month.

This administration intends to be candid about its errors. For a wise man once said, “An error does not become a mistake until you refuse to correct it.” We intend to accept full responsibility for our errors… We’re not going to have any search for scapegoats… the final responsibilities of any failure are mine, and mine alone.

This was from a speech that President John F. Kennedy gave after the failed Bay of Pigs invasion of Cuba. It speaks to a problem that is very common in any organization, the perpetuation of bad practices because of the perceived value of what has already been invested. As it is also know, sunk costs.

We all have a tendency to want to justify the bad decisions we make and humans have a great capacity for rationalization. This also makes me think of a book I read a while back called, The Logic of Failure, Recognizing and Avoiding Error in Complex Situations. The book has many examples of things that seems like sensible actions that lead to sometimes catastrophic problems. For those of you interested in this sort of thing you can find the book here.