Latency and the Impact on Enterprise Application Performance

Latency is today the least understood but most important factor in application performance.

Network performance is measured by two totally independent parameters: bandwidth (measured in megabytes per second) and round trip latency (measured in milliseconds).

While the average user is more aware of how bandwidth impacts performance, in the world of broadband connectivity, it’s latency that is actually the bottleneck for the performance of most enterprise applications.

Latency as a Business-Critical Factor

Fast-responding applications lead to better productivity. As network performance improves on a global basis, users today are highly sensitive to even small latencies. A prime example is Amazon, which found that every 100 milliseconds (0.1 seconds) increase in the load time of its website decreased their total sales by one percent.

Your organization likely also has seen increased bounce rates when the website hasn’t loaded at top speed. And if you personally have ever used a laggy application that hesitates when performing specific operations, you know how extremely disconcerting – and frustrating – it can be.

The latency problem is made worse by “chatty” application protocols or naive programs that were designed by failing to take latency into account.

A Real-World Example

Consider a simple straightforward script that deletes a folder. The script recursively deletes all the files in the folder, and then deletes the folder. Deleting a file is a very quick exercise, around one millisecond (1 ms).

Now, let’s say we have a folder with 10,000 files.

The script running over a local file server would perform quite quickly, deleting the 10,000 files in 10 seconds.

But the script would not perform as well if it were running over a file server located in a remote location with a round trip latency of 80 ms (a typical latency for distributed organizations in the U.S. reaching from coast to coast).

Now the operation takes 1 ms but there is another 80 ms round trip latency on top of it. Therefore, the same 10 second deletion process would take…  13.5 minutes!

That’s 81X slower. Talk about a monster productivity killer.

Latency Solutions

What could an organization do to overcome the effects of latency on this deletion script? One solution would be to parallelize the script so that it deletes multiple files at once. Instead of a single thread, we could use 10 threads, which would result in a 10X performance gain and a decrease of the total time spent to 1.3 minutes. Still not great.

A better approach would be to send 10,000 filenames to the server without waiting for response for each one. This approach, often called “pipelining,” “asynchronous processing,” or “bulk operations” would decrease the execution time to just over 10 seconds, virtually eliminating the effect of latency.

However, this depends on the network protocol actually supporting such asynchronous processing operations, which often they do not. It also requires the client program (or script in our case) to be written in a specific way that allows the operating system to properly queue the deletion operations and use the asynchronous facilities of the protocol (rather than a naive flow that instructs the file system to delete one by one). A better approach, but also much more complicated.

The Caching Method

Another method to eliminate latency problems is through file caching. For example, if I keep a local cache of the file system in a remote location, then accessing the files in that cache can be achieved with zero latency.

And if the cache is “write through,” i.e. allowing local writes, the user would not observe the latency problem because even write operations such as file deletion will be done locally, and then later propagated over the WAN.

(Caching has its own challenges, with different approaches to cache coherence, but this is a topic for another blog post.)

Latency, Proximity, and the Speed of Light

Unlike bandwidth speeds of network links, which are growing at a fast pace, latency is not expected to significantly drop over the foreseeable future because it is governed by laws of physics. Typically, the speed of light in fiber optic cable measures approximately 200,000 km per second. To put it another way, the signal will take 5 ms to travel 1,000 kilometers in fiber.

Thus the round trip latency between Sydney and New York, a 16,000-kilometer distance, can never be less than 80 milliseconds. And in practice, it likely will be much longer because the fiber probably would travel a longer route, and there could be additional delays due to various network equipment in the path.

In reality the observed latency for the Sydney-New York link is a tad over 200 ms.

Here are some commonly observed round trip latencies on the Internet:

Latency is not going away; at least, not until we reach significant breakthroughs in quantum physics.

That’s why we believe that latency is today the most important factor in application performance, but one that’s least understood or appreciated by users.

To solve the latency problem software developers and IT engineers must be aware of software methods in order to optimize the communications such as WAN – including multithreading, caching, and pipelining – in order to reduce the amount of round trips that need to be done over the wire.

Aron Brand is CTERA’s Chief Technology Officer. Follow Aron on Twitter  and read his other posts.