What You Need to Know About Latency

Why proximity matters – About latency and the speed of light

Latency is today the least understood but most important factor in application performance.

Network performance is measured by two totally independent parameters: bandwidth (measured in megabytes per second) and round trip latency (measured in milliseconds).

While the average layman is more aware of bandwidth, in the world of broadband connectivity, the latency is actually the bottleneck for the performance of most business applications .

Faster responding applications lead to better productivity . People are highly sensitive to even small latencies . For example, Amazon has found that every 100 ms increase in load time of Amazon.com decreased their total sales by one percent.

If you ever used a laggy application that hesitates when performing specific operations , you know this is is extremely disconcerting .

The latency problem is made worse by “chatty” application protocols or naïve programs that were designed by failing to take latency into account .

Consider a simple straightforward script that deletes a folder . The script recursively deletes all the files in the folder, and then deletes the folder. Deleting every file is very quick, say 1 millisecond .

Now let’s say we have a folder with 10,000 files.

When running over a local file server , the script would run quite quickly , performing 10,000 deletions in 10 seconds.

But now say we want to run it over a file server that is located in a remote location with a round trip latency of 80 ms (typical latency from US east coast to west coast )

Now the operation takes 1ms but there is another 80ms round trip latency on top of it.  Now the same 10 second deletion process takes  13.5 minutes !  That’s 81 times slower . Talk about a monster productivity killer.  That would be unbearable .

Writing this script so that it would work correctly is challenging , one solution is to parallelize the script so that it deletes multiple files at once .  Say we use 10 threads – this would result in a tenfold decreasing of the total time to 1.3 minutes . Still not great.

A better approach would be to send 10,000 filenames to the server without waiting for response for each one . This approach, often called “pipelining” or “asynchronous processing” or “bulk operations” would decrease the execution time to just over 10 seconds, virtually eliminating the effect of latency. However, this depends on the network protocol actually supporting such asynchronous processing operations, which often they do not .   It also requires the client program (script in our case) to be written in a specific way that allows the operating system to properly queue the deletion operations use the asynchronous  facilities of the protocol (rather than a naïve flow that instructs the filesystem to delete one by one).

Another method to eliminate latency problems is file caching. For example , if I keep a local cache of the filesystem in a remote location, accessing the files in the cache can be achieved with zero latency; And if the cache is “write through” i.e. allowing local writes , the user would not observe the latency problem, since even write operations such as the deletion that we gave as an example , will be done locally, and then later propagated over the WAN . Caching has it’s own challenges , with different approaches to cache coherency, but this is a topic for another blog post.

Unlike bandwidth speeds of network links which are growing at a fast pace, latency is not expected to significantly drop over the foreseeable future , because it is governed by laws of physics, typically, the speed of light in fiber optic cable which is on the order of 200,000km per second. To put it another way, the signal will take 5ms to travel 1,000 kilometers in fiber.

Thus the round trip latency between Sydney and New York, a 16,000-kilometer distance, can never be less than 80 milliseconds, an in practice much more since the fiber in this case will probably travel a longer route, and there will be additional delays due to various network equipment in the path. In reality the observed latency for this link is a tad over 200ms.

Here’s a table of commonly observed round trip latencies on the Internet:

Latency is not going away , at least until we reach significant breakthroughs in quantum physics.

Latency is today the least understood , but most important factor in application performance.

To solve the latency problem software developers and IT engineers must be aware of software methods in order to optimize the communications – including multithreading, caching, and pipelining – so that to reduce the amount of round trips that need to be done over the wire .