Category Archives: Protocols

Troubleshooting the hidden dangers of TCP’s Nagle algorithm and delayed acknowledgement

As we all know, TCP/IP is a great protocol suite.  However, there are times when it can become the bottleneck.  This is especially true if you use TCP/IP for real time transactions where small data sizes are the norm (think financial institutions).  In this session, I’ll show you why Nagle algorithm and delayed acknowledgement was developed.  But more importantly, I’ll highlight the unintended consequences when the two features interact – badly – with each other.  After watching this session, you will be able to spot the hidden dangers of using TCP/IP for real time transactions.   Enjoy, and as always, I would really appreciate your feedback and suggestions. Here is the video:

And as always, any and all feedback and suggestion are welcome.  Thank you and Enjoy!

Hansang Bae

Wireshark Tutorial Series. Tips and tricks used by insiders and veterans

For those of you who have attended Sharkfest in the past, you already know that protocol analysis is near and dear to my heart. It’s also a field where experience and art still matter. As great as Wireshark is as a tool, it still takes coaxing by an analyst to ferret out root cause. And as networks and applications become more complex, keeping up will be challenging.

But the one thing that I noticed over the years is that people rush to install sniffers without really thinking about it. It’s almost as if people expect sniffers to magically spit out the root cause, served on a silver platter! In reality, it takes fair amount of protocol and application knowledge to truly bring a tool like Wireshark to bear.

I started posting to this blog so that I can help budding protocol analysts and perhaps show interesting tricks-of-the-trade to veteran users. To become good in this field, it takes a fair amount of practice. It takes practice to know how to capture the right data, where to capture the data, what filters to use, and how to interpret the data. So how do you go about getting started? First, you can watch the accompanying video/tutorial session (see below for the link.) Next, make sure you setup your Wireshark in a consistent manner – the video tutorial covers this.

Ever wonder how router jockeys like me can scroll through a “sho run” output so quickly? It’s because I’ve done it for so long that the eyes are trained to filter out unneeded information. That’s the key to training – knowing what to filter out so your brain can get to work on the important stuff. It turns out protocol analysis works the same way. You have to train your brain to filter out the noise. Setting up your Wireshark environment will go a long way to maximizing productivity.

There is no “right way” to setup Wireshark. There’s only “my way” and everyone else’s – by definition – is wrong! Some like destination address to be the first column just like in DOS Sniffer. Others prefer using Wireshark’s default order. Whatever your style is, make sure it’s consistent. And if you’re just starting out, perhaps you can benefit from my setup. Even Anthony Bourdain in his book “Kitchen Confidential” talks about “mise-en-place.” It’s a term used by chefs and signifies how the cooking stations are setup. It’s important because it makes them more productive. For the same reason, you need to develop your own Wireshark mise-en-place!

If you still have not modified the default layout of Wireshark, you’re definitely missing out. In the video, I’m going to help you setup Wireshark so that you can become more productive. And we’re going to embark on a journey where I show you all the secrets to protocol analysis. I’m like the “magicians’ tricks revealed” guy. I’m going to help make you a rock star – where protocol analysis is concerned – in your company. If you’re an industry veteran, don’t be alarmed. The first few sessions are geared towards beginners so they can catch up. After that, I promise you that we’ll be in the weeds!

Hope you enjoy it, and I’d love to hear your comments. You can reach me at [email protected]

How To Read IPv6 Addresses

A common complaint about IPv6 is that addresses are “hard to read”. If you’ve been in the networking world any length of time IPv4’s dotted quad is most likely seared into your brain and clumps of hexadecimal digits of varying lengths can can be hard to wrap your head around. However, those clumps can provide useful information.

Below I’ll go over some of the address types I’ve seen and show you what information they provide. Continue reading

Debugging SSL Servers Over IPv6

An old-school method of debugging TCP-based services is to use telnet:

$ telnet www.wireshark.org 80
Trying 67.228.110.120...
Connected to www.wireshark.org.
Escape character is '^]'.
HEAD / HTTP/1.0
Host: www.wireshark.org

HTTP/1.1 200 OK
Date: Fri, 16 Oct 2009 19:31:47 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: max-age=3600
Vary: Accept-Encoding
X-Slogan: Be good. You never know who's running Wireshark nearby.
Content-Length: 9628
Connection: close
Content-Type: text/html

Connection closed by foreign host.

It’s like giving your web server a big ol’ hug.

Most telnet clients do something very clever here. If you connect to a port other than 23 (or whatever getservbyname returns when you feed it “telnet”) they will disable telnet protocol negotiation and switch to line mode. This gives you a raw, line-based connection which is just the thing you need to interact with an HTTP, POP, IMAP, FTP, or NNTP server.

Adding SSL and IPv6 to the mix complicates things. I’m in the process of making Wireshark’s public-facing services available over IPv6. It would be helpful to be able to test connectivity to each service before adding its corresponding AAAA record. Standard telnet clients support 6, but not SSL. OpenSSL’s s_client command speaks SSL, but not over IPv6 (not on my systems, at least):

$ openssl s_client -connect '[2607:f0d0:2001:e:1::123]:443'
getservbyname failure for f0d0:2001:e:1::123]:443
$ openssl s_client -connect ipv6.wireshark.org:443
gethostbyname failure
connect:errno=110

Luckily Fyodor released Nmap 5 a while back. Nmap 5 includes ncat, which lets you connect over SSL+IPv6. It is now my new favorite service-poking utility.

$ ncat -6 --ssl -v 2607:f0d0:2001:e:1::123 443
Ncat version 5.00 ( http://nmap.org/ncat )
SSL connection to 2607:f0d0:2001:e:1::123:443. bugs.wireshark.org
SHA-1 fingerprint: F6BA 2EE9 DEEF 74D3 B4B0 86D7 F5DB 6237 FF7F 896A
HEAD /bugzilla/ HTTP/1.0
Host: bugs.wireshark.org

HTTP/1.1 200 OK
Date: Fri, 16 Oct 2009 20:26:23 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8

On many Linux distributions you can also use telnet-ssl:

telnet-ssl -z ssl 2607:f0d0:2001:e:1::123 443

Filtering DSCP

The second byte in the IPv4 header (aka “those bits you’ve probably never, ever looked at”) is used for Differentiated Services, or DiffServ. It’s split into two parts: the 6 most significant bits define the DSCP (differentiated services code point) and the two least significant bits are for ECN (explicit congestion notification). You can use DSCP to divide your traffic into different classes. For example, Asterisk might use the following DiffServ value, which corresponds EF (Expedited Forwarding):

DSCP  ECN
10111000

If your networking equipment is sufficiently aware, this traffic will receive preferential treatment.

You can filter these values pretty easily using the ip.dsfield.dscp display filter — just right-click on the DSCP field in the packet like so:

Applying a DSCP display filter

Applying a DSCP display filter

What if you need to use DSCP in a capture filter? Continue reading

Does IPv6 Adoption Depend on Akamai?

The ongoing effort to switch the entire planet over to IPv6 has a chicken-and-egg problem: there is little incentive to deploy it if no one else is using it. This is expected to change as IPv4 addresses become more scarce, but for the time being uptake is dismal (in the U.S. at least).

One easy way to measure IPv6 adoption is to see how many of Alexa’s top sites have AAAA records. Hurricane Electric and Lars Eggert do a good job of this. Following in their footsteps my own version of this data is shown below:

IPv6 Adoption - Alexa top 1000

IPv6 Adoption - Alexa top 1000

It shows IPv6 adoption in Alexa’s top 1000 sites along with the number of sites using Akamai. The number of IPv6 sites is pretty small but the number of “testing” sites is encouraging. See the notes below for a detailed explanation and breakdown.

Who is Akamai? They’re a giant content and application delivery provider. Continue reading