Tip on The Official Wireshark Blog

Troubleshooting the hidden dangers of TCP’s Nagle algorithm and delayed acknowledgement

Fri, 11 Jan 2013 23:58:10 +0000

As we all know, TCP/IP is a great protocol suite. However, there are times when it can become the bottleneck. This is especially true if you use TCP/IP for real time transactions where small data sizes are the norm (think financial institutions). In this session, I’ll show you why Nagle algorithm and delayed acknowledgement was developed. But more importantly, I’ll highlight the unintended consequences when the two features interact – badly – with each other. After watching this session, you will be able to spot the hidden dangers of using TCP/IP for real time transactions. Enjoy, and as always, I would really appreciate your feedback and suggestions. Here is the video:

http://www.youtube.com/watch?v=2CMueBcQNtk&feature=share&list=PL18B4C1339C54900A

And as always, any and all feedback and suggestion are welcome. Thank you and Enjoy!

Hansang Bae

Comments 🔗

Comment by credible58 on 2013-01-13 23:57:32 +0000 🔗

Great video, Hansang.

Comment by Hansang Bae on 2013-01-14 12:53:06 +0000 🔗

@credible58, thank you! If you have any ideas for future sessions, by all means, please let me know.

Comment by Jasper Bongertz on 2013-01-21 06:46:43 +0000 🔗

Nice presentation, Hansang, as usual. The only thing that I might have added to the slides would haven been an animation of what would be different without the push flags.

Comment by Hansang Bae on 2013-01-21 07:17:54 +0000 🔗

Ah, good point. I removed some popup hints, quizzes etc. because youtube doesn’t support it. That would have been a good one, especially since I talk about it on the PPT. BTW, I just spent an hour anonymizing an upcoming CIFS troubleshooting scenario. What a pain that was! 🙂

Comment by Andrew on 2013-01-23 12:52:27 +0000 🔗

Great presentation, you gave a very clear explanation of the rules on the message sequence chart that made understanding the actual packet trace easy. I also like that you made the packet trace available.

I’m left with one question though, why does packet #7 in the trace get to go immediately? This packet is only 226 bytes long so isn’t an MSS and there is unacknowledged data in flight (the bytes in #4, #5 and #6) so even though the Push bit is set in #7 my read of the Nagle rules on your slide #4 is that the only way #7 can be sent is if the Nagle override timeout expired? Yet the time between packet #7 and #6 is 16us which means the override timer cannot have expired. I’m presumably missing something though?

Comment by Hansang Bae on 2013-01-24 16:20:17 +0000 🔗

Andrew,
Give me a day or two. I’m on the road and will be back this weekend. I will reply when I’m back in the office. thanks

Comment by Hansang Bae on 2013-01-29 13:06:52 +0000 🔗

Andrew,
I’m glad you picked up on it. There are other small nuances in this trace that I didn’t get into . I didn’t delve into them just yet because this was more of an intro. 20/20 hindsight, I think it might have been better to tackle the bread and butter “nagle/delayed-ack” case first. But I’m very happy that you picked up on the fact that 10.10.10.10 probably has Nagle turned off. Another user Aaron also contacted me about packet #7! So I’m very happy that you guys are picking up on the small nuances of protocol analysis!

There are other oddities in the trace, and I may come back to it after a few more sessions.

Comment by Hai Wanxue on 2013-01-30 01:28:40 +0000 🔗

Great session.
I wrote something about TCP Nagle algorithm and Delayed ACK before in Chinese.
http://wenku.baidu.com/view/fbfda71aa300a6c30c229f99.html

Comment by Hansang Bae on 2013-01-30 08:00:45 +0000 🔗

Hai,
Thanks. Please feel free to download the presentation and pcap files and use it to your heart’s content! I see your presentation was much more thorough, was it for a class you were teaching? I can’t read it (if it was Korean, I could! 🙂 ) but I see enough TCP related acronyms to get the feel of it.

Thanks,

Hansang

Comment by Hai Wanxue on 2013-01-30 18:49:59 +0000 🔗

Hansang,
I am working at one company for technology consultant.
Writing it only as case because I often encountered them at work.
Thanks.

Comment by Anonymous Coward on 2013-02-05 17:33:39 +0000 🔗

These retransmissions are killing me!

Comment by Hansang Bae on 2013-02-05 21:23:25 +0000 🔗

That’s pretty funny, K! 🙂 Sorry folks, that’s a little insider joke!

Comment by jaspreet on 2013-02-19 23:18:53 +0000 🔗

Very very knowledgeable video. thanks for all the efforts taken to create this good stuff!

Comment by Hansang Bae on 2013-02-20 08:20:40 +0000 🔗

It’s my pleasure jaspreet! Glad you found it useful.

thnx

Hansang

Comment by krishnayeddula on 2013-03-06 19:34:11 +0000 🔗

Superb Video. It does nothing but enhances the Zeal to learn. That bus example was simple yet powerful.

Hansang,
People are thirsty here.Some more please.

Comment by Hansang Bae on 2013-03-09 18:33:37 +0000 🔗

@krishnayeddula, @Cyril, @Beulah, @Money..
Thank you. Another video will be coming out shortly. I’m glad that my passion for protocol analysis is coming through! 🙂

Looking forward to Sharkfest ’11

Thu, 12 May 2011 21:57:20 +0000

I’ve been looking over the session schedule for Sharkfest ’11. Once again Janice and Sheri have created an event which guarantees a wealth of knowledge and insight for everyone attending.

What to expect 🔗

Sharkfest is small. This is on purpose. We limit the size of the conference in order to allow more one-on-one communication between the attendees and presenters.

It has a high knowledge density. Our strategy is to gather together a bunch of people who are excited about Wireshark and protocol analysis, and know what the heck they’re talking about. We do our best to make sure the presentations focus on usable information with a minimum of fluff.

How to get the most out of Sharkfest 🔗

Sharkfest is active, not passive. Mingle. Compare notes. Many of the attendees are Wireshark power users, but many are not. Everyone has something insightful to share. The worst thing you can do is keep to yourself.

For the past three years I’ve had the opportunity to witness the top people in protocol analysis exchanging and sharing ideas. I look forward to seeing the same thing this year. See you there.

Comments 🔗

Comment by madinah on 2011-06-01 08:51:43 +0000 🔗

please, you know how to use wireshark

and how to read ports from wireshark

Comment by madinah on 2011-06-01 08:58:05 +0000 🔗

please, you know how to use wireshark

and how to read ports from wireshark

send step using wireshark

Running Wireshark as You

Thu, 04 Feb 2010 20:11:57 +0000

Running Wireshark on Linux involves an interesting challenge¹: Capturing packets requires root access, but Wireshark is big program and we strongly recommend against running it with elevated privileges. On Linux it’s common to see Wireshark running as root, but this is nearly unheard for similarly-sized applications like Firefox and GIMP. How can we avoid running Wireshark as root?

UPDATE 2010-02-10: Made changes suggested by Jaap and Balint.

A good way 🔗

Notice how I said “capturing packets requires root” above? Here’s a secret — Wireshark doesn’t capture packets. A separate program called dumpcap does. Compared to Wireshark, dumpcap is tiny. It’s much less complex and much safer to run as root. We can make it so that dumpcap runs as root and that only users in a particular group can run it:

$ sudo -s
# groupadd -g wireshark
# usermod -a -G wireshark gerald
# chgrp wireshark /usr/bin/dumpcap
# chmod 4750 /usr/bin/dumpcap

A better way 🔗

It’s also possible to let dumpcap do its job without involving root access at all. For a very long time Linux has allowed the use of fine-grained permissions called capabilities. In many recent distributions you can use the setcap utility to add capabilities to individual files.

Dumpcap needs CAP_NET_RAW and CAP_NET_ADMIN, so what do we need to feed setcap? On my Ubuntu Karmic system the setcap man page points you to cap_from_text. Cap_from_text points you to _cap_names, an array in the kernel. It would be nice if the setcap man page included a list of capability names along with a few examples. As it turns out, the names need to be in lower-case.

$ sudo -s
# sudo apt-get install libcap2-bin
# groupadd -g wireshark
# usermod -a -G wireshark gerald
# chmod 750 /usr/bin/dumpcap
# setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

You can also set these capabilities for Wireshark and TShark directly. Fully-functional filesystem capabilities is something the Linux world has needed for a very long time. I’m glad they’re finally seeing wide deployment.

Who’s Doing This? 🔗

Debian and Gentoo are using group-based permissions for Wireshark. Ubuntu is working on it. Hopefully the other distributions will follow suit.

1. This is a problem on other systems too, but it’s usually easier to solve. On Windows you can run the NPF service at startup. On OS X you can use ChmodBPF.

Comments 🔗

Comment by Jake on 2010-02-08 05:45:00 +0000 🔗

Note that the recent Debian packages already implement this, something you’ll notice when installing them. Thanks Balint. For Ubuntu it’s still on Wishlist status.

Comment by Balint on 2010-02-10 04:11:39 +0000 🔗

Gerald,
Could you please suggest using the wireshark system group instead of packetcapture?
Gentoo and Debian already use wireshark group, and it would be easier for newcomers to use the same group name everywhere.

I plan to update the Debian packages to use setcap, thanks for the hint.

Comment by Gerald Combs on 2010-02-10 10:59:09 +0000 🔗

I changed the example group to “wireshark” and mentioned Gentoo, Debian, and Ubuntu.

Comment by Balint on 2010-02-17 10:28:51 +0000 🔗

It would also make sense to use groupadd -g -r wireshark instead of groupadd -g wireshark to make the new group a system group. That would match and nicely coexist with Debian’s and Ubuntu’s installation method.

Holiday Gift Opening Tip

Thu, 17 Dec 2009 04:47:42 +0000

This is a blister pack:

We take gift-giving safety seriously here at the CACE Technologies World Domination Headquarters.

These are aviation shears:

This is what you can do to a blister pack in just a few seconds using aviation shears:

The shears were designed to cut sheet metal. They go through annoying packaging with precision and ease. You know those scissors that can cut through a penny? Aviation shears can cut through a penny and through those scissors.

They’re the best thing I’ve found so far for the job.

Javascript Background Animation

Fri, 13 Nov 2009 05:11:26 +0000

The Windows 7 taskbar does this nifty little background animation when you mouse over an open application’s icon. I wanted to see if the effect could be duplicated on a web page.

Suppose we have a couple of inline elements:

<div>
  <img class="bgmove" src="wsbadge-empty.png">
</div>

<h1>
  <span class="bgmove">Packets!</span>
</h1>

The image is mostly transparent:

We have a background that’s slightly larger than our elements, which lets us position it using negative offsets:

.bgmove {
  background-image: url('wsbadge-bg.png');
  background-repeat: no-repeat;
  background-position: -50px -20px;
}

jQuery lets us do two important things: track mouse motion and determine an element’s dimensions. This lets us shift the backround around when we mouse over each element:

$(document).ready(function(){
  $("div img.bgmove").pngFix();
    $(".bgmove").mousemove(function(e){
    var elWidth = $(e.target).outerWidth();
    var elHeight = $(e.target).outerHeight();
    var bgWidth = 279;
    var bgHeight = 95;
    var x = e.pageX - this.offsetLeft;
    var y = e.pageY - this.offsetTop;
    //var offX = -1 * (x * (bgWidth - elWidth) / elWidth);  // Against the mouse
    var offX = (x * (bgWidth - elWidth) / elWidth) - (bgWidth - elWidth);  // With the mouse
    var offY = -1 * (y * (bgHeight - elHeight) / elHeight); // Against the mouse
    //var offY = (y * (bgHeight - elHeight) / elHeight) - (bgHeight - elHeight);  // With the mouse
    bgPos = offX + 'px ' + offY + 'px';
    this.style.backgroundPosition = bgPos;
    });
})

You can see it all in action on the demo page. Combining this with jQuery animation is left as an exercise for the reader. The effect works in Firefox, Safari, Chrome, and IE 7+. IE 6 has trouble with the transparent PNG.

Debugging SSL Servers Over IPv6

Fri, 16 Oct 2009 21:00:50 +0000

An old-school method of debugging TCP-based services is to use telnet:

$ telnet www.wireshark.org 80
Trying 67.228.110.120...
Connected to www.wireshark.org.
Escape character is '^]'.
HEAD / HTTP/1.0
Host: www.wireshark.org

HTTP/1.1 200 OK
Date: Fri, 16 Oct 2009 19:31:47 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: max-age=3600
Vary: Accept-Encoding
X-Slogan: Be good. You never know who's running Wireshark nearby.
Content-Length: 9628
Connection: close
Content-Type: text/html

Connection closed by foreign host.

It’s like giving your web server a big ol’ hug.

Most telnet clients do something very clever here. If you connect to a port other than 23 (or whatever getservbyname returns when you feed it “telnet”) they will disable telnet protocol negotiation and switch to line mode. This gives you a raw, line-based connection which is just the thing you need to interact with an HTTP, POP, IMAP, FTP, or NNTP server.

Adding SSL and IPv6 to the mix complicates things. I’m in the process of making Wireshark’s public-facing services available over IPv6. It would be helpful to be able to test connectivity to each service before adding its corresponding AAAA record. Standard telnet clients support 6, but not SSL. OpenSSL’s s_client command speaks SSL, but not over IPv6 (not on my systems, at least):

$ openssl s_client -connect '[2607:f0d0:2001:e:1::123]:443'
getservbyname failure for f0d0:2001:e:1::123]:443

$ openssl s_client -connect ipv6.wireshark.org:443
gethostbyname failure
connect:errno=110

Luckily Fyodor released Nmap 5 a while back. Nmap 5 includes ncat, which lets you connect over SSL+IPv6. It is now my new favorite service-poking utility.

$ ncat -6 --ssl -v 2607:f0d0:2001:e:1::123 443
Ncat version 5.00 ( http://nmap.org/ncat )
SSL connection to 2607:f0d0:2001:e:1::123:443. bugs.wireshark.org
SHA-1 fingerprint: F6BA 2EE9 DEEF 74D3 B4B0 86D7 F5DB 6237 FF7F 896A
HEAD /bugzilla/ HTTP/1.0
Host: bugs.wireshark.org

HTTP/1.1 200 OK
Date: Fri, 16 Oct 2009 20:26:23 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8

On many Linux distributions you can also use telnet-ssl:

telnet-ssl -z ssl 2607:f0d0:2001:e:1::123 443

Comments 🔗

Comment by Martijn Pepping on 2009-10-19 08:40:32 +0000 🔗

Instead of the telnet-ssl command the openssl command can be used, which is more likely to be available on a system.

For example, the openssl command can be used as:
$ openssl s_client -connect hostname:443

In which ‘443’ represents the tcp-portnumber the SSL-service is listening on.

Comment by Gerald Combs on 2009-10-19 08:42:49 +0000 🔗

Does ‘openssl s_client -connect’ support IPv6 on your system? It doesn’t on any of mine. See the red error text in the post.

Comment by SSL Star on 2009-11-04 08:23:42 +0000 🔗

Great advice. I dont know how many times I have to tell people the very same things. Glad I’m not the only one.

Capture Filters and Offsets

Mon, 05 Oct 2009 13:03:53 +0000

A couple of questions have come up on the wireshark-users mailing list recently about using capture filters for MPLS and VLANs. Each user was having the same problem yet these are different network technologies — what do they have to do with each other?

The answer is offsets.

Let’s take an up-close and personal look at the capture filter “ip src host 10.16.32.48”. We can do this by running tcpdump -d, which takes a filter, compiles it, and dumps out the result. The dump of our filter looks like this:

(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 5
(002) ld       [26]
(003) jeq      #0xa102030       jt 4    jf 5
(004) ret      #96
(005) ret      #0

If this makes no sense don’t worry. You just need to know that the first two lines look for the IP ethertype (0x800) starting at byte 12 and the next two lines look for the IP address 10.16.32.48 (0xa102030) starting at byte 26. This is the minimum amount of checking required for that capture filter if you’re running IP over Ethernet.

What happens if you’re using 802.1q?

802.1q inserts an extra four bytes in front of the ethertype, so this filter won’t do what you want. The ethertype will be at offset 16 instead of 12, and the IP source address will be at offset 30 instead of 26. Libpcap and WinPcap don’t know you’re using .1q, and adding a check to the filter code would add a lot of unnecessary overhead. We have to add the word “vlan” to our filter to get the right offsets, e.g. “vlan and ip src host 10.16.32.48”:

(000) ldh      [12]
(001) jeq      #0x8100          jt 2    jf 7
(002) ldh      [16]
(003) jeq      #0x800           jt 4    jf 7
(004) ld       [30]
(005) jeq      #0xa102030       jt 6    jf 7
(006) ret      #96
(007) ret      #0

There’s a gotcha here, though. What if we change the “and” to an “or”?

(000) ldh      [12]
(001) jeq      #0x8100          jt 6    jf 2
(002) ldh      [16]
(003) jeq      #0x800           jt 4    jf 7
(004) ld       [30]
(005) jeq      #0xa102030       jt 6    jf 7
(006) ret      #96
(007) ret      #0

We’re looking for the IP address at byte 30. Shouldn’t we be looking at both 26 and 30?

The filter compiler uses a base offset for fetching data from the packet. Any time you use “vlan”, “mpls”, or “pppoes” in a capture filter, this offset is increased from that point on. It’s also cumulative. That is, while all of these filters are logically equivalent, they’re not in practice:

ip src host 10.16.32.48 or vlan or vlan (Looks for 10.16.32.48 at offset 26)

vlan or ip src host 10.16.32.48 or vlan (Now it’s looking at offset 30)

vlan or vlan or ip src host 10.16.32.48 (Offset 34. When will it ever end?)

“vlan” and “mpls” increase the base offset by 4. “pppoes” increases it by 8.

Wireshark’s Welcome Screen

Fri, 25 Sep 2009 16:04:28 +0000

Wireshark 1.2 added a nifty welcome screen which lets you start capturing, load a capture file, get help, and lots of other things. At the very top of the screen we brag about our popularity.

You can change that message to anything you like.

🔗

If you’re running Windows

Go to About→Wireshark→Folders.
Double-click on Personal Configuration.
Open preferences with a text editor, such as Notepad. Not a word processor, such as Word. This is a plain text file. Notepad good. Word bad. Very, very bad.

If you’re running OS X, Linux, or similar 🔗

Open ~/.wireshark/preferences in a text editor.

Now that you’re deep in the bowels of your preferences file 🔗

Find the following block of text:

# Custom start page title.

gui.start_title: The World's Most Popular Network Protocol Analyzer

# Show version in start page, can be useful in custom builds.

# TRUE or FALSE (case-insensitive).

gui.version_in_start_page: FALSE

Change gui.start_title to your custom message.
If you’d like the version to show up, change gui.version_in_start_page to TRUE.
Save the file and exit your editor.
Restart or start Wireshark.

Et voilà:

Filtering DSCP

Mon, 21 Sep 2009 22:34:34 +0000

The second byte in the IPv4 header (aka “those bits you’ve probably never, ever looked at”) is used for Differentiated Services, or DiffServ. It’s split into two parts: the 6 most significant bits define the DSCP (differentiated services code point) and the two least significant bits are for ECN (explicit congestion notification). You can use DSCP to divide your traffic into different classes. For example, Asterisk might use the following DiffServ value, which corresponds EF (Expedited Forwarding):

DSCP  ECN
10111000

If your networking equipment is sufficiently aware, this traffic will receive preferential treatment.

You can filter these values pretty easily using the ip.dsfield.dscp display filter — just right-click on the DSCP field in the packet like so:

Applying a DSCP display filter

What if you need to use DSCP in a capture filter?

To match against a particular DSCP codepoint using BPF (WinPcap/libpcap’s filtering language) you need to take the bit pattern, left-shift it two places to account for the ECN, and mask out the ECN. For EF (101110) you’d have do something like this:

Take 101110 and shift it left two bits: 10111000
Convert it to hex: 0xb8
Create a filter, masking out the ECN bits:

ip[1] & 0xfc == 0xb8

Cisco has a list of code points at http://www.cisco.com/en/US/tech/tk543/tk757/technologies_tech_note09186a00800949f2.shtml

P.S. To make matters more confusing, the DiffServ field was originally called “Type of Service.”

Missing Packets and Chimnies

Thu, 17 Sep 2009 00:33:43 +0000

You’ve just fired up Wireshark on your Windows Server 2003 or 2008 system and you’re not seeing nearly the amount of traffic you should. What’s happening?

The Windows Server 2003 Scalable Networking Pack introduced a feature called TCP Chimney Offload. Chimney offloading lets the OS networking stack hand off established TCP connections to the NIC for processing. This frees up the CPU, bus, and memory for other things and lets you scale up the number of connections you can handle. Hooray! Once the OS hands a connection off to the NIC, that traffic completely bypasses WinPcap and therefore doesn’t show up in Wireshark. You see the TCP connection setup and non-TCP traffic but no TCP data. Oops.

How do you fix the problem? It depends on your environment.

You can disable chimney offloading as described in KB 91222 (Server 2003) or KB 951037 (Server 2008). If you have a gigabit NIC you can probably get away with leaving it disabled. If you have a 10 gig NIC this might affect your performance. You can also SPAN or tap in and capture on an external machine, assuming you’re sufficiently equipped.

Tip from Laura: SMB2 Vulnerability Wireshark Filter

Tue, 08 Sep 2009 20:38:19 +0000

The big tech news this morning was a recently-discovered SMB2 vulnerability in Windows Vista, 2008, and 7. Laura Chappell created a Wireshark display filter for identifying offending traffic:

((smb.cmd == 0x72) && (smb.flags.response == 0)) && !(smb.pid.high == 0)

See the full report on her site for more information.

Comments 🔗

Comment by Jay on 2009-10-07 14:19:26 +0000 🔗

Hi,
I am doing some Software Performance Test(thru network environment) and Im using Wireshark to analyze it.
My Capture filter is ether host xx:xx:xx and Display filter is smb2.
What is the filter syntax should I use to remove the packet using GUID handle file:xxx under smb2 protocol.
Please help.