Detecting Heartbleed Traffic

The big news in the tech industry this week is The Heartbleed Bug, a vulnerability that affects a large portion of secure web sites on the Internet. I updated the Wireshark and WinPcap web sites on Monday (along with reissuing and revoking certificates) shortly after OS patches were released.

Our web sites are protected going forward, but what about the past? We have a Shark appliance in our environment but that leads to a challenge. We had about 350 GB of HTTPS on our network on Monday alone. This is just slightly too large to load into Wireshark.

Fortunately one of my coworkers (P.J. Malloy) came up with a BPF filter that matches Heartbleed traffic:

http://www.riverbed.com/blogs/Retroactively-detecting-a-prior-Heartbleed-exploitation-from-stored-packets-using-a-BPF-expression.html

Applying this filter directly on the Shark appliance gave me a much smaller number of packets which I could easily analyze in Wireshark. So far I haven’t found anything suspicious.

We’re switching to Qt.

Today I released the next development version of Wireshark, 1.11.0. This marks a major change in the direction of the project. We’re switching our user interface library from GTK+ to Qt. Both libraries make it easy for developers write applications that will run on different platforms without having to rewrite a lot of code. GTK+ has had a huge impact on the way Wireshark looks and feels and on its popularity but it doesn’t cover our supported platforms as effectively as it should and the situation is getting worse as time goes on.

Making such a large change wasn’t an easy decision. It means rewriting thousands of lines of code and requires a lot of careful design. We might be the largest standalone application to make this transition (feel free to correct me below). However, I think it’s well worth it and that it’s important to the long-term direction of the project. Ultimately it came down to one thing:

Wireshark’s job is to show you what’s happening on your network. If it can’t run on your system then it’s not doing that job.

If you’re using Windows, Mac OS X, or Linux Mint we need to support Windows, Mac OS X, and Linux Mint. If you’re using an iPad or a Galaxy Note we need to give you a long, hard, nonplussed stare and think about supporting IOS and Android at some point.

When I started writing Ethereal (Wireshark’s original name) it looked like this:

ethereal-0.2.0

It looked like that on Linux and Solaris. It didn’t look like that anywhere else because those were the only two platforms we supported. I chose GTK+ for the user interface toolkit because it made the most sense. Compared to the options available at the time it had a number of advantages. It was small, light, easy to work with, had an active development team, and had a compatible license. A short time later we added support for Windows. GTK+ had some initial compatibility issues on Windows but it has gotten better over time.

Since then the platform landscape has changed but unfortunately GTK+ hasn’t. Most notably Mac OS X has gained popularity and people are starting to ask about Wireshark for their tablets. GTK+ supports OS X but it’s definitely a second-tier platform, making it hard to install and use. We either have to require X11 (like Inkscape) or use an experimental native GTK+ port (like GIMP). Either way it requires a lot of effort on the part of developers to produce a substandard application for users. With GTK+ Wireshark doesn’t look or act at all like a Mac OS X application:

osx-x11

Qt on the other hand provides a nice, clean user experience on all of the platforms that we currently support. If you install the 64-bit version it now looks like this:

osx-qt

What does this mean for users?

If you’re a power user you should probably keep using the GTK+ flavor for the time being. You should also test the Qt flavor from time to time to see how your workflow will change. Both are installed by default on Windows and Linux.

If you’re running OS X you should use the Qt flavor. For common tasks it should have a better workflow. Again, if it doesn’t we aren’t doing our job.

What does this mean for developers?

If you’re developing a new feature using GTK+ you should stop. You’re very likely wasting your time. If you would like to help with the migration grab the code and start developing.

What works?

Everything under the “File” and “Edit” menus.

You can capture, filter, and inspect traffic.

You can edit preferences.

You can follow streams.

You can view TCP stream graphs.

What doesn’t work?

Everything else. No capture options, I/O graphs, flow graphs, VoIP analysis, etc. We still have a lot of work to do.

Switching to Qt is a long and arduous process but I’m excited about what the future holds.

P.S. Solaris is still supported.

solaris-qt

Wireshark Tutorial Series #2. Tips and tricks used by insiders and veterans

Yes, I know it’s been a while since tip #1 video (https://blog.wireshark.org/2012/10/wireshark-tutorial-series/?utm_source=rss&utm_medium=rss&utm_campaign=wireshark-tutorial-series) and this one. Judging by the number of views and comments, it is helping. So keep me honest by reminding me to post more often!

In this short video (http://www.youtube.com/watch?v=aIiosBw2YH4), I discuss the dangers of using default values without fully understanding what the consequences are. In Sharkfest 2013, Christian Landström gave an excellent session on the reassembly feature of Wireshark. Unfortunately, it wasn’t recorded and I wanted to convey the message. The PDF of his excellent session can be found here:  http://tinyurl.com/lko37zb

Enjoy!

Hansang Bae

Fifteen Years

Fifteen years ago I released a little network protocol analyzer. At the time it wasn’t very special. It only dissected five protocols and only ran on Linux and Solaris. I decided to share it with the world and released it as open source. I made use of quite a bit of open source software at that point (and still do), and it seemed like a good way to give back to the community.

Immediately after the release an amazing thing happened — I started receiving code from people around the world. They had problems similar to mine and were able to modify the little analyzer to suit their needs. They were also kind enough to contribute those modifications back. Those contributions haven’t stopped to this day and Wireshark has grown into a mature, feature-rich, award-winning network analysis tool. People around the world use it to troubleshoot networks, develop software and protocols, and to learn about networking.

Wireshark has been a source of pride many times over the years but I’m particularly proud of two accomplishments. First, your network is not a black box. This is important inasmuch as our daily lives depend on networks operating efficiently, reliably and securely. Wireshark lets you peer into your network and see how it operates at a low level. It’s also accessible. Anyone can download and run it. Protocol analysis is a fascinating world and if you don’t understand what you’re looking it we have a large community of users and educators that can help you.

Second, it turns out that a protocol analyzer makes a great open source project and Wireshark is an example of open source at its best. People who know a lot about protocols are typically know how to write software. If you provide a platform that allows them to give meaning and context to the bits and bytes that make up network packets they will do so on a massive scale. Wireshark now supports over 1,000 protocols and 140,000 different protocol fields and those numbers keep growing.

We’ve had quite a few challenges in the past and we still do. These days all of your interesting traffic is either off in the cloud or speeding across your LAN at multiple gigabits a second. Wireshark doesn’t run on my tablet and it looks awful on my Macbook. However, these are solvable problems and I’m looking forward to the challenge of fixing them.

When I made that first release I had no idea how big it would become or how much it would impact my life. Wireshark is a great and wonderful thing due to the following people:

  • The development team, who seem to generate an endless stream of brilliance and cleverness.
  • The user community, for their knowledge and enthusiasm, and for putting up with the odd wart here and there.
  • Educators, researchers, developers, and everyone else who help people peer into their networks.
  • My employer for sponsoring the project and being such an awesome place to work, and my co-workers for their passion for network performance.
  • My family and friends, for their support and encouragement over the years.

It’s been a wonderful journey so far and I can’t wait to see what lies ahead. Thank you all.

Troubleshooting the hidden dangers of TCP’s Nagle algorithm and delayed acknowledgement

As we all know, TCP/IP is a great protocol suite.  However, there are times when it can become the bottleneck.  This is especially true if you use TCP/IP for real time transactions where small data sizes are the norm (think financial institutions).  In this session, I’ll show you why Nagle algorithm and delayed acknowledgement was developed.  But more importantly, I’ll highlight the unintended consequences when the two features interact – badly – with each other.  After watching this session, you will be able to spot the hidden dangers of using TCP/IP for real time transactions.   Enjoy, and as always, I would really appreciate your feedback and suggestions. Here is the video:

And as always, any and all feedback and suggestion are welcome.  Thank you and Enjoy!

Hansang Bae

Wireshark Tutorial Series. Tips and tricks used by insiders and veterans

For those of you who have attended Sharkfest in the past, you already know that protocol analysis is near and dear to my heart. It’s also a field where experience and art still matter. As great as Wireshark is as a tool, it still takes coaxing by an analyst to ferret out root cause. And as networks and applications become more complex, keeping up will be challenging.

But the one thing that I noticed over the years is that people rush to install sniffers without really thinking about it. It’s almost as if people expect sniffers to magically spit out the root cause, served on a silver platter! In reality, it takes fair amount of protocol and application knowledge to truly bring a tool like Wireshark to bear.

I started posting to this blog so that I can help budding protocol analysts and perhaps show interesting tricks-of-the-trade to veteran users. To become good in this field, it takes a fair amount of practice. It takes practice to know how to capture the right data, where to capture the data, what filters to use, and how to interpret the data. So how do you go about getting started? First, you can watch the accompanying video/tutorial session (see below for the link.) Next, make sure you setup your Wireshark in a consistent manner – the video tutorial covers this.

Ever wonder how router jockeys like me can scroll through a “sho run” output so quickly? It’s because I’ve done it for so long that the eyes are trained to filter out unneeded information. That’s the key to training – knowing what to filter out so your brain can get to work on the important stuff. It turns out protocol analysis works the same way. You have to train your brain to filter out the noise. Setting up your Wireshark environment will go a long way to maximizing productivity.

There is no “right way” to setup Wireshark. There’s only “my way” and everyone else’s – by definition – is wrong! Some like destination address to be the first column just like in DOS Sniffer. Others prefer using Wireshark’s default order. Whatever your style is, make sure it’s consistent. And if you’re just starting out, perhaps you can benefit from my setup. Even Anthony Bourdain in his book “Kitchen Confidential” talks about “mise-en-place.” It’s a term used by chefs and signifies how the cooking stations are setup. It’s important because it makes them more productive. For the same reason, you need to develop your own Wireshark mise-en-place!

If you still have not modified the default layout of Wireshark, you’re definitely missing out. In the video, I’m going to help you setup Wireshark so that you can become more productive. And we’re going to embark on a journey where I show you all the secrets to protocol analysis. I’m like the “magicians’ tricks revealed” guy. I’m going to help make you a rock star – where protocol analysis is concerned – in your company. If you’re an industry veteran, don’t be alarmed. The first few sessions are geared towards beginners so they can catch up. After that, I promise you that we’ll be in the weeds!

Hope you enjoy it, and I’d love to hear your comments. You can reach me at hans...@riverbed.com

Sharkfest ’12 Recap

The fifth annual Sharkfest wrapped up a couple of days ago. You are no doubt feeling terrible if you missed it. Don’t despair, you can find a retrospective at the Sharkfest web site. The events team tried to make sure each session was recorded so we should have videos to go along with the slides soon.

As usual I wanted to attend more presentations than physics allowed but unfortunately missed some, so I’m waiting for the retrospective videos along with everyone else. (Hansang Bae observed that this is one of the few conferences where presenters attend other sessions.) Here are some quick notes about the sessions I was able to attend:

Monday’s keynote was delivered by Cliff Stoll. He kept insisting that he didn’t have much to say while simultaneously delivering an informative and entertaining talk.

Steve Riley provided a lot of insights into cloud computing: how it’s changing computing and how it can best be utilized.

Loris Degioanni (aka my boss) gave a great presentation on the new platform API that some people have been working on here at Riverbed. I think this is going to be a must-have once people realize the cool and crazy ways that this can be leveraged.

Wireless was well-represented Thomas D’Otreppe taked about OpenWIPS-ng, his new wireless IPS/IDS project. Metageek’s Ryan Woodings talked about wireless below the packet level. Mike Kershaw and Michael Ossman talked about the development of the Kisbee and Ubertooth adapters, and how you too can design and implement real hardware devices.

Hansang Bae showed modern TCP stacks and network devices can collude to make connectivity really slow, and at a really low level.

Laura Chappell showed off a lot of the new features in Wireshark 1.8. I even learned something — you can type a color name (e.g. “bisque3″) into the color picker in the preferences. She also released the second edition of Wireshark Network Analysis. It’s even heavier than the first edition, which I learned first-hand.

I led one session this year, Ask The Experts. We ended up talking about ways to improve Wireshark and a few of the core developers and I discussed these ideas the next day.

We got logs of great feedback on the conference as well. Several people commented that we didn’t have as many devlopment-oriented sessions this year as we have in the past. Maybe we can add more next time.

Wireshark is a great and wondrous thing due to its community. The developers, users, educators, sponsors all play a big role in its development and evolution. I see small reminders of this every day but it becomes abundantly clear each year at Sharkfest.

Thanks to the attendees, presenters, and to the Riverbed events team for creating a wonderful experience. I can’t wait to see what next year’s Sharkfest has in store.

Wireshark and Pcap-ng

When Wireshark 1.8.0 is released in the next few months it will introduce two major features: the ability to capture from multiple interfaces at once and the ability to annotate packets. These features have been on the wishlist for years and they will make Wireshark much more useful. They both share dependency on pcap-ng.

Packet comment example

Packet comment example

Pcap-ng is Wireshark’s new default file format. It is more complex than its predecessor (pcap) but it’s also more flexible. Along with multiple interface types and annotations you can store host information, extended interface information, and much more. Wireshark has had basic support for pcap-ng for several years now and 1.8.0 will bring it to the forefront.

But there’s a catch. Suppose you sit down at a computer with Wireshark 1.7 or 1.8 installed. What file format will Wireshark use by default? Unfortunately the answer is “It depends.” If this is a fresh install of Wireshark on a new computer you’ll get pcap-ng files. If this is an upgrade from 1.6 or below you’ll get pcap files. If the preferences were changed or copied from another machine the default format will be affected as well.

There’s another catch. Suppose you’ve just saved a capture file and you want to read it using tcpdump, Cascade Pilot, Snort, or any of the dozens of other applications that use the classic pcap file format. What will happen? Again, it depends. If that application has been updated to support pcap-ng or if it’s using a recent version of libpcap it will probably work. Otherwise don’t hold your breath.

If you’re a Wireshark user you might want to try a recent build of Wireshark 1.7 to see the new features and new file format in action. If you’re a developer and your application reads pcap files you’re most likely in for some grief. For that I apologize. You should probably investigate adding pcap-ng support now before your users start complaining.

Despite the migration pains the switch to pcap-ng is long overdue and will make Wireshark more useful to more people.

Used Cars and Stub Installers

The Wireshark development team works hard to earn the respect of our users. This includes making sure that downloading and installing Wireshark is as easy and trouble-free as possible. Right now the vast majority of our users can go to www.wireshark.org, follow the big green arrows, and immediately download the appropriate Wireshark package for their platform.

For many years a number of third party sites have also offered Wireshark downloads. Typing “wireshark download” into your favorite search engine will turn up a bunch of them, usually just below links to wireshark.org. These sites are popular and often provide valuable services such as reviews and malware prescreening. They also reside outside the Wireshark ecosystem — we don’t link to them and aren’t affiliated with any of them.

"This is the Cadillac of invasive toolbars at a Chevy price!"

Sometimes these sites abuse their relationship with their users. For example a few months ago Download.com started using a stub installer which tries to get you to install various toolbars and who-knows-what-else before it installs the package you ultimately want, much like a sleazy car salesman trying to bundle add-ons you don’t want or need.

This sort of bottom-feeding behavior is harmful to our user community and exploits the goodwill we have with our users. Brian Krebs and Gordon “Fyodor” Lyon describe the problem with much more depth and eloquence than I can.

I sent a request to Download.com to disable their stub installer for Wireshark. They complied, but there are dozens of other download sites. Trying to keep tabs on all of them would result in a never-ending game of Whac-A-Mole®.

On behalf of the Wireshark development team I promise to provide easily accessible, direct downloads of Wireshark from wireshark.org just as we always have. If you choose to download Wireshark somewhere else we can’t guarantee that the experience will be free of shenanigans so please be careful.

Sharkfest ’11 Recap

The fourth annual Sharkfest was held last week. If you missed it, don’t worry. We are busy uploading all of the presentations to sharkfest.wireshark.org.

The conference started with a keynote by Dr. Steve McCanne, CTO of Riverbed. He described the history of BPF and how its optimizing compiler works. It was interesting to see all of the problems he ran into and the solutions he found. The presentation was easy to follow despite its technical level. (Steve created BPF, and co-wrote tcpdump and libpcap.)

The keynote set the tone for the rest of the conference, which featured a lot of talented speakers. Jeff Carrell’s IPv6 was so popular it had be repeated several times over the course of the conference. Lara Chappell, Hansang Bae, and Betty Dubois drew their usual huge crowds. The other presenters did great jobs as well.

People have great ideas for improving Wireshark. Plenty of these ideas and suggestions were heaped our way throughout the conference, especially during the last two sessions. I hope to spend some time this summer implementing some of them.

Wireshark is very much a community-owned project. It is where it is today because of the developer and user communities. Steve stressed this fact during his keynote and it was readily apparent throughout the conference. On behalf of the developer team, the Sharkfest organizers, and Riverbed I extend my heartfelt thanks to the Wireshark community. This continues to be an amazing adventure and I can’t wait to see what the future has in store.