Earlier this week, HughesNet scheduled a maintenance outage to do whatever it is that they need to do for maintenance (which, in the past, has included replacing equipment damaged by golf-ball sized hail at the ground stations). When the connection came back up early the next morning, it was plagued by mysterious and intermittent RSTs on HTTP connections and 2% packet loss. Owch! Having approximately 320268309285049386509258 errands to run, I didn’t get to examine the connection until last night…

Now, my initial reaction should have been to power cycle the modem. This tends to solve a surprising number of issues, at least temporarily. But I wanted to figure out what was going on first, and pride got in the way besides–after all, I’m a Linux geek, right? And Linux geeks don’t reboot things, ever. Except for kernel updates. And power outages. And spontaneous fits of rage. Rebooting things to solve a problem is a Microsoft thing, yargh! (And it would’ve solved the problem right away too, but then you wouldn’t be reading this post, so…foo on the easy way out!)

I spent a half-hour or experimenting before I glanced at my Cacti page and saw this:

The gap between Thursday and Friday was caused by the outage. Immediately after the connection was restored, bandwidth spikes, and then…hey, what’s that constant baseline of about 60Kbit/s? (Trust me, I zoomed in.)

Trusty iftop revealed that, yes, I was indeed getting about 55-60Kbit/s on WAN:

 

Aside from the traffic to 192.168.0.1 (the HT1000 modem), which is probably my monitoring script, all of the other endpoints were unexpected. The WAN interface was receiving ~55Kbit/s of traffic from the Internet. None of the traffic had been solicited by a machine inside the LAN, so the traffic hit my external firewall and was DROPped. For most people, this is no more than a nuisance–out of sight, out of mind, right?

But satellite users are a different breed. Oh, yes. No 150GB all-you-can-eat cornucopia for us. Bandwidth is gold. Every byte that passes through the modem is counted against the precious 10GB/month quota. (All right, it’s 20GB/month if you manage to use the extra 10GB provided between the hours of 2AM-8AM, which is good for early risers or people who like to watch Burn Notice on Hulu to fight off early-morning insomnia.)

55Kbit/s (which, if you look closely at the graph, is near constant except for a brief dip around Saturday morning) is 580MB/day, or 16.99GB/month. Obviously, this is a problem if your ISP only provides you with 10GB/month.

So I dd a little more sniffing on my router’s network interfaces and established that there was zero traffic on any of the LAN ports aside from the SSH connection I was using to log in to the router. I ran a capture on the WAN interface, got up and had some tea, enjoyed a little music, and came back twenty minutes later to examine the data with Wireshark.

 

Ah.

BitTorrent.

More specifically, other peoples’ Bittorrent traffic.

See, I’m funny about certain things. I don’t torrent, in part because for a decade I had dialup and didn’t have the bandwidth to do it; in part because I don’t want to get grumpy letters from a grumpy ISP who got a grumpy legal document from a grumpy RIAA worker who has no other purpose in life but to make other people miserable; and in part because I find it morally undesirable for non-free copyrighted works. (Of course, BitTorrent is a great way to quickly fetch large Linux ISO files, and it eases the load on download servers too.)

I can guarantee that no one else in this household has even heard of BitTorrent, let alone installed a BitTorrent client. (I’ve checked.) I can also guarantee that there are no foreign clients on my network. (I’ve checked.) And of course, I can guarantee that none of this BitTorrent traffic is related to me because I haven’t torrented anything since CentOS 6 (about two years ago, before I had a satellite connection) and I don’t even have a BitTorrent client on my computer.

Now, assuming that all this mystery traffic is BitTorrent traffic based on a few BitTorrent handshakes is a stretch. However, this past March, I noticed similar “mystery” traffic, although it was only a trickle: a few BitTorrent DHT packets per minute. Of course, this time, the rate was much higher, and there were many more other hosts involved: while analyzing last night’s capture, Wireshark counted over 2000 IP endpoints from a 20-minute period.  I’d say BitTorrent is a decent guess.

The question is: why? My theory is that my modem was given a new IP address when the outage was over and the outage must have been short enough for HughesNet’s CGN system to not drop open the BitTorrent connections that had been destined for whomever had this IP address immediately prior to me.

It makes me wonder how many Internet users are under scrutiny for the actions of the user who was previously using their dynamic IP address.

In the end, I rebooted the modem. (Again, I should’ve done this earlier, but if I hadn’t, then you wouldn’t be reading this article.) I did the elegant thing by yanking the power cord out of the wall, waiting five minutes, and plugging it back in. When the modem reinitialized itself, it had a new IP address. Lo and behold, my idle traffic was back down to about 0Kbit/s–where it should be. My packet loss went back to 0%, where it should be, and I have yet to see a Connection Reset message from Firefox. Now, 50KBit/s doesn’t explain the RSTs or the packet loss. My guess here is that whatever IP Gateway (CGN box) with which the modem associated is overloaded by BitTorrent traffic. But that’s just a guess.

Now, anything connected to the Internet is going to see some unsolicited traffic. It’s inevitable. Port scanners are a constant nuisance, for instance. Even on dialup, my router often recorded port scanning attempts. (Once, when I had made the mistake of running SSH on port 22, I found that a bot had spent a considerable amount of time trying to log in to my router with common username/password combinations. On dialup.) However, on metered connections, this traffic is still counted.

Maybe this is a cause of some of the “phantom download” posts on the HughesNet community support forums?

EDIT:

Yes, the download was counted; HughesNet reports 6-13MB/hr last night (when nothing was running) while my baseline is usually around 80KB/hr. Owch!

I captured a few handshakes with a few BitTorrent “hashes” (?) inside. For your amusement, here’s what people were apparently downloading with this IP address before the outage occurred and the IP was leased to me:

• The Eagle’s Greatest Hits
• Sons of Anarchy, the Complete Season 5, HD
• The Ultimate Best of Pearl Jam
• The Walking Dead Season 2
• The Walking Dead Season 3 Episode 4
• John Mayer – Paradise Valley
• Philip Phillips – The World from the Side of the Moon
• Young Jeezy – TM103 Hustler’s Ambition