Greenhill. Saturday night, 11:30 PM. I’m in bed, at long last, the sleep tank running empty. It’s been a busy week, after a series of database upgrades and the usual start-of-season challenges: users new to Tabroom, experienced users who forgot things since they last tabbed in March, new features not quite working, whack a mole bugs from changes over the summer. At that moment, BMan texts me to tell me the coaches of a Certain Program Somewhere out west are complaining that only some of them are getting the text message blasts for their debaters’ rounds.
Text messaging isn’t as reliable as you may think. The failure rate for SMS messages sent individually is somewhere around .5%. For a bulk blast such as Tabroom sends, it can hover somewhere around 1-2%. If that doesn’t sound like a lot, consider: when you blast out a pairing for a Policy round at Greenhill, you are sending a text to 432 people. If the failure rate is even .5%, 2-3 people aren’t getting that message. Blast out six prelims in LD and Policy, and a bunch of elims, and you have about 40-50 instances of someone not getting their texts. Sometimes it’s Mom stalking you from home who misses a text. Sometimes it’s you.
On President’s Day weekend, Tabroom sent out 114,988 text messages in three days. .5% failures mean 576 people didn’t get one text blast; up it to 2% and that’s 2,301 lost messages.
Furthermore, Tabroom and JOT don’t actually send out texts. Sending a text message directly is not free — the carriers would charge us to do it. Since we’re a non profit speech and debate association, and not a global megabank, we use a side door: many carriers offer a little-publicized service where you can text message their users via an email, free. If your phone number is 978-555-1234, and you’re on Verizon, emails to firstname.lastname@example.org come to you as a text. The address you email to is different for every carrier, which is why we ask for your carrier when you sign up. That’s also why some smaller carriers are missing from that list — they don’t offer this service.
Anything you can email is also a thing you can spam. The carriers are pretty vigilant about monitoring these email gateways for spam attempts — you’d be really pissed if you started getting Viagra ads on your texts, after all. A service such as ours which sends out 114,984 texts in the course of a weekend, while sending virtually nothing during the work week, will understandably look like a spammer to anti-spam software. So sometimes it will happen that all T-Mobile customers suddenly stop getting Tabroom texts. I can sometimes negotiate around that, and notify the carriers that we are actually good citizens here, but not always quickly, especially on a weekend.
All of this is to say your SMS message blasts are handy, but you should still check the pairings online when they come out. It also means if you don’t get a particular blast, it’s not a problem I can do much about, and am inclined to brush it off a bit. Report it when I’m finally in bed after a long two days, exhausted and stressed, well… you can go ahead and assume that is not a time of any week I’m eager and ready to bust out the computer and sling some code to save you from having to check an online pairing because you aren’t getting texts. It being BMan — a good friend whom I respect and value, and who has done me multiple kind favors and helped me out on a number of occasions — I of course replied with a rude graphic image and ample profanity, rolled over, and went to sleep.
The next morning, however, I woke feeling better. I miraculously got about 7 hours sleep, which is a debate tournament equivalent of three full nights’ worth. I double fisted Starbucks chai lattes. And the wiki was running better, so things were relatively calm. I had some time on my hands. At that point, the lovely Mr Vincent comes in and tells me that he, too, hasn’t gotten a single blast for his student’s rounds, but has been getting all of his for judging.
That’s a great bug report, by the way. Whining that “it doesn’t work” gives me very few clues what could be wrong, and that in turn gives my surly nature an excuse to dismiss the bug and blame you. My threshold on that is pretty low. Just ask Menick. But CV’s fact pattern was more interesting — he was only getting SOME messages, and the ones he got followed a consistent pattern. Those details made me immediately realize there’s more to the story here, and take him seriously.
I pull up his follower record, and there it is. His email was following the debater, and it looks OK. His phone number was listed in the same record: (214) 748-3647.
“But,” he says, “that’s not my phone number.”
I pull up all the entries he’s following. The number listed: (214) 748-3647.
I look at his login, which is where his own judge blasts are going, and that number was correct. Huh.
I check where area code 214 is, and it’s Dallas. I’m at Greenhill, so I roll my eyes. “Is some kid at the tournament trying to prank me and messing with the system somehow?” I tell the database to pull up every record of this phone number following entries or judges in Tabroom.
The count was 7,632.
Evil Hackers are poorly understood by the general public; they are not diabolical Armani wearing supermen using specialized equipment to cast secret magic across the network and bend systems by the power of their will. If someone ever says “I’ve hacked into their security cameras and am tracking their movements now” after five minutes of furious typing, the correct response is to nod warily but politely, close the padded door and ask the nurse to up their medication.
Hacking in the real world is a matter of finding some flaw or mistake in the design of a system. Systems are very complicated things, and here’s a lot of surface area for mistakes.
Servers are just computers that run software to respond to remote requests, and all software has flaws. Hackers look specifically for software flaws that react to having specific data inputs by giving system access to the sender of that data. Such data thus acts as a key to that type of server, bypassing its ordinary restrictions and limits. These flaws are the so-called backdoors, and they can lie undiscovered and unknown on systems for years, sometimes decades, before being spotted and exploited. Some backdoors are hypothetical, and are even patched before anyone figures out how to use them.
A zero-day is a kind of backdoor that can be used immediately, for which no software patch yet exists. Someone out there finds a backdoor, and writes software called a rootkit which sends the particular data to systems running the software containing the vulnerable code. Sometimes these root kits are written just to prove that a backdoor is serious, and push the software maintainers to quickly develop a fix. But sometimes they are released on the internet for anyone to use, and those become dangerous; writing a root kit is the hard part, an activity reserved for a handful of true masters. Once they are released publicly, there are thousands of people who know relatively little about computing who can nonetheless successfully use them to hack into systems.
Users of systems affected by a zero-day can do nothing about it except yank their systems off the network, or hope and pray they’re not targeted. Zero-days are given scary names, like Sandworm, Shellshock, and Heartbleed, and written about like they spell doom for civilization and humanity, which policy debaters seeking impact files find very helpful. They’re big news. The firms affected by zero-days do suffer major damage to their reputations, and therefore lose sales, so they have a strong incentive to quickly fix any such flaws. Once such fixes, or patches, are released, however, they need to be installed.
The process of patching a main server is not much more complicated than applying Windows or Mac software updates, once the patches are available. You download them, apply them, and sometimes reboot the server. Institutions with a lot to lose, like banks, credit processors, or the government are very diligent about fixing vulnerabilities of this sort. Those types of systems are therefore usually hacked by going after the people in those organizations instead of their computers — the big government data revelations of recent years have come from leakers or careless employees, not hackers.
Instead, successful backdoor attacks recently have struck one step down the food chain: major retailers like Target and Home Depot. Such companies are still valuable hacking targets, as they possess millions of customer records and credit card numbers. But computing isn’t their core business, and they’re not necessarily run by people who realize how dangerous insecure systems can be. Sooner or later, some bean counter up the corporate food chain decides they don’t have much risk after all, and why are we spending so much money on computer security? “I have anti-virus on my computer and that works fine. Who cares if those nerds in the basement insist we need this stuff?” Target’s CEO was fired after their hack, which suggests that their lax approach to security came from the top.
Tabroom’s code runs on top of a large number of other pieces of software: an operating system, a database, a web server, programming language interpreters and libraries that handle standard things like formatting and timezone conversions, and various ancillary bits of software like an email server and remote access server. A backdoor in any of those bits of software could open up my machine to a remote attack and a rootkit. So I have to keep up with patches on them, and naturally, I spend a hell of a lot less time and money on security than Target or Home Depot did. So why was I skeptical that we got hacked?
The part of that software stack most likely — by far — to have some remote vulnerability is the Tabroom.com code itself. All that other software we use is made by large teams of coders, many having members entirely dedicated to security. We, shall we say, don’t have that. But our low resources reflect our low value — Tabroom’s specific vulnerabilities are incredibly small potatoes. A root kit built against Tabroom specific code would get you access to exactly two machines in the entire world, and one is kind of old and slow. We’re big in debate, but tiny overall.
It’s a lot harder to find an exploit in the web server we use, but if you succeed, you’d have the key to millions of systems, including ours. So the web server gets the real hammering, and my code gets none. The type of hacking a little niche site like ours has to worry about is teenagers with more time than sense, armed with root kits they know how to use but don’t truly understand. They set their kits to blast the internet at random, looking for unpatched systems vulnerable to their attacks. Such hackers are called script kiddies. Their motive, apart from the thrill of the hunt, is to use my machine to host computer game servers, or share porn and illegal software with their friends at junior high. They might put my machine into a network of other hacked servers and use it as part of an denial of service attack— where you flood a website with more traffic than it can handle, aiming to knock a site offline. Ultimately they’re unlikely to care about Tabroom itself, and are only interested in it as a reasonably powerful computer connected to the internet that isn’t traceable to them, so they can do other illicit things without someone telling their parents.
We suffer constant generic attempts by script kiddies, aimed at the underlying software on the system. It doesn’t take a lot of effort to stay up on patches to stay ahead of them, mostly because so many other small site admins are lazy, and don’t. I’m the neighbor with the alarm system — a burglar can still rob me and probably get away, but she may as well rob the house next door that lacks one. Unless we really piss the Internets off, something I try not to do, we’re relatively safe from folks trying to break my own brittle Tabroom specific code, as they’d gain so little from it. If you’re going to hack one specific website, you have to really mean it.
And that brings us to the Ashley Madison hack. That site was in a bad spot: their security resources were probably rather limited since they were not very big, but the cost of their data being exposed was clearly very high. Risk times magnitude weighs the terminal impact. And most people, their attackers included, found the Ashley Madison service appalling, which further increases the incentive for an attack. Folks whose morals would not permit them to hack a bank had no compunction about targeting Ashley Madison.
The attackers went out for blood, and they sure got it. They focused specifically on Ashley Madison, stole all their data, and made it public. Their aim was nothing less than bringing the whole service and its parent company down, and it appears they have succeeded. Moral of the story: if you run a odious, sexist website that caters to powerful assholes, while keeping deeply damaging personal data about said assholes in a database, try to stay up on your security patches.
What they did not do, after all that time and effort spent finding flaws in the Ashley Madison code, is sit around changing people’s text message alert phone numbers, whose terminal impact is annoying me on a Sunday morning. Therefore, I didn’t have a hard time dismissing the presence of a hacker after a cursory check for evidence thereof.
If not an evil attacker then, where the hell did (214) 748-3647 come from?
I investigated further. I set myself up to follow one of my own debaters, and put in my number, which is from the 978 area code. And then I look it up in the database:
I try some other numbers. Same result, every time. Now, I’m really alarmed. I wonder in horror if I’ve written some bug that substitutes a random Tabroom user’s phone number in for anyone else who follows an entry or judge. I say a silent prayer for forgiveness from whatever poor sod out there was so targeted, and whose prize was having their phone go through a sudden seizure every time Yale, Greenhill or Georgia State released a round.
So I check the logins list — where you put your number in once and it permanently follows your own judge records. And it turns out no active Tabroom login has that number. So if this a person in the debate community, they’re not a very active debater or coach.
I read over the following-code very very carefully, and check the value of the variables therein at every stage of its operation. Nope, it was fine.
Finally, I bypass my code altogether, and run a direct database command to change the follower record I have created to my 978 phone number. This method accesses the database directly, bypassing all Tabroom code, and therefore any bugs I might have created.
And I get (214) 748-3647.
I then send a runner to find Father Hahn and tell him to bring a gallon of holy water to tab. Kid thought I was joking, because the good Father never came, but I’m not sure I was.
There are moments in computing where you sink into despair. You write a function to do one thing, and it acts in a way so contrary to expectations, intuition and experience that you start to lose hope and faith in the world as a predictable place. You say to yourself: “Is this the day, self, that the universe has decided to dedicate a small part of its attention solely towards driving me crazy? Have I finally snapped? If this code functions that way, then is the speed of light still a constant?” Also, I’m usually flying solo at tournaments, without someone to provide a sanity check. There are approximately ten people in the entire country who fluently understand both tabulation and tech. I cannot begin to explain some problems to everyone else, because we don’t share enough technical language and basic knowledge to even have a conversation. It’d be like your mom offering to help you cut a counterplan. The hours spent explaining the premise wouldn’t be worth the twenty minutes of help.
So when I reach the end of logic, and find myself walking above an empty canyon in bafflement, the way Wile E Coyote walked on thin air after running out of road, falling only when he looked down — during those moments I can only sit there and stare at the screen, wonder what I’m missing, and hope I don’t start weeping. It’s important for computer people to keep up an all-knowing appearance — folks are afraid enough of tech without realizing that I’m sitting there, with all the tournament data in my hands, in a blind panic and ready to run for the nearest airport. That’s why Sarah Donnelly gets instantly concerned when she observes that I’m wearing my “oh-no face.”
I decide instead to take a mental break, which can often help in those situations. I start to randomly wander around the internet. I wonder if I can look up who has this phone number. Sometimes you can Google a phone number and find out who it belongs to, if they’ve posted it on the web somewhere. Instead, I get…a Wikipedia page for Mersenne primes?
Mersenne primes are prime numbers one less than a power of two. 3 is a Mersenne prime, as is 127. Powers of two are important in computing, since all computer storage is ultimately binary data, consisting of solely zeros and ones. That’s because data is represented with small magnetic charges, which are either present (one) or absent (zero) in a given region of a disk surface, giving you 2 digits to work with. Each such tiny field of data is called a bit, which is the smallest unit of computer data. So computers count by 2s, not by tens like we do. Base-2 counting means every additional digit, or bit, represents an additional power of two. 1 in binary is 1 in decimal counting too. But 10 is 2. 100 is 4. 1000 is 8. 1 1111 means you add the powers of 2 shown together: 1+2+4+8+16= 31, which is one less than the next digit alone, 10 0000 = 32. 31 is a Mersenne prime, since it satisfies both conditions: it’s prime, and it’s one less than a power of two.
The amount of data a computer can store is always one below a power of two. A computer with just eight bits of memory (a unit known as a byte) could express the number 255, which is 1111 1111. But it can’t express 256, which requires nine bits: 1 0000 0000. Folks in computing therefore tend to be very familiar with powers of two. If someone you know refers to 512 as a “nice round number”, they’re a computer geek. A geeky t-shirt I may or may not own proclaims “There are 10 types of people in the world, those who understand binary and those who don’t.”
A few years ago hard drive vendors pulled a fast one on its customers and redefined data storage units. Normal people generally assumed a gigabyte equaled 1,000 megabytes, since we think in terms of tens, but in fact it was equal to 1,024 megabytes (minus one bit), which is a power of 2. But the vendors took advantage of that assumption, and attempted to redefine a gigabyte to 1000 MB, and then in turn a terabyte to 1000 GB, basically shorting you of space. Pop that “1 TB” drive into a computer, and the computer will report that it’s only about 980 GB in size, because it defines these units properly in binary terms, not decimal ones.
If you have a certain number of bits, you can express numbers up to one below 2 to the power of that number of bits. Those limits are therefore often Mersenne primes. 32 bits of memory, to draw a non-random example, can represent a number that’s (2^31 – 1) in size.
Which would be 2147483647.
When you design a database you must pre-declare how large the data in any given column is going to be limited to. The reason is that the database will allocate the space for data in set, predetermined chunks. If you say a given data field should be big enough to hold 255 characters, and then put only 2 characters into it, that field will still take up 255 characters worth of storage. So if you’re storing, say, state abbreviations, good practice is to limit the field to 2 characters only, as correct values will never be larger. With stuff like proper names or so on, you make a decent guess, and leave some padding. If anyone in debate has a first or last name longer than 63 characters, be prepared for disappointment when you register on Tabroom.
Before last week’s data conversion, I stored phone numbers as strings, which can be letters, numbers or punctuation, instead of just raw numbers. So if you typed in (978) 555-1234, that’s what I’d store, spaces, parens and all. But the text-email gateways only want numbers, nothing else. So every time Tabroom sent a blast out, it would have to first remove all those non-number characters. That costs a little extra processing time, but more importantly, it’s bug prone: I had to do it every time I accessed phone numbers, so if I had to change how I do that, I had to change it every place I do so. And inevitably I’d forget at least one and create a bug. Also, storing characters takes up more space than storing raw numbers only, and keeping all that extra punctuation is wasteful since I was just going to remove it anyway. So, in the update, I changed how that works, and now strip out the non-numeric characters before storing your number, keeping only the numbers themselves.
Database designers tend to pay close attention to string sizes, but when storing integer numbers, it’s easy to get a bit lazy and just declare the default sized integer, an INT: 32 bits. That’s a default for historical reasons. Most computer processors made from around 1995 until 2005 had 32 bit instruction sizes, which means the largest numbers they could process in a single chunk within the chip itself was 32 bits in size. A 32 bit computer can deal with larger numbers, but it has to pull tricks and any such operations are slower and less precise than dealing with ones that fit in its pipeline. This limit affected how quickly and precisely 32 bit CPUs could do math involving large numbers, how much RAM memory their systems could manage, and even how long they could maintain their internal clocks: 32 bit processors will stop keeping correct time on some operating systems in 2032. Nowadays standard processors have 64 bit instruction pipelines, but 32 bit limits remain common throughout computing, because folks got used to that being a reasonable default and it’s stayed that way since.
Save a number into a database field that’s beyond its ability to express, and the database will save instead the largest number it can, the Mersenne prime 2147483647.
And thus, my bug.
Why I didn’t see it until now is a lesson in psychology. In Tabroom, by default you must enter each ballot twice before it is accepted. I discourage people from using visual scanning as an audit method, because when you see or hear a number and have to confirm that a second number matches, sometimes your brain sees what it expects to see, rather than what it actually sees. It’s better to have to read and interpret the ballot twice, preferably by two different people, so no expectation biases exist. That same human flaw stung me — I was looking at it expecting to see a phone number, and it was indeed a valid area code followed by seven digits, so I saw a phone number, and didn’t recognize it for what it was. As much as that seems like a really random number, it’s fairly well known in computing because 32 bit size limits are very common. It’s known as max int. I could have recognized it, but I was in the wrong mental context.
It so happens nobody has max int as a phone number; we called it and got a busy signal. I did not spend the weekend spamming some poor schmuck in Dallas with thousands of demands that they be affirmative against Lexington SM in US 302 in front of DHeidt while also judging an LD round in WLH in New Haven and watching a Northwestern team try to run the K in Atlanta — which actually happened, so at least my experience wasn’t the weirdest of the weekend. My messages, and your text blasts, just disappeared into the ether, rejected by bewildered cell carriers which did not have that number and would you please stop asking us to text it, Tabroom.com?
When the Gangnam Style video became the most popular Youtube video ever, it was the first to achieve more than 2,147,483,647 views, and for a while the counter stuck there. It was rumored that Google fell victim to this same error. That story turned out later to be untrue, a Googlish joke.
After an hour and change of searching, despair, crazed laughter, learning new things, beating my head against my screen, and both sass from and mocking of BMan, it took six words to fix:
ALTER TABLE follower MODIFY cell BIGINT;
A BIGINT is 64 bits. It can store a number up to 18446744073709551615. Unless the phone network grows by a lot, we should be good now.
Then I went and got another chai.
I hate computers.