@mildconcern@tech.lgbt
Greenhill. Saturday night, 11:30 PM. I’m in bed, at long last, the sleep tank running empty. It’s been a busy week, after a series of database upgrades and the usual start-of-season challenges: users new to Tabroom, experienced users who forgot things since they last tabbed in March, new features not quite working, whack a mole bugs from changes over the summer. At that moment, BMan texts me to tell me the coaches of a Certain Program Somewhere out west are complaining that only some of them are getting the text message blasts for their debaters’ rounds.
Text messaging isn’t as reliable as you may think. The failure rate for SMS messages sent individually is somewhere around .5%. For a bulk blast such as Tabroom sends, it can hover somewhere around 1-2%. If that doesn’t sound like a lot, consider: when you blast out a pairing for a Policy round at Greenhill, you are sending a text to 432 people. If the failure rate is even .5%, 2-3 people aren’t getting that message. Blast out six prelims in LD and Policy, and a bunch of elims, and you have about 40-50 instances of someone not getting their texts. Sometimes it’s Mom stalking you from home who misses a text. Sometimes it’s you.
On President’s Day weekend, Tabroom sent out 114,988 text messages in three days. .5% failures mean 576 people didn’t get one text blast; up it to 2% and that’s 2,301 lost messages.
Furthermore, Tabroom and JOT don’t actually send out texts. Sending a text message directly is not free — the carriers would charge us to do it. Since we’re a non profit speech and debate association, and not a global megabank, we use a side door: many carriers offer a little-publicized service where you can text message their users via an email, free. If your phone number is 978-555-1234, and you’re on Verizon, emails to 9785551234@vtext.com come to you as a text. The address you email to is different for every carrier, which is why we ask for your carrier when you sign up. That’s also why some smaller carriers are missing from that list — they don’t offer this service.
Anything you can email is also a thing you can spam. The carriers are pretty vigilant about monitoring these email gateways for spam attempts — you’d be really pissed if you started getting Viagra ads on your texts, after all. A service such as ours which sends out 114,984 texts in the course of a weekend, while sending virtually nothing during the work week, will understandably look like a spammer to anti-spam software. So sometimes it will happen that all T-Mobile customers suddenly stop getting Tabroom texts. I can sometimes negotiate around that, and notify the carriers that we are actually good citizens here, but not always quickly, especially on a weekend.
All of this is to say your SMS message blasts are handy, but you should still check the pairings online when they come out. It also means if you don’t get a particular blast, it’s not a problem I can do much about, and am inclined to brush it off a bit. Report it when I’m finally in bed after a long two days, exhausted and stressed, well… you can go ahead and assume that is not a time of any week I’m eager and ready to bust out the computer and sling some code to save you from having to check an online pairing because you aren’t getting texts. It being BMan — a good friend whom I respect and value, and who has done me multiple kind favors and helped me out on a number of occasions — I of course replied with a rude gesture and ample profanity, rolled over, and went to sleep.
The next morning, however, I woke feeling better. I miraculously got about 7 hours sleep, which is a debate tournament equivalent of three full nights’ worth. I double fisted Starbucks chai lattes. And the wiki was running better, so things were relatively calm. I had some time on my hands. At that point, the lovely Mr Vincent comes in and tells me that he, too, hasn’t gotten a single blast for his student’s rounds, but has been getting all of his for judging.
That’s a great bug report, by the way. Whining that “it doesn’t work” gives me very few clues what could be wrong, and that in turn gives my surly nature an excuse to dismiss the bug and blame you. My threshold on that is pretty low. Just ask Menick. But CV’s fact pattern was more interesting — he was only getting SOME messages, and the ones he got followed a consistent pattern. Those details made me immediately realize there’s more to the story here, and take him seriously.
I pull up his follower record, and there it is. His email was following the debater, and it looks OK. His phone number was listed in the same record: (214) 748-3647.
“But,” he says, “that’s not my phone number.”
I pull up all the entries he’s following. The number listed: (214) 748-3647.
I look at his login, which is where his own judge blasts are going, and that number was correct. Huh.
I check where area code 214 is, and it’s Dallas. I’m at Greenhill, so I roll my eyes. “Is some kid at the tournament trying to prank me and messing with the system somehow?” I tell the database to pull up every record of this phone number following entries or judges in Tabroom.
The count was 7,632.
Evil Hackers are poorly understood by the general public; they are not diabolical Armani wearing supermen using specialized equipment to cast secret magic across the network and bend systems by the power of their will. If someone ever says “I’ve hacked into their security cameras and am tracking their movements now” after five minutes of furious typing, the correct response is to nod warily but politely, close the padded door and ask the nurse to up their medication.
Servers are just computers that run software to respond to remote requests, and all software has flaws. Hacking in the real world is a matter of finding such a flaw and exploiting it. Systems are very complicated things, and here’s a lot of surface area for mistakes. Most are harmless, but some are the dreaded backdoors. A backdoor is a flaw that, when sent specifically formatted data input, will react by giving the sender system access. Backdoors can lie undiscovered and unknown on systems for years, sometimes decades, before being spotted and exploited.
Finding these flaws and the data that triggers them is hard work for seriously talented and knowledgeable people. However, hacking is not limited to that elite group. Every so often, someone out there finds a backdoor, and writes a rootkit which sends the particular data to systems running the software containing the vulnerable code. Sometimes these root kits are written just to prove that a backdoor is serious, and push the software maintainers to quickly develop a fix. But sometimes they are released on the internet for anyone to use, and those become dangerous. While only a few masters out there can create an effective root kit, many thousands know enough to use them once they exist. If a root kit is released, and no effective defense or patch to close vulnerable systems yet exists, the flaw becomes known as a zero-day exploit.
Users of systems affected by a zero-day can do nothing about it except yank their systems off the network, or hope and pray they’re not targeted. Zero-days are given scary names, like Sandworm, Shellshock, and Heartbleed, and written about like they spell doom for civilization and humanity, which policy debaters seeking impact files find very helpful. They’re big news. They can be used to steal private information, or manipulate financial or government systems. Big and important firms therefore have a strong incentive to quickly fix any such flaws the minute a fix, or patch, is released.
The process of applying patches to a server or its software is not much more complicated in principle than applying Windows or Mac software updates. Institutions with a lot to lose, like banks, credit processors, or sensitive government departments are therefore usually diligent about patches. So backdoors aren’t often the weapon of choice when attacking a bank or credit card company. It’s far easier to wheedle a password out of a non-security conscious contract employee by claiming to be the IT department on the phone than to penetrate their network security. Leakers or careless employees, not backdoors, are the weak point.
Root kits find better targets one step down the food chain: major retailers like Target and Home Depot. Such companies possess millions of customer records and credit card numbers. But computing isn’t their core business, and their leadership tends to be less aware of security issues. Sooner or later, some bean counter up the corporate food chain asks the dread question: “why are we spending so much money on computer security? I have anti-virus on my computer and that works fine. Who cares if those nerds in the basement insist we need this stuff?” And IT isn’t exactly filled with people who can argue their case to humans effectively. So a budget gets cut, security goes lax, and backdoors are left unguarded longer.
Tabroom is not a single program. It relies on a stack: an operating system, a database, a web server, code interpreters, and libraries that handle standard things like formats, printing, and timezone conversions. There’s even various ancillary bits of software like an email server and ways I can access the system. A backdoor in any part of that stack could open up my machine to a rootkit. I have to patch it all, and naturally, I spend rather less time and money on security than Target or Home Depot did.
So why was I skeptical that we got hacked? Well, bluntly put, nobody cares.
All that software in the stack is made by large teams of coders, many having members entirely dedicated to security. The part most likely — by far — to have some exploitable vulnerability is the Tabroom.com code itself. We don’t exactly have a dedicated team of security experts reviewing our code for exploits. But our low resources reflect our low value — a backdoor in Tabroom would be incredibly small potatoes. Finding one gets you access to exactly two systems in the entire world, and one is kind of old and slow. We’re big in debate, but tiny overall. It’s a lot harder to find an exploit in the web server we use, but if you succeed in that, you’d have the key to millions of systems, including ours. So the web server gets the real hammering, and my code gets none.
The type of hacking a little niche site like ours has to worry about is teenagers with more time than sense, armed with root kits they know how to use but don’t truly understand. They set their kits to blast the internet at random, looking for unpatched systems vulnerable to their attacks. Such hackers are called script kiddies. Their motive, apart from the thrill of the hunt, is to use my machine to host computer game servers, or share porn and illegal software with their friends at junior high. They might put my machine into a network of other hacked servers and use it as part of an denial of service attack— where you flood a website with more traffic than it can handle, aiming to knock a site offline. Ultimately they’re unlikely to care about Tabroom itself, and are only interested in it as a reasonably powerful computer connected to the internet that isn’t traceable to them, so they can do other illicit things without someone telling their parents.
We suffer constant generic attempts by script kiddies, aimed at the stack. Patching that is easy, and done regularly. It also helps that many other small site admins are lazy, and don’t. I’m the neighbor with the alarm system — a burglar can still rob me and probably get away, but she may as well rob the neighbor that lacks one. We’re relatively safe from folks trying to break my own brittle Tabroom specific code, as they’d gain so little from it.
Unless we really piss the Internets off. And that brings us to Ashley Madison.
That site was in a bad spot: their security resources were probably limited due to their size, but the cost of their data being exposed was clearly very high. Also, most people, their attackers included, found the Ashley Madison service appalling, which further increases the incentive for an attack. Folks whose morals would not permit them to hack a bank had no compunction about targeting Ashley Madison. The attackers went out for blood, and they sure got it.
Their aim was nothing less than bringing the whole service and its parent company down, and they succeeded. They focused specifically on Ashley Madison, stole all their data, and made it public. Clients were embarassed, awful conduct exposed everywhere, and the service and its company died. Moral of the story: if you run a odious, sexist website that caters to powerful assholes, while keeping deeply damaging personal data about said assholes in a database, try to stay up on your security patches.
What they did not do, after all that time and effort spent conducting a targeted attack, is sit around changing people’s text message alert phone numbers, whose full impact is annoying me on a Sunday morning. Therefore, I didn’t have a hard time dismissing the presence of a hacker after a cursory check for evidence thereof.
If not an evil attacker then, where the hell did (214) 748-3647 come from?
I investigated further. I set myself up to follow one of my own debaters, and put in my number, which is from the 978 area code. And then I look it up in the database:
(214) 748-3647.
I try some other numbers. Same result, every time. Now, I’m really alarmed. I wonder in horror if I’ve written some bug that substitutes a random Tabroom user’s phone number in for anyone else who follows an entry or judge. I say a silent prayer for whatever poor sod out there whose phone went through a sudden seizure every time Yale, Greenhill or Georgia State published a round.
I check the accounts database: the place that stores your number permanently, to follows your own competitor or judge records. No active Tabroom login has that number. So if this a person in the speech & debate community, they’re not very active.
I read over the following-code very very carefully, and check the value of the variables therein at every stage of its operation. Nope, it was fine.
Finally, I bypass my code altogether, and run a direct database command to change the follower record I have created to my 978 phone number. This method accesses the database directly, bypassing all Tabroom code, and therefore any bugs I might have created.
And I get (214) 748-3647.
I then send a runner to find Father Hahn and tell him to bring a gallon of holy water to tab. Kid thought I was joking, because the good Father never came, but I’m not sure I was.
There are moments in computing where you sink into despair. You write a function to do one thing, and it acts in a way so contrary to expectations, intuition and experience that you start to lose hope and faith in the world as a predictable place. You say to yourself: “Self, is this the day that the universe has decided to focus a small part of its attention solely on driving me crazy? Have I finally snapped? If the database functions that way, then is the speed of light still a constant?” Also, I’m usually flying solo at tournaments, without someone to provide a sanity check. There are approximately ten people in the entire country who fluently understand both tournament tabulation and tech. I cannot begin to explain some problems to anyone else, because we don’t share enough technical language to even have a conversation. It’d be like your mom offering to help you cut a counterplan. The hours spent explaining the premise wouldn’t be worth the twenty minutes of help.
So when I reach the end of logic, and find myself like Wile E Coyote running on thin air after running out of road, falling only when I look down — during those moments I can only sit there and stare at the screen, wonder what I’m missing, and hope I don’t start weeping. It’s important for computer people to keep up an all-knowing appearance — folks are afraid enough of tech without realizing that I’m sitting there, with all the tournament data in my hands, in a blind panic and ready to run for the nearest airport. That’s why Sarah Donnelly gets instantly concerned when she observes that I’m wearing my “oh-no face.”
I decide instead to take a mental break, which can often help in those situations. I start to randomly wander around the internet. I wonder if I can look up who has this phone number. Sometimes you can Google a phone number and find out who it belongs to, if they’ve posted it on the web somewhere. Instead, I get…a Wikipedia page for Mersenne primes?
Mersenne primes are prime numbers one less than a power of two. 3 is a Mersenne prime, as is 127. Powers of two are important in computing, since all computer storage is ultimately binary data, consisting of solely zeros and ones. That’s because data is represented with small magnetic charges, which are either present (one) or absent (zero) in a given region of a disk surface, giving you 2 digits to work with. Each such tiny field of data is called a bit, which is the smallest unit of computer data. So computers count by 2s, not by tens like we do. Base-2 counting means every additional digit, or bit, represents an additional power of two. 1 in binary is 1 in decimal counting too. But binary 10 is 2. 100 is 4. 1000 is 8.
1 1111 means you add the powers of 2 shown together: 1+2+4+8+16= 31, which is one less than the next digit alone, 10 0000 = 32. Mersenne primes like 31 are easy to spot in binary: they are all 1s. Not all numbers that are all 1s are Mersenne primes, however, since they’re not always prime numbers — 15 is binary 1111, one less than a power of two, but not prime.
The amount of data a computer can store is always one bit less than a power of two. Eight bits of memory (a unit known as a byte) could store the number 255, which is 1111 1111. But it can’t store 256, which requires nine bits: 1 0000 0000. Folks in computing therefore tend to be very familiar with powers of two. If someone you know refers to 512 as a “nice round number”, they’re probably a programmer. A t-shirt I may or may not own proclaims “There are 10 types of people in the world, those who understand binary and those who don’t.”
A few years ago hard drive vendors pulled a fast one on its customers and redefined data storage units. Normal people generally assumed a gigabyte equaled 1,000 megabytes, since we think in terms of tens, but in fact it was equal to 1,024 megabytes (minus one bit), which is a power of 2. But vendors took advantage of that assumption, and attempted to redefine a gigabyte to 1000 MB, and then in turn a terabyte to 1000 GB, basically shorting you of space. Pop that “1 TB” drive into a computer, and the computer will report that it’s only about 980 GB in size, because it defines these units in binary terms, not decimal ones.
If your storage contains a certain number of bits, you can express numbers up to one below 2 to the power of that number of bits. Those limits are therefore often Mersenne primes. 32 bits of memory, to draw a non-random example, can represent a number that’s (2^32 – 1) in size. That’s the limit for an unsigned integer, which are only positive. The more common signed integers sacrifice a single bit to indicate whether the number is positive or negative. The largest signed integer you can store in 32 bits is therefore equal to (2^31) – 1.
Which would be 2147483647.
When you design a database you must pre-declare how large the data in any given column is going to be limited to. The reason is that the database will allocate the space for data in set, predetermined chunks. If you say a given data field should be big enough to hold 255 characters, and then put only 2 characters into it, that field will still take up 255 characters worth of storage. So if you’re storing, say, state abbreviations, good practice is to limit the field to 2 characters only, as correct values will never be larger. With stuff like proper names or so on, you make a decent guess, and leave some padding. If anyone in debate has a first or last name longer than 63 characters, be prepared for disappointment when you register on Tabroom.
Before last week’s data conversion, I stored phone numbers as strings, which can be letters, numbers or punctuation, instead of just raw numbers. So if you typed in (978) 555-1234, that’s what I’d store, spaces, parens and all. But the text-email gateways only want numbers, nothing else. So every time Tabroom sent a blast out, it would have to first remove all those non-number characters. That costs a little extra processing time, but more importantly, it’s bug prone: I had to do it every time I accessed phone numbers, so if I had to change how I do that, I had to change it every place I do so. And inevitably I’d forget at least one and create a bug. Also, storing characters takes up more space than storing raw numbers only, and keeping all that extra punctuation is wasteful since I was just going to remove it anyway. So, in the update, I changed how that works, and now strip out the non-numeric characters before storing your number, keeping only the numbers themselves.
Database designers tend to pay close attention to string sizes, but when storing integer numbers, it’s easy to get a bit lazy and just declare the default sized integer, an INT: 32 bits. That’s a default for historical reasons. Most computer processors made from around 1995 until 2005 had 32 bit instruction sizes, which means the largest numbers they could process in a single chunk within the chip itself was 32 bits in size. A 32 bit computer can deal with larger numbers, but it has to pull tricks and any such operations are slower and less precise than dealing with ones that fit in its pipeline. This limit affected how quickly and precisely 32 bit CPUs could do math involving large numbers, how much RAM memory their systems could manage, and even how long they could maintain their internal clocks: 32 bit processors will stop keeping correct time on some operating systems in 2032. Nowadays standard processors have 64 bit instruction pipelines, but 32 bit limits remain common throughout computing, because folks got used to that being a reasonable default and it’s stayed that way since.
Save a number into a database field that’s beyond its ability to express, and the database will save instead the largest number it can, the Mersenne prime 2147483647.
And thus, my bug.
Why I didn’t see it until now is a lesson in psychology. In Tabroom, by default you must enter each ballot twice before it is accepted. I discourage people from using visual scanning as an audit method, because when you see or hear a number and have to confirm that a second number matches, sometimes your brain sees what it expects to see, rather than what it actually sees. It’s better to have to read and interpret the ballot twice, preferably by two different people, so no expectation biases exist. That same human flaw stung me — I was looking at it expecting to see a phone number, and it was indeed a valid area code followed by seven digits. So I saw a phone number, and didn’t recognize it for what it was. As much as that seems like a really random number, it’s fairly well known in computing because 32 bit size limits are very common. It’s known as max int. I could have recognized it, but I was in the wrong mental context.
It so happens nobody has max int as a phone number; we called it and got a busy signal. I did not spend the weekend spamming some poor schmuck in Dallas with thousands of demands that they be affirmative against Lexington SM in US 302 in front of DHeidt while also judging an LD round in WLH in New Haven and watching a Northwestern team run the K in Atlanta — which actually happened, so at least my experience wasn’t the weirdest of the weekend. My messages, and your text blasts, just disappeared into the ether, rejected by bewildered cell carriers which did not have that number and would you please stop asking us to text it, Tabroom.com?
When the Gangnam Style video became the most popular Youtube video ever, it was the first to achieve more than 2,147,483,647 views, and for a while the counter stuck there. It was rumored that Google fell victim to this same error.
After an hour and change of searching, despair, crazed laughter, learning new things, beating my head against my screen, and both sass from and mocking of BMan, it took six words to fix:
ALTER TABLE follower MODIFY cell BIGINT;
A BIGINT is 64 bits. It can store a number up to 18446744073709551615. Unless the phone network grows by a lot, we should be good now.
Then I went and got another chai.
I hate computers.