the loneliest moment

A paradox of debate is that an activity where thousands of people argue with each other at once can still leave us lonely.

I don’t buy into personality type tests, but I am certainly an introvert. Constant exposure to people wipes me out, but when I go to tournaments, my place is usually at the center of the storm. That costs energy. So, after all is done and I fly or drive home, I’m ready for a break from all you screaming performers.

But I live alone. Before this year, I’d return home to an empty and cold house and it would be too quiet, too fast. The stampede of debaters and judges who all rely on me for assignments and results were gone, and I’m just left with an echo. It’s strange to be lonely at the same moments you most want to be alone, but on those Tuesdays Afterwards? I was.

But that was the Before Times. The age of lockdowns has been unkind to us all. I’m a loner, sure, but also a nomad: I used to travel constantly, and even when I was home, I’d leave the house every day, if only to go work in a coffee shop or the library. The buzz of people around helps me focus, and the journey forms a boundary between work and not-work. In other words, I need people around that I can ignore. And it’s a rare introvert who can make do without human contact at all. I’ve found myself talking back to podcasts, or pacing around the living room for an hour at a time, or wasting gas just to change the scenery.

It feels like the worst year of our lives, even for those of us who’ve so far avoided direct harm from the greater wounds 2020 offers: the virus itself, the poverty it has sparked, the smoke and flames of an entire coast ablaze, or the constant reality that police can end black lives and stay free. Against that balance of misery, my own debts seem minor. But the lesson of a lonely time is that sometimes, no one is there to look out for you. Right now, no one can be, even if they wish to. You have to be careful of your own troubles, even if others have it worse. Being lonely and down might be the baseline right now, normal by majority vote. But it is no less healthy or easy to shoulder.

I bear the dubious honor of being Debate Famous. There are several ways to become Debate Famous – you can win a lot of tournaments, serve on a lot of tab staffs, national boards or committees, or just be obnoxious for long enough. Some manage all three! But I took perhaps the easiest route, and joined a tiny band with few entrance requirements and even fewer members. I am one of the Techies.

I cheat in calling us a band, for one of the defining features of a debate techie is that we all drift on separate islands. Each debate techie is tied to a Project. These projects are usually the residue of a hobby idea that got out of control. Tabroom is certainly that: it now runs 3,000ish tournaments a year and serves millions of hits per weekend. It’s a luxury beyond telling that the NSDA can make it my full time job and dedicate others to helping with the tsunami of emails that results from its popularity.

But at the end of the day, even with that support behind me, I’m still alone. Nobody understands Tabroom and its internal workings half so well as I do. If something major breaks, I can fix in minutes what might take anyone else several days. Speech & debate is never so lucrative that our scant resources can hire me a co-programmer; they’ve had to stretch to underpay me enough to do this full time. And that’s the root of my Debate Fame, because I am Tabroom, and it is me, as far as the debate world sees it. Every user of Tabroom therefore is one who relies on Palmer.

Such “fame” comes with a price, for technology is brittle. From time to time, some hidden capacity limit is reached, or a buried mistake in code I wrote six years ago decides TODAY IS THE DAY! My machines start to stagger, and disconnect, and lock up. My phone starts dancing with messages, half of which start “You probably already know about this, but… .” People have no choice, because only I can type the keys that get your tournament started again. Sometimes I fix it immediately, and people barely notice the trouble. Sometimes it takes me an hour, or two – or four – to find the illness and its cure. And those times are the loneliest moments I know.

If you were in a tab room with me, you’d just see me beating the crap out of my laptop keyboard. You’d hear me mutter incomprehensible nonsense to myself, as if I were chanting spells to appease furious eldritch demons of silicon and resin. I’d be zeroed in, focused on the screen, phone definitely muted. But I’m still utterly aware of you all. I can feel your eyes, the eyes of each and every one of the tens of thousands of you who rely on me to continue your tournament day. You might not know where I am, or even what I look like, but I can feel your eyes all the same, in the place where my neck meets my shoulders that tighten and coil with the strain of it. And I can’t step away, I can’t delay, because there’s no one to hand the problem to, and thousands are waiting.

It’s a pretty steep personal cost, this consequence of the realities of our activity, and the ever insufficient resources we have to stretch to meet our problems. Software is delicate, with so many layers and complexities that are impossible to fully predict or understand. Imagine an engineer designing a bridge: they carefully calculate known stress factors, material strengths, expected weights and the like to arrive at a construction whose weakest point is much stronger than the load it will bear. Now make the engineer do it without knowing what material half the bridge is made out of. Throw three hundred hollow rivets into their supply. Then build sixteen more bridges stacked atop the first one, all with unknown materials and different designers. Would you drive across those bridges? But that’s software development for you. Thankfully, unlike our hypothetical stack of bridges, nobody dies when speech & debate tech collapses, though you wouldn’t know it listening to people sometimes.

Because of this constant ticking disaster we call software, companies can spend billions on people like me. Vast teams of techies find and fix expensive problems, but those billions can only make outages happen less often, not never. Google had serious downtime just two days ago, and Tabroom’s entire annual budget is a rounding error in their departmental catering bill. But Google’s wizards are not underpaid, or alone. Their problems are greater in size, but not much different in kind.

When Debate Techies get together, that’s what we talk about, those lonely moments. You may imagine great rivalry between Speechwire and Tabroom and TRPC. There is none; I can think of no greater personal nightmare than Speechwire disappearing and having to fill even a portion of the gap it’d leave behind. But even if we did view each other as the Enemy, you would never see me laugh at Ben when Speechwire goes haywire, and he would never do the reverse. The price of admission to the little club of Debate Techies is understanding what that moment feels like. You cannot see another suffer it without sympathy and remembering your own terror.

The era of covid has affected debate like everything else. Online debate was always a side hobby project of a few visionaries, but never got much real traction – until suddenly in April it was everything. Every member of the little band of debate techies had to drop all plans and change our entire world overnight. 2020 Nationals was going to be the first all-online balloted Nationals anyway, but overnight that was no longer a Project but an Assumption. Priten suddenly joined our ranks with his terrific Classroom.cloud project, and therefore saved the TOC and Nationals both. He got his very own baptism, with a slowdown and lockup the first day of Nationals; I spent those moments in the Des Moines office preventing people from calling him. I know.

But Classrooms is based on Zoom and therefore can be pricey. Large expensive tournaments can swing it, but in the world as it existed in April, the type of small local tournaments I grew up in – the Massachusetts local that charges about seven bucks an entry – could not begin to afford it. Circuit kids may mock those types of tournaments, but speech & debate wouldn’t exist without them. And so, we set about trying to find a way to keep them alive, and the result is NSDA Campus. I’ve helped with Campus at the edges, but again all speech & debate tech must be lonely, and this burden belongs to Hardy.

It turns out spinning up a custom private video conferencing on-demand service given about 3 person-months of work and a shoestring budget isn’t simple. So we’ve had our problems. The first couple of weeks went fine, but then we hit a threshold of usage that triggered an odd undocumented condition in our proxy service — the traffic cop that keeps all the traffic for your particular PF round going to the same server so you can see your opponents and judge. The proxies went nuts and started sending people anywhere and nowhere. It was not because we didn’t have enough server capacity — we’re running our servers on Amazon’s cloud, which also hosts services you might have heard of, like “Netflix.” As I write this 128 machines are serving Campus rooms just fine. The flaw was a condition buried deep in someone else’s code that would only manifest when we had more than 400 rooms going at once. It’s fixed and behind us.

Today we hit another, tripped by a new threshold of 3,000 users. Hardy found these new issues, and fixed them too. Because that’s what we do in the little club of Debate Techies.

It’s hard to test these things. We don’t employ a couple thousand people who can be drafted to all join online rooms at once, just so we can see what breaks. There are ways to simulate that type of load in testing environments, but setting such things up is time consuming too, and each of us lonely techies has an enormous list of problems to deal with right now. It’s hard to find time for the future’s problems when when we have so many already on fire in the present. And of course, only one of us can solve most of them. Hardy is the person who understands my tech province best, and vice-versa – but if we traded to-do lists, our productivity would nearly vanish. The difference is like translating text into a language you’re fluent in, versus one where you have to look up every 5th word.

And so was Hardy was condemned to another pair of lonely moments for each of those bugs. He knew they’d come. We all do.

Because today the demons came for him and not me, I feel more free to speak, to point out the underlying realities of our activity, and to shed light on the effort we’re making to keep our speech & debate circus going despite a global pandemic. The type of tech required to do debate online is only barely ready for what we’re asking it to do. If covid-19 hit 10 or even 5 years ago, speech & debate would simply have shut down. As it is, we haven’t caught our breath. Everyone in the debate tech world has been running full tilt for months now, trying to get this all to work – and sometimes, we fail. We can promise you effort. We can never promise perfection. Neither can Google and its billions.

But perhaps, on behalf of the other members of my little tribe, I can ask for more clemency, at least in public. It is unkind at best to churn out memes about a service that someone just spent three days not sleeping to fix for you. It’s unwise to hit Facebook to air grievances or unfounded theories as to what the issues are. And most of all, it’s bad form to suggest we give up on the whole project, and give up affordable speech & debate tournaments with it, because of a few software kinks that made you wait around a couple hours – at home, no less – in the early days of an immensely complicated project that did not exist and was not planned for six months ago.

When you do so, you underestimate the realities and the economics that go into the projects that us lonely techies are keeping alive so that debaters can debate and speechies can speak. But more than anything, what you do most of all is take our loneliest moment, and make it lonelier. In the era of covid, when the gaps between us in real life are so huge and enforced by a deadly and invisible enemy, we should be careful before adding to another’s isolation.

And do remember, none of us in debate tech are in it for the fabulous wealth and prizes. We do this because we’re tied to the activity, and find it worthwhile to make your competition work better and faster in normal times – or to work at all, now in the age of pandemic. We do it because I too once paced in prep rooms before giving extemp speeches, while Priten and Hardy blitzed through policy debate rounds and Ben prepped his IEs.

Any of us could expand our wealth and free time by leaving speech & debate, and the lonely moments it causes us. We stay because of the satisfaction and our connection to this community. So consider please the living, breathing, lockdown-trapped & lonely person at the other end of that link before you decide to trample on us because the fragile tech we’re trying to build snapped today. Even if the meme is funny.

Making room for beauty

I have long concealed a dark scandalous secret.  I’m not a true computer nerd.

Don’t protest.  It’s true.

Yes, I have a lot of the skills of nerdosity.  I can and do program for a living.  I can and do fix computers all the time.  I can and do understand them at a level that almost everyone else cannot.   But I know the difference between me and the True Nerds; I don’t design and implement operating systems, or cryptography schemes, or new programming languages or frameworks, and ultimately it’s because  I lack the passion for it.  For me, technology is operational, and interesting only insofar as it is useful.  I only occasionally tinker; once the Thing Is Working, I am satisfied and leave it alone in favor of things that are not.  So I don’t tend to dig in and reach that next level of true understanding that a True Nerd finds so satisfying.

And yet, I spend almost all my life mashing a keyboard and churning out computer code.  I travel across the country on a regular basis to do onsite training, tech support and more coding even from cheap hotels, high schools or colleges as I can find the time.  My family is never quite sure what time zone I inhabit at any given time.  I don’t own pets for fear they’d surely die, and my plants tend to be the type that can sustain minor droughts.  I sure don’t do it for the money; I could probably triple my annual income by focusing on my geekery alone and going to work for Google or some such masterpiece of the Nerd Kingdom.   I do not get to travel in the fun sense much more than the average person; for all that I’m constantly in different places, I mostly inhabit classrooms and airport hotel ballrooms, and such things look the same in Miami and San Diego and Philadelphia and wherever else I find myself.

But I’m not complaining.

I work as a software consultant to the world of speech and debate.  I work with the National Speech and Debate Association for most of my time, and have side work with the Boston Debate League serving inner city debate in Boston, and consult with numerous individual tournaments as well; I’m writing now from an airplane headed towards the Pi Kappa Delta Nationals, a collegiate debate and speech competition, after tabbing the American Debate Association nationals last weekend; last few months saw me at Cal Berkeley for a high school tournament attended by over 3,000 people, and before that the University of Texas at Austin, Charlestown High in Boston, Emory University, Lexington High in MA, and before that UC Berkeley again.

I have an awful lot of Delta miles.

Such tournaments are amazing experiences that we who live with the world don’t always step back to appreciate.  On the weekend of the Cal Berkeley tournament I helped run that event where 3,000 high school students got up in front of judges and spoke.  Some spoke of high philosophy and the morality of handgun ownership, some spoke pre-prepared dramatic presentations, some spoke of the US surveillance state and its limits and benefits, some gave speeches they wrote themselves on a topic of their own choice, and still others overrode the set topics they were assigned  to debate and instead injected their own culture, identity and viewpoints into their debate rounds.  But all of them spoke,  multiple times, in front of audiences large and small, about topics whose depth and emotional impact often belied the age of the speakers; high school and college students, almost all between 14 and 22 years of age.

Middle schoolers compete  to0, some as early as fifth grade; I just didn’t happen to go to any tournaments with them.  Not yet, anyway.

While I was at  Cal, an equally large number of students were doing the same thing across the country at Harvard, with smaller but still large events happening elsewhere, at UPenn, at  Pinecrest in Florida, and in countless other high schools across the country.  President’s Day is a remarkable weekend in the world of speech and debate;  during it,  well over ten thousand young people across the country stand and speak  anywhere between three and twenty  times apiece.

There are intense controversies within the debate and speech world.  Some competitors  play fast and loose with the rules of the material presented in the dramatic events, or address uncomfortable and controversial material in their speeches, and not everyone approves.  Some debaters object to the idea that others can and do ignore the official topic in a lot of rounds to promote  their own agendas, or can engage in sometimes quite personal ad-hominem attacks or tactics to win a round.  Others still dislike how arcane and rapid-paced many  debates have  become, freezing out communication and persuasion in favor of a baroque form of logic, and arguments in quantity instead of quality.  The edifice of speech and debate is undeniably imperfect, and often unsatisfying.

But it is never static; it is a living work, a collective action by a cast of thousands who make it what it is at any given moment.   Our current controversies do not get in the way of the ultimate mission: to encourage young people to speak, and stand and be listened to; to overcome the huge fear most people have of standing up and being heard.  The core of speech and debate, the core of being heard and believed, is knowing what to say; speech and debate encourages critical thinking and breaking boundaries, rewarding people for finding a different way of expressing an idea that nobody else thought of.  Those mavericks are the ones who get the biggest trophies.   Small wonder, then, that our rules are fluid and flexible and often abandoned; they’re under constant attack, along with every other idea in speech and debate.  But even in the resulting chaos, there is no better crucible for young minds.

And the effect is clear.  The parents of my team can never get over what happens to their children when they join speech and debate.  One confessed she started having to look up words her 15 year old casually used at the dinner table.  The students share their insights with their families and other friends.  Donald Rumsfeld, during testimony before the National Commission for Terrorist Attacks  in 2004  , called the person who sets the annual debate topic the most powerful person in the country.  Debaters can instantly speak  with authority about hegemonic foreign policies, afro-pessimism and social justice, or meta-ethical frameworks behind moral decisions.  Speech kids might start talking about the economy or the election at the drop of the hat, or be able to convince you in their performance that a full cast play is happening in front of  you, while just one person performs it.

We hope that  getting the young of the country to be unafraid to think and speak on what matters will create a  habit that  sticks.  And stick, it has.  I have former students running for public office right now, directing Hollywood shows, clerking for Supreme Court justices — and teaching, learning and doing new things that don’t fit easy categories.  Debate is home to counter intuitive ideas that later become mainstream, as we work them out.  A lot of debate ideas sound patently ridiculous when they’re first advanced in the round, but the students capable of creating those ridiculous ideas go on to learn how to create breathtaking ones, and do so with the same skills we encourage: questioning everything, not allowing boundaries to stand in their way, and then thinking nothing of standing up and delivering their ideas to audiences large or small.

And we don’t talk over each other, at least not as much as you’d think.

At tournaments, two things happen.  One of them is this activity that I can only call pure beauty in its engagement and intricacy and energy.   The other is, unfortunately, practical: we do an awful  lot of waiting around.  Schedules must be produced, judges assigned to rounds, rooms opened and closed, ballots entered and results tabulated before the next schedule goes out.  The logistic  elements of a tournament are staggering, and often confusing and daunting to the newcomer.  Parents who ask what time things will  end are sometimes laughed at; tournament schedules are more often aspiration than promise.   These delays are not  intended and never desired, but often can’t be avoided; we have an awful lot of moving pieces at tournaments and even one that goes awry can sometimes throw the whole affair  off.

My primary claim to fame is creating  and maintaining Tabroom.com, a site that tries to make the whole thing as automatic as possible.  Tabroom does  scheduling, online ballots, registration intake and confirmation, communications and whatever else I can think of that makes things easier on tournament directors.  Tabroom.com has  grown by leaps and bounds in popularity, which imposes its costs and stresses in terms of support requests and cries for help.   Thanks to the NSDA, I do have assistance in manning the support lines, but also a new challenge: while I’m keeping the wheels spinning on Tabroom,  I’ve also been  feverishly working on Tabroom’s successor site, which will be called Treo.  The core technologies at the heart of Tabroom.com are aging and due for replacement; Treo will take advantage of new advances in frameworks,  languages and methods.

Tournaments, for me, are not fun.  They run me ragged.  Running a tournament is a 5AM to midnight type of job.  Most people run tournaments only once or twice a year, leaving time to  recover.  I do it every weekend.  I  would collapse if I were truly in the trenches every moment, so I have to fight very hard against my own impulses to carve out more time for sleep.  Even as I do it, and try purposefully to be selfish, I still never get  enough real rest while I’m at speech and debate tournaments.  I work almost every day, rarely taking a full 24 hours off of tabbing or coding or whatever else I do.  But all the while, I’m seeking ways to make one more button to shave off ten minutes here, fifteen there — and sooner or later, those minutes become hours and hours become days.

And I do it all not because I’m a nerd.  I do it because the better Tabroom and later Treo get, then tournaments will have more  beauty and less filler.  I aim to make the task of running speech and debate contests  ever easier, ever more automatic.  The better the software, the more time we spend on debate and speech itself.  It will then be easier for others to coach new programs and bring new students to tournaments.  It will be easier to host tournaments and run them, and provide the opportunity to more kids.

That’s why I do what I do.  That’s why I play a professional nerd even though my heart isn’t truly in it. I could do work that brought me  more direct happiness, but  I doubt I could find something to do  with more meaning.

Today, March 15th, is National Speech & Debate Education Day, by Senate proclamation no less.  It’s the USA’s participation in World Speech Day.  The day is intended to promote the collective work of intellect and beauty that I struggle each day to make a little better around the edges.  I’m not a true nerd, but I play one in the speech & debate world, to support and make ever more room for that beauty, and bring it to ever more kids.

And that, to me, is more than enough motivation.

Text blasts, Psy, and French math

Greenhill. Saturday night, 11:30 PM. I’m in bed, at long last, the sleep tank running empty. It’s been a busy week, after a series of database upgrades and the usual start-of-season challenges: users new to Tabroom, experienced users who forgot things since they last tabbed in March, new features not quite working, whack a mole bugs from changes over the summer. At that moment, BMan texts me to tell me the coaches of a Certain Program Somewhere out west are complaining that only some of them are getting the text message blasts for their debaters’ rounds.


Text messaging isn’t as reliable as you may think. The failure rate for SMS messages sent individually is somewhere around .5%. For a bulk blast such as Tabroom sends, it can hover somewhere around 1-2%. If that doesn’t sound like a lot, consider: when you blast out a pairing for a Policy round at Greenhill, you are sending a text to 432 people. If the failure rate is even .5%, 2-3 people aren’t getting that message. Blast out six prelims in LD and Policy, and a bunch of elims, and you have about 40-50 instances of someone not getting their texts. Sometimes it’s Mom stalking you from home who misses a text. Sometimes it’s you.

On President’s Day weekend, Tabroom sent out 114,988 text messages in three days. .5% failures mean 576 people didn’t get one text blast; up it to 2% and that’s 2,301 lost messages.

Furthermore, Tabroom and JOT don’t actually send out texts. Sending a text message directly is not free — the carriers would charge us to do it. Since we’re a non profit speech and debate association, and not a global megabank, we use a side door: many carriers offer a little-publicized service where you can text message their users via an email, free. If your phone number is 978-555-1234, and you’re on Verizon, emails to 9785551234@vtext.com come to you as a text. The address you email to is different for every carrier, which is why we ask for your carrier when you sign up. That’s also why some smaller carriers are missing from that list —  they don’t offer this service.

Anything you can email is also a thing you can spam. The carriers are pretty vigilant about monitoring these email gateways for spam attempts —  you’d be really pissed if you started getting Viagra ads on your texts, after all. A service such as ours which sends out 114,984 texts in the course of a weekend, while sending virtually nothing during the work week, will understandably  look like a spammer to anti-spam software. So sometimes it will happen that all T-Mobile customers suddenly stop getting Tabroom texts. I can sometimes negotiate around that, and notify the carriers that we are actually good citizens here, but not always quickly, especially on a weekend.

All of this is to say your SMS message blasts are handy, but you should still check the pairings online when they come out. It also means if you don’t get a particular blast, it’s not a problem I can do much about, and am inclined to brush it off a bit. Report it when I’m finally in bed after a long two days, exhausted and stressed, well… you can go ahead and assume that is not a time of any week I’m eager and ready to bust out the computer and sling some code to save you from having to check an online pairing because you aren’t getting texts. It being BMan — a good friend whom I respect and value, and who has done me multiple kind favors and helped me out on a number of occasions —  I of course replied with a rude gesture and ample profanity, rolled over, and went to sleep.


The next morning, however, I woke feeling better. I miraculously got about 7 hours sleep, which is a debate tournament equivalent of three full nights’ worth. I double fisted Starbucks chai lattes. And the wiki was running better, so things were relatively calm. I had some time on my hands. At that point, the lovely Mr Vincent comes in and tells me that he, too, hasn’t gotten a single blast for his student’s rounds, but has been getting all of his for judging.

That’s a great bug report, by the way. Whining that “it doesn’t work” gives me very few clues what could be wrong, and that in turn gives my surly nature an excuse to dismiss the bug and blame you. My threshold on that is pretty low. Just ask Menick. But CV’s fact pattern was more interesting — he was only getting SOME messages, and the ones he got followed a consistent pattern. Those details made me immediately realize there’s more to the story here, and take him seriously.

I pull up his follower record, and there it is. His email was following the debater, and it looks OK. His phone number was listed in the same record: (214) 748-3647.

“But,” he says, “that’s not my phone number.”

I pull up all the entries he’s following. The number listed: (214) 748-3647.

I look at his login, which is where his own judge blasts are going, and that number was correct. Huh.

I check where area code 214 is, and it’s Dallas. I’m at Greenhill, so I roll my eyes. “Is some kid at the tournament trying to prank me and messing with the system somehow?”  I tell the database to pull up every record of this phone number following entries or judges in Tabroom.

The count was 7,632.


Evil Hackers are poorly understood by the general public; they are not diabolical Armani wearing supermen using specialized equipment to cast secret magic across the network and bend systems by the power of their will. If someone ever says “I’ve hacked into their security cameras and am tracking their movements now” after five minutes of furious typing, the correct response is to nod warily but politely, close the padded door and ask the nurse to up their medication.

Servers are just computers that run software to respond to remote requests, and all software has flaws.   Hacking in the real world is a matter of finding such a flaw and exploiting it.   Systems are very complicated things, and here’s a lot of surface area for mistakes.   Most are harmless, but some are the dreaded backdoors.   A backdoor is a flaw that, when sent specifically formatted data input, will react by giving the sender system access.   Backdoors can lie undiscovered and unknown on systems for years, sometimes decades, before being spotted and exploited.

Finding these flaws and the data that triggers them is hard work for seriously talented and knowledgeable people.     However, hacking is not limited to that elite group.   Every so often, someone out there finds a backdoor, and writes a rootkit which sends the particular data to systems running the software containing the vulnerable code.   Sometimes these root kits are written just to prove that a backdoor is serious, and push the software maintainers to quickly develop a fix.     But sometimes they are released on the internet for anyone to use, and those become dangerous.   While only a few masters out there can create an effective root kit, many thousands know enough to use them once they exist.     If a root kit is released, and no effective defense or patch to close vulnerable systems yet exists, the flaw becomes known as a zero-day exploit.

Users of systems affected by a zero-day can do nothing about it except yank their systems off the network, or hope and pray they’re not targeted.   Zero-days are  given scary names, like Sandworm, Shellshock, and Heartbleed, and written about like they spell doom for civilization and humanity, which policy debaters seeking impact files find very helpful.   They’re big news.   They can be used to steal private information, or manipulate financial or government systems.   Big and important firms therefore have a strong incentive to quickly fix any such flaws the minute a fix, or patch, is released.

The process of applying patches to a server or its software is not much more complicated in principle than applying Windows or Mac software updates.   Institutions with a lot to lose, like banks, credit processors, or sensitive government departments are therefore usually diligent about patches.   So backdoors aren’t often the weapon of choice when attacking a bank or credit card company.     It’s far easier to wheedle a password out of a non-security conscious contract employee by claiming to be the IT department on the phone than to penetrate their network security.     Leakers or careless employees, not backdoors, are the weak point.

Root kits find better targets one step down the food chain: major retailers like Target and Home Depot.   Such companies   possess millions of customer records and credit card numbers.   But computing isn’t their core business, and their leadership tends to be less aware of security issues. Sooner or later, some bean counter up the corporate food chain asks the dread question: “why are we spending so much money on computer security?   I have anti-virus on my computer and that works fine.   Who cares if those nerds in the basement insist we need this stuff?” And IT isn’t exactly filled with people who can argue their case to humans effectively.   So a budget gets cut, security goes lax, and backdoors are left unguarded longer.

Tabroom is not a single program.   It relies on a stack:   an operating system, a database, a web server, code interpreters, and libraries that handle standard things like formats, printing, and timezone conversions.   There’s even various ancillary bits of software like an email server and ways I can access the system.   A  backdoor in any part of that stack could open up my machine to a rootkit.   I have to patch it all, and naturally, I spend rather less time and money on security than Target or Home Depot did.

So why was I skeptical that we got hacked?   Well, bluntly put, nobody cares.

All that software in the stack is made by large teams of coders, many having members entirely dedicated to security.   The part most likely  â€” by far  â€” to have some exploitable vulnerability is the Tabroom.com code itself.   We don’t exactly have a dedicated team of security experts reviewing our code for exploits.     But our low resources reflect our low value  â€” a backdoor in Tabroom would be incredibly  small potatoes.   Finding one gets you access to exactly two systems in the entire world, and one is kind of old and slow.   We’re big in debate, but tiny  overall.     It’s a lot harder to find an exploit in the web server we use, but if you succeed in that, you’d have the key to millions of systems, including ours.   So the web server gets the real hammering, and my code gets none.

The type of hacking a little niche site like ours has to worry about is teenagers with more time than sense,  armed with root kits they  know how to use but don’t truly understand.  They set their  kits to blast the internet at random, looking for unpatched systems vulnerable to their attacks.  Such hackers are called script kiddies.  Their motive, apart from the thrill of the hunt, is to use my machine to host computer game servers, or share porn and illegal software with their friends at junior high.  They might put my machine into a network of other hacked servers and use it as part of an denial of service  attack— where you flood a website with more traffic than it can handle, aiming to knock a site offline.  Ultimately they’re unlikely to care about Tabroom itself, and are only interested in it as a reasonably powerful computer connected to the internet that isn’t traceable to them, so they can do other illicit things without someone telling their parents.

We suffer constant generic attempts by script kiddies, aimed at the stack.   Patching that is easy, and done regularly.     It also helps that many other small site admins are lazy, and don’t.   I’m the neighbor with the alarm system — a burglar can still rob me and probably get away, but she may as well rob the neighbor that lacks one.   We’re relatively safe from folks trying to break my own brittle Tabroom specific code, as  they’d gain so little from it.

Unless we really piss the Internets off.   And that brings us to Ashley Madison.

That site was in a bad spot: their security resources were probably limited due to their size, but the cost of their data being exposed was clearly very high.   Also, most people, their attackers included, found the Ashley Madison service appalling, which further increases the incentive for an attack.   Folks whose morals would not permit them to hack a bank had no compunction about targeting Ashley Madison.   The attackers went out for blood, and they sure got it.

Their aim was nothing less than bringing the whole service and its parent company down, and they succeeded.  They focused specifically on Ashley Madison, stole all their data, and made it public.   Clients were embarassed, awful conduct exposed everywhere, and the service and its company died.   Moral of the story: if you run a odious, sexist website that caters to powerful assholes, while keeping deeply damaging personal data about said assholes in a database, try to stay up on your security patches.

What they did not do, after all that time and effort spent conducting a targeted attack,   is  sit around changing people’s text message alert phone numbers, whose full impact is annoying me on a Sunday morning.   Therefore, I didn’t have a hard time dismissing the presence of a hacker after a cursory check for evidence thereof.


If not an evil attacker then, where the hell did (214) 748-3647 come from?

I investigated further. I set myself up to follow one of my own debaters, and put in my number, which is from the 978 area code. And then I look it up in the database:

(214) 748-3647.

I try some other numbers. Same result, every time. Now, I’m really alarmed. I wonder in horror if I’ve written some bug that substitutes a random Tabroom user’s phone number in for anyone else who follows an entry or judge.  I say a silent prayer for whatever poor sod out there whose phone went through a sudden seizure  every time Yale, Greenhill or Georgia State published a round.

I check the accounts database: the place that stores your number permanently, to follows your own competitor or judge records.   No active Tabroom login has that number.  So if this a person in the speech & debate community, they’re not very active.

I read over the following-code very very carefully, and check the value of the variables therein at every stage of its operation.  Nope, it was fine.

Finally, I bypass my code altogether, and run a direct database command to change  the follower record I have created to my 978 phone number. This method accesses the database directly, bypassing all Tabroom code, and therefore any bugs I might have created.

And I get (214) 748-3647.

I then send a runner to find Father Hahn and tell him to bring a gallon of holy water to tab. Kid thought I was joking, because the good Father never came, but I’m not sure I was.

There are moments in computing where you sink into despair. You write a function to do one thing, and it acts in a way so contrary to expectations, intuition and experience that you start to lose hope and faith in the world as a predictable place. You say to yourself: “Self, is this the day that the universe has decided to focus a small part of its attention solely on driving me crazy? Have I finally snapped?  If the database functions that way, then is the speed of light still a constant?”  Also, I’m usually flying solo at tournaments, without someone to provide a sanity check.  There are approximately ten  people in the entire country who fluently understand both tournament tabulation and tech.  I cannot begin to explain some  problems to anyone else, because we don’t share enough technical language to even have a conversation.  It’d be like your mom offering to help you cut a counterplan.   The hours spent explaining the premise wouldn’t be worth the twenty minutes of help.

So when I reach the end of logic, and find myself like Wile E Coyote running on thin air after running out of road, falling only when I look down — during those moments I can only sit there and stare at the screen, wonder what I’m  missing, and hope I don’t start weeping.   It’s important for computer people to keep up an all-knowing appearance  â€” folks are afraid enough of tech without realizing  that I’m sitting there, with all the tournament data in my hands, in a blind panic and ready to run for the nearest airport.   That’s why Sarah Donnelly gets instantly concerned when she observes that I’m wearing my “oh-no face.”

I decide instead to take a mental break, which can often help in those situations.  I start to randomly wander around the internet.   I wonder if I can look up who has this phone number.   Sometimes you can Google a phone number and find out who it belongs to, if they’ve posted it on the web somewhere. Instead, I get…a Wikipedia page for Mersenne primes?


Mersenne primes are prime numbers one less than a power of two. 3 is a Mersenne prime, as is 127. Powers of two are important in computing, since all computer storage is ultimately binary data, consisting of solely zeros and ones.  That’s because data is represented with small magnetic charges, which are either present (one) or absent (zero) in a given region of a disk surface, giving you 2 digits to work with.  Each such tiny field of data is called a bit, which is the smallest unit of computer data.  So computers  count by 2s, not by tens like we do.  Base-2 counting means every  additional digit, or bit, represents an additional power of two.   1 in binary is  1 in decimal counting too.  But binary 10 is 2.   100 is 4.   1000 is 8.

1 1111 means you add the powers of 2 shown together: 1+2+4+8+16= 31, which is one less than the next digit alone, 10 0000  =  32.   Mersenne primes like 31 are easy to spot in binary: they are all 1s.     Not all numbers that are all 1s are Mersenne primes, however, since they’re not always prime numbers — 15 is binary 1111, one less than a power of two, but not prime.

The amount  of data a computer can store is  always one bit less than a power of two.   Eight bits of memory (a unit known as a byte) could store the number 255, which is 1111 1111.  But it can’t store 256, which requires nine bits:  1 0000 0000.  Folks in computing therefore tend  to be very familiar with powers of two.   If someone you know refers to 512 as a “nice round number”, they’re probably a programmer.  A t-shirt I may or may not own proclaims “There are 10 types of people in the world, those who understand binary and those who don’t.”

A few years ago hard drive vendors pulled a fast one on its customers and redefined data storage units.  Normal people generally assumed a gigabyte equaled 1,000 megabytes, since we think in terms of tens, but in fact it was equal to 1,024 megabytes (minus one bit), which is a power of 2.   But vendors took advantage of that assumption, and attempted to redefine a gigabyte to 1000 MB, and then in turn a terabyte to 1000 GB, basically shorting you of space.  Pop that “1 TB” drive into a computer, and the computer will report that it’s only about 980 GB in size, because it defines these units in binary terms, not decimal ones.

If your storage contains a certain number of bits, you can express numbers up to one below 2 to the power of that number of bits. Those limits  are therefore often Mersenne primes.    32 bits of memory, to draw a non-random example, can represent a number that’s (2^32 – 1) in size.   That’s the limit for an unsigned integer, which are only positive.     The more common signed integers sacrifice a single bit to indicate whether the number is positive or negative.   The largest signed integer you can store in 32 bits is therefore equal to (2^31) – 1.

Which would be  2147483647.


When you design a database you must pre-declare how large the data in any given column is going to be limited to. The reason is that the database will allocate the space for data in set, predetermined chunks. If you say a given data field should be big enough to hold 255 characters, and then put only 2 characters into it, that field will still take up 255 characters worth of storage. So if you’re storing, say, state abbreviations, good practice is to limit the field to 2 characters only, as correct values will  never be larger. With stuff like proper names or so on, you make a decent guess, and leave some padding.  If anyone in debate has a first or last name longer than 63 characters, be prepared for disappointment when you register on Tabroom.

Before last week’s data conversion, I stored phone numbers as strings, which can be letters, numbers or punctuation, instead of just raw numbers. So if you typed in (978) 555-1234, that’s what I’d store, spaces, parens and all.  But the text-email gateways only want numbers, nothing else.  So every time Tabroom sent a  blast out, it would have to first remove  all those non-number characters.  That costs a little extra processing time, but more importantly, it’s bug prone: I had to do it every time I accessed phone numbers, so if I had to change how I do that, I had to change it every place I do so.  And inevitably I’d forget at least one and create a bug.  Also, storing characters takes up more space than storing raw numbers only, and keeping  all that extra punctuation is wasteful since I was just going to remove it anyway. So, in the update, I changed how that works, and now strip out the non-numeric characters before storing your number, keeping only the numbers  themselves.

Database designers tend to pay close attention to string sizes, but when storing integer numbers, it’s easy to get a bit lazy and just declare the default sized integer, an INT: 32 bits.   That’s a default for historical reasons.  Most computer processors made from around 1995 until 2005 had 32 bit instruction sizes, which means the largest numbers they could process in a single chunk within the chip itself was 32 bits in size.  A 32 bit computer can deal with larger numbers, but it has to pull tricks and any such operations are slower and less precise than dealing with ones that fit in its pipeline.    This  limit affected how quickly and precisely 32 bit CPUs could do math involving large numbers, how much RAM memory their systems could manage, and even how long they could maintain their  internal clocks: 32 bit processors will stop keeping correct time on some operating systems in 2032.  Nowadays standard processors have 64 bit instruction pipelines, but 32 bit limits remain common throughout computing, because folks got used to that being a reasonable default and it’s stayed that way since.

Save a number into a database field that’s beyond its ability to express, and the database will save instead the largest number it can, the Mersenne prime 2147483647.

And thus, my bug.

Why I didn’t see it until now is a lesson in psychology.  In Tabroom, by default you must enter each ballot twice before it is accepted.   I discourage people from using visual scanning as an audit method, because when you see or hear a number and have to confirm that a second number matches, sometimes your brain sees what it expects to see, rather than what it actually sees.   It’s better to have to read and interpret the ballot twice, preferably by two different people, so no expectation biases exist.   That same human flaw stung me — I was looking at it expecting to see a phone number, and it was indeed a valid area code followed by seven digits.   So I saw a phone number, and didn’t recognize it for what it was.  As much as that seems like a really random number, it’s fairly well known in computing because 32 bit size limits are very common.  It’s known as max int. I could have recognized it, but I was  in the wrong mental context.


It so happens nobody has max int  as a phone number; we called it and got a busy signal. I did not spend the weekend spamming some poor schmuck in Dallas with thousands of demands that they be affirmative against Lexington SM in US 302 in front of DHeidt while also judging an LD round in WLH in New Haven and watching a Northwestern team run  the K in Atlanta — which actually  happened, so at least  my experience wasn’t the weirdest of the weekend.  My messages, and your text blasts, just disappeared into the ether, rejected by bewildered cell carriers which did not have that number and would you please stop asking us to text it, Tabroom.com?

When the Gangnam Style video became the most popular Youtube video ever, it was the first to achieve more than 2,147,483,647 views, and for a while the counter stuck there.  It was rumored that Google fell victim to this same error.

After an hour and change of searching, despair, crazed laughter, learning new things, beating my head against my screen, and both sass from and mocking of BMan, it took six words to fix:

ALTER TABLE follower MODIFY cell BIGINT;

A BIGINT is 64 bits. It can store a number up to 18446744073709551615. Unless  the phone network grows by a lot, we should be good now.

Then I went and got another chai.


I hate computers.

Semiotics

I actually wouldn’t mind ordinals; the idea you’d spend hours figuring out distinctions between your 75th and 76th judge is spurious.   If you can’t distinguish between a couple judges, then it doesn’t matter what order you put them in: just put one as 75 and one as 76 and move on with life.   It’s easier in practice than you’d think.   Plus, Tabroom actually has rather nice features built around ordinals; it can auto-generate a pref sheet for you based on your past ratings of a judge as a starting point.

And you’ll pardon me if I casually shrug off the conservative objections to Something New from the group of tabbers whose original schedule for switching over to online tabbing & balloting would have had us trying it out for the first time in early 2016.

But to see the point I’m trying to make about narrower band categories, you start with an ordinal sheet; the “true” sheet of how a given debater prefers judging.   A true ranking of preferences won’t necessarily be a smooth linear progression; it’ll have little clumps.   There’ll be a nest of three judges in the topmost spot, and then maybe a clear 4th, then maybe 3-4 judges tied for 5th, and so on.   But the size and location of those clumps will be random enough that the pref sheet is essentially a gradient.

A tiered system necessarily imposes arbitrary boundaries on that gradient, turning them into bands.   The fewer (and wider) the bands, the more information is lost.   Menick says that when you rate a judge a 2, they’re a 2, and thus are magically mutual with every other 2.   But plonking a label on a judge doesn’t make it so.   The judge could be my favorite 2 and your least favorite 2, in which case the judge isn’t very mutual at all; there’s a lot of slope between our opinions.   The most mutual judge, in fact, may be my favorite 2 and your least favorite 1, being separated only by one notch made more significant by the random chance of the tournament policy.

Look at tournaments that require you to rate 25% of the field a 1 and 25% a 2.   You’ve now rated half your judges in the top two tiers; and you’re going to get all kinds of judge matchups where the gap in preferences may differ by as much 24% of the field.   A 1-2 matchup in that case could be killer.   It’s not uncommon — and not difficult — for such tournaments to put out pairings with all 1-1 matchups, maybe a few 2-2s.   These matchups sometimes really stink; you end up with your opponent’s favoritest judge in the world when it’s the person you held your nose and plunked a 1 because you really needed just one more.   I’ve been on the wrong end of those pairings.

Now take Lex, where 11% were 1, and 11% were 2.   Lex’s 1-2 pairings are a little more precise; they all fall within the top 22% of your pref sheet.   Further, by drawing a line in the middle of that pool of judges, you can minimize their number.   There’s nothing a tournament requiring 25% of the field to rank 1 can do to minimize judge pairings that may be 24% of the field off on people’s actual preferences; it doesn’t have the data, and so doesn’t know that another 1-1 might actually be far more mutual in the debaters’ actual preferences.   Lex didn’t have any matchups 25% of the field off, and minimized the number of pairings that are 11-22% of the field off.     It’s more mutual, by which I mean more reflective of the actual preferences underlying the numbers on the pref sheet, not the fake mutuality of categories, which conflate the signifier and the signified.

I don’t buy the argument that new schools are going to be screwed by not figuring this out; that’s an argument to abandon prefs, not an argument to make them blunt versus narrow.   It’s also non-unique harm; new schools have to figure out theory, framework, impacts, cards, and spreading too.   It’s not the pref sheet keeping rookies out of the bid round.   A new school could get a judge that really favors their style and disfavors mine, and chances are my kid will beat them anyway, if only because I will know the judges’ preferences and the new coach won’t.   “You have to be smart to do that!” is true of nearly everything in debate, and should rather remain so.   Plus, LD is way more open to new schools and disruption than policy is; new schools are competitive on the national circuit all the time.   The answer to this is helping and teaching new schools how debate operates, not changing how debate should operate.   I put my money where my mouth is there, too: I’ve filled out pref sheets for lots of other schools.

In the end the debate is immaterial, I believe.   Whatever the merits, I think the trendline towards greater precision is inevitable.   TRPC limited LD to 6 while Policy could choose 6 or 9; 6 remained the norm in LD but in Policy most tournaments use 9 — in other words, both ended up using the largest number of tiers they could.   College policy, thanks to the good professors Larson and Bruschke, had software that did ordinal prefs years ago, ordinals they have used, with few exceptions.   Now that Tabroom has lifted the arbitrary 6/9 limits, I think high school debate, for high stakes tournaments anyway, will move towards ordinals, no matter what we say or do.   I embrace it, but even if I didn’t, I would think it’s going there anyway.     It’s a natural progression outwards from strikes to prefs, when you think about it; strikes are simply a really inaccurate pref sheet.   So I’d say, practice your ordinals…

…and just wait until I explain the percentile system to you.

Answering innumeracy with data

He still doesn’t understand my point about increased mutuality, but I’ll write that up when it’s not   12:30 AM on a Tuesday.

For now:

  • Percentage of 1-off judges at Lexington:     8.46%
  • Percentage of 1-offs at Columbia:   8.33%

Man, really blew mutuality to shreds with those 9 tiers at Lex, didn’t we?

At Columbia, 12.5% of the VLD pool did not pref; at Lexington it was only 9%, thus making the job harder at Lexington to boot.     As we say in the business, “No Link.”