Category: now

TRAVEL

I had a fantastic time in Taiwan. First I got to see the country solo, though not so much of it as I had planned. I landed on the northern end of the island at the same time a typhoon was busy pummeling the southern. My side trip to see the old city of Tainan was therefore canceled; instead I stuck to the northern third, where the storm turned out to be a nothingburger despite two canceled schooldays.  But a train ride down to Hualien and a car tour of Jiufen and Shifen on the eastern coasts satisfied my standing urge to escape the city.  And Taipei is marvelously comfortable for an American.

After the solo portion, I was wrapped in the over the top hospitality of the parents of the Taipei American School debate program, where I was wined, dined and regaled with about 81,102 toasts, though I confess I didn’t count. I confess that for whatever reason Taiwan has never been on my priority list of places to visit. But now it’s on my list of places to return to.

One of the things I did manage while there is to spend a whole day writing travel blog posts from earlier trips, so my sequence about last year’s Japan adventure will be posting once a week until done. Let’s see if I can get more written between the end of that and now.

No more big journeys for me until Thanksgiving, when I launch on a European Gallavant.

TABROOM

We had a brief downtime on Saturday, which thankfully happened just as I was waking up anyway in Taipei. The downtime has apparently caused a fair fury of speculative debugging around the socials, because the error messages indicated that a disk was out of space. But the lesson of this speculation is that errors can be misleading if you don’t have the full picture.

Some myths, debunked: Tabroom does indeed run in the cloud, not on a server we run ourselves. That fact isn’t so magic as you might expect: all “the cloud” means is someone else’s server. Cloud services are still subject to the resource limits any other server has. In particular, database servers are tricky to parallelize, or run on multiple machines, so we are not vulnerable to a single instance’s downtime. So while our web servers run 2 instances during the week and anywhere from 4-16 instances during the weekend, our primary database server remains singular. That limitation is from the core tech, and is not specific to Tabroom. So, we are stuck with it.

The root cause of this downtime was not insufficient disk space. Tabroom’s database takes up 36gb of space; that’s all your registered entries, ballot comments, and event descriptions rolled up into one mess of data. The database has its own dedicated disk, separate from the operating system and general server it runs on. That disk currently has 128gb of space; not the largest, but we pay for fast instead of big here, and it’s still 4x as large as Tabroom’s data needs. It’ll do fine for a decade, and we can expand it at will when and if we need to.

That was not the full disk you were seeing errors about.

Instead, a badly written query created years ago for a rarely used results page that experienced a sudden surge in popularity this weekend. That query failed to limit its scope: in order to calculate its output, it was pulling every ballot and ballot score in Tabroom. In 2016 when it was written, that made it slow but not particularly noteworthy. In 2024, every time someone went to that page, a 23 gigabyte temporary file was created on the server disk to run this one query.

At that point, it was only a matter of time: no server can indefinitely handle several hundred 23 GB files being dumped on it. At 4:30 PM CST, the disk hit its limit. Kaboom.

Fortunately, it was a simple matter, once I woke up, to clear the disk, fix the query, and kick the server. That’s life when you’re the sole maintainer of a project sometimes. I simply cannot go back and test and check every one of the thousands of queries that Tabroom regularly runs. When I do confront something like this, I do a review and put up guardrails around this exact thing happening again, but that only solves for the problems I’m aware of. Are there other ticking time bombs in the code?  Probably! Is this true of every other online service on earth?  Definitely!

But I promise you the issue is not that we haven’t found a good enough deal on enterprise disks, or the NSDA is being cheap on the hosting provider. We’ve been pretty lavish this year in terms of server resources, actually. But this particular problem would have blown up no matter how much overkill we’d built into our hosting setup; it was simple the result of the terribly common human errors. That’s what my job is. Consider that your typos at worst can insult someone, or temporarily hurt a student’s grade. Mine can bring down most of speech & debate. No amount of paranoid care can entirely prevent that, even though I do take quite a bit of it.

The worst part of it?  That results page with the query doesn’t actually work properly anyway; its formatting is broken. And since people are for some reason now fascinated by this page, they won’t stop emailing us. I’m going to put up a notice about that, but perhaps will just take it down. There’s little value in spending a week trying to fix this bad spaghetti code when I’m just going to have to rewrite it soon anyway; instead I’ll just move it up the list of things to be rewritten early.

The rewrite goes apace. Right now I’m working on standing up a testing framework which should very much help in finding bad queries before they go bad in production. Having a proper testing framework from the beginning of the rewrite reduces the chances that future changes will go back and hurt existing code without me knowing about it. But it also means I have to slim down that 36GB of data y’all have created over the years, to a set of data that is complete enough that I can test every scenario but doesn’t take 2 hours to load on the testing database. This work is drudgery, but invaluable, which is the worst kind of drudgery. But given the jetlag, it’s probably about what my brain is up for right now.

OTHERWISE

Fall is here!  It’s the best time of the year in New England, except that I’m allergic to it, and still jet lagged from Taiwan. But the days are also growing sadly shorter, which isn’t helping with the lag. I’m hoping to get up into the north country this weekend and spend some time in the outdoors crispness; I’m hoping it won’t be a total tourist mob scene in the White Mountains, but likely we’re past foliage peak up there anyhow.

This blog is and has long been hosted on WordPress. But recently, the WordPress project has decided to set itself on fire, thanks to an apparent hissy fit by the founder. He runs both the nonprofit that owns the open source code and update servers, and a for-profit hosting company, Automattic, built on the same software. He claims that a competing hosting company, WPEngine, isn’t giving enough back to the community and somehow abusing trademarks in a nebulous way. But WPEngine isn’t required to give anything back at all, and the trademark claims seem spurious to me; that was enough to raise my antenna. And then the founder leveraged control over the nonprofit to cut off WPEngine from the open source code and security updates. They took over a plugin created and maintained by WPEngine, and pushed out their own changes to it, as well as renaming it, under the guise of “security.”  This update would have auto-installed on thousands of blogs without the administrators thereof consenting to the change or even being aware of it.

That final bit crossed the Rubicon in my book; I no longer trust WordPress, and will therefore soon transition this site to another platform. Honestly, WordPress was never a perfect fit for me anyway, and because it is so common on the web it also requires a lot of security filtering; even my little blog suffers near constant hacking attempts. The most obvious alternative appears to be Ghost, which has the virtue of integrating in email subscriptions, so if you are one of the three people who regularly like to keep updated on my blather, you’ll have that as an option soon.

I don’t really want or need an additional side project, but so it goes sometimes in the world of open source.

September steals into the world with a lovely crisp week of New England fall weather, cool and perfect, but with that bright bright sun that we see all summer and miss all winter.  I went up to York, ME for Labor Day, giving me a chance to walk in the cooling breeze during sunset and wave goodbye to summer, a stolen tradition but one I quite like.

And now we have the reality of the approaching school year.  Boo, hiss.

Tabroom

I am crawling to the bitter end of the list of things I have to do with Tabroom before I can feature freeze it.  I had hoped to finish that before leaving for a sojourn with Mock Trial folks last month but it was fantasy, as ever.  My last challenge is to build a front end that can take advantage of Tabroom living in the cloud now, which will let people other than me autoscale the power of our installation upwards if the service is lagging.  Right now we can scale it up, but the process is picky and technical which means only Hardy and I can do it, and as anyone in front line support will tell you, you need at least 3 people for 24/7 coverage.

So once this is done, I can permit others in the NSDA hit the “More power!!!” button when there is a slowdown.  The process of programming it is quite tedious, however.  One major requirement is making sure that folks without a programming background can understand the nature of the problem before hitting buttons that will cost us a lot of money.  There are times when there are Tabroom slowdowns for individuals that aren’t actually server overloads — their local internet is having trouble, or the provider’s is.  A system can report load metrics to tell you if they’re struggling and why — but these are a little arcane and hard to read, and stored in multiple locations.  So part of this task is me having to read badly formatted data from six different sources and present it to a colleague in my department such a that an intelligent non programmer can understand and act on it.

That job is nitpicky and tedious, and prone to look right when in fact it is wrong.  When you’re trying to sift through a dozen bits of data that are all decimal numbers between 0 and 1, and you pick the wrong column, it still appears okay unless you check it very carefully.  It’d be easier if the wrong answers were all 439,981 when the right answers were 0.31.

But the nice news is I’ve been able to write more of this backend in NodeJS and not increase my rewrite woes yet further on the cusp of being able to focus on it exclusively.

I’m also working on a pretty comprehensive set of documents for Tabroom aimed at Mock Trial usage.  It’s coming along well, though it is reminding me that we really do need to show some love to the docs for Speech & Debate usage as well. I’m hoping I can actually use some of this MT stuff to help out S&D.  Two public speaking activities, helping each other.

Tournaments & Travels

My slimmed down schedule includes two this month:  the Kentucky Season Opener on 9/7 weekend, and the Jack Howe Memorial at CSU Long Beach on 9/28 weekend.   And then, in more distant and exotic news, I depart for Taiwan, partly for the Taiwan Speech & Debate Invitational on 10/12 weekend, but also for a week of seeing what the island has to offer first hand.  I admit Taiwan has never been high on my travel radar before, because I didn’t know much about it.  I sat down on a long flight last week to do some reading, and it took me exactly one blog post to go from “How should I fill my time there?” to “How on earth can I narrow this list down so it’s manageable?

Otherwise this is also the stunning time of year when New England gets to lord our superior weather over the rest of the world.  It doesn’t happen often, so we tend to grab it with both hands when it does.  The humidity blows out into the ocean, taking the bugs with it, and then the leaves turn bright, and I start moseying northwards to the forests more often.

Writing

I haven’t done squat with the eight or so ideas I have for a travel blog that people keep pestering me about.  But I’ve written three full chapters of this book I’ve been toying with.  I’ll likely never have the gumption to share it with anyone else, but it’s been edifying practice to write it out, and it gives me an excuse to put the coding linter down sometimes.

I made a clipboard to write with out of purple heartwood that came out decent. No photos yet, and I suspect I made it a touch too thin and it’ll warp, but as a first shot working with a new hardwood, I’m decently pleased.  If it does warp, I’ll try a layered version next and see.

I have returned from Alaska; and therefore have finished the 50.  Almost nothing I booked in advance went as planned, thanks to a forest fire in Fairbanks, followed by another in Denali, then drenching rain for a week in Kenai, and finally a tweaked knee in Wrangell-St Elias that finally prompted an return a few days early.  But I still had an excellent time, and it so happened that my early return meant I dodged the great Crowdstrike outage.  My original flights would have left me stranded in Seattle for days.

It did mean I didn’t do half the hiking miles I wanted to.  On returning, I have so far controlled my impulses to run off to Tuckerman’s Ravine or Mt Katahdin to make up for it, but it’s only a matter of time.

METABLOG

This WordPress install actually got hacked. Someone was careless with some passwords, or was running an outdated plugin or four or something on my tiny server, so all of a suddenly my humble little 3-hits-daily blog was spewing spam and CAPTCHA attempts out, and generally being a bad citizen.  So I spent a lot of time cleaning that up and deepening my defenses.  But now I feel more invested in it, I suppose.

TABROOM

I have a couple of weeks of things to do before I can then take a deep breath and freeze the introduction of any new features into Perl/Mason legacy code, and start working exclusively in JS frameworks.  It hardly feels real, and it doesn’t help that most of the tasks I do have between then and now are rather tedious and annoying cleanups of very old code.  It’s a hard thing to do in July and August, when New England can just stun us with clear light stabbing everywhere through an ocean breeze.  I am making progress, but it’s requiring a few pulls of the chain to get the old motor running again.

These remaining tasks are mostly Nationals specific nitpicks of no use or concern to anyone outside the NSDA, but I am hoping to get the notification system working better in a Tabroom native way (which means fewer adblocker issues) and running on iPads and iPhones better.

NEXT MONTH IN WHERE THE HELL ARE YOu

I’ll be spending a week in Mendocino because I can.  This time I might actually have company there.  I’ve never been able to show the place off to anyone else, despite a lot of trying.  Then I’ll cap that trip with a meeting with the Mock Trial folks in LA.  Otherwise I’m sticking to the northeast for the rest of the summer.

NON TABROOM TYPING

I’ve been writing a lot but the chances I post whatever it is I’m tinkering with right now are really low, to be honest.  However it’s been interesting enough that I’ve found a couple nights I look up feeling ‘a bit tired’ and realize it’s 2AM and perhaps I should sleep instead.  I started working on my tablet with a keyboard attached to it, instead of the open distraction festival that is my laptop.  It’s worked tremendously.

I have written two of the required six posts about travel that I’ve set as my threshold for doing more travel blogging, though, so there’s at least some hope there.

READING

I’ve finished the 3rd Strong Towns book, and remain a fan of the organization and its goals.  I found them via the podcast, but as they’ve grown that’s become a much smaller part of their media footprint of late.  I re-read Piketty’s Capital because I felt like being a nerd, and Faulkner’s Light in August because I felt like being a different type of nerd.

I also read two books of Sanderson’s Stormlight Archive series, but then I looked up how many there were left, and it turns out he’s produced 4 of a planned ten volumes.  I decided to put the series into park. I found the series to be engaging enough but plodding in terms of major plot points.  I think authors in that genre have to put a lot of work into building a World, but then they become enamored of the world, or exhausted by the amount of work it took to create. So instead of telling their story with a full start and ending, they end up just living there.  Their outlined story inflates with filler, and the spaces between actual events grow and grow.  Sooner or later you’re reading an account of the main character shopping for dinner ingredients. Or, you know things that the characters don’t, and grow tired of waiting for them to catch up.

So I’m putting that down and going to reach for the classics pile again, especially now that driving around thousands of miles of Alaska is no longer on the agenda and a 45 hour audio book isn’t so appealing.  I have a copy of some Cather, who I’ve read only once and enjoyed.

KILLING SOCIAL MEDIA

It struck me a little recently how much life is different in 2024 for me than any other year because I no longer have a serious social media outlet.  I was one of the first 1,000 users on Facebook, back when it was only open to those with a @harvard.edu address.  Once I finally axed the Facebook account in 2017 I went through brief phases of being active in various corners of Reddit, and then more on Twitter, especially during early Covid when it was the best route directly to some of the under-reported actual experts on respiratory viruses and current research.  But then Twitter became Xitter (pronounced Shitter) and along the way fired my sister, so I killed my account and do not touch the site anymore.

Folks keep sending me links to Xitter. I never click on any of them, but they keep coming.

I do post stuff to Instagram from time to time, mostly as a heartbeat to people I don’t keep in regular contact with.  And I tinkered around with Bluesky and Mastodon for a while.  But I’ve come to realize that they were like the raw sugar candy of communication: empty calories that concealed the nutrient deficits they were creating for me.  It felt like I’d interacted with people in mass, but truly I had read eighty-seven quick bad hot takes.

They took a silent toll.  I’ve never been notably anxious; my mental curses tend to run more towards slowness and lethargy, not spinning around rapidly.  But online social discourse is drenched in terror right now.  Are there things to fear in the present political moment? Yes. Are they so omnipotent and unavoidable that we should ban optimism altogether? No. But engagement sells, and fear drives engagement, and now that’s all you find.  And even I found it was damaging my calm more than the underlying reality does.

So, I started muting the most conspicuously doomerish on Bluesky and Mastodon.  After a while I found both of them were growing rather silent. And after a time I stopped checking either altogether.  And now, neither lives on my phone.  And almost by magic I find myself more productively writing things that matter, not little zings about the latest political event that will affect nothing at all and persuade no one of anything.

I think a lot of debate people probably end up with a parasocial relationship to politics: the actual agents do not know or care we exist, but we imagine that if we have the 2024 equivalent of Josh Lyman’s four TVs going at all times that our finger is on the pulse. It becomes a part of folks’ personality to know what’s going on instantly.  People felt the need to text me immediately after the shooting attempt at the Pennsylvania rally for the orange creep, or when Biden dropped out of the race, as if it matters that I know these things within the hour, or even the week. As it happens Alaska is a big wild land with a lot of gaps in cell phone coverage, and I learned about neither piece of news quickly.  And it did not affect either my life, or how either story unfolded.

And that speed makes people vulnerable to their own narratives. Without time to deliberate the specifics of an event before the next one comes, you’re at the mercy of your cognitive biases and shortcuts, who are so often false friends.  It’s important to be hostile to your own beliefs. Before allowing a concept into your mind, first subject it to deep scrutiny, to make sure it’s worthy.  And then go back and re-open old ones too, and make sure they’re still current. I’ve turned pretty radically from one conclusion to another over the years on a number of fronts, but I think it’s part of how I remain myself.

And so it was with the shooting; it infected even Instagram, which I keep because it’s supposedly limited to pretty pictures.  Suddenly there were walls of text assuming this event had sealed the doom of all civilization; presumably because the jerk would win the election as a result, or something?  The mechanism of the doom is never clear, just the feeling.  Just who would change their vote because someone took a shot at the guy and missed?  How was this landslide going to start rolling?

And in the event, a week later Biden dropped out of the race, and the world changed again. Nobody remembers the event that doomed the world.  There’s a lot of events that “mean we’re all screwed!” that nobody remembers anymore. Like a lot of politics, it was mostly ephemera.  And while I can name a lot of books that changed me,  I can name zero tweets that have done the same.

Tabroom

The season is officially over as of last Friday night.   I got home late Saturday and slept for the next 14 hours.

The Tabroom Inbox is an unmitigated hit.  15,323 users have accessed it since I rolled it out without announcement on Friday May 24th, a month ago today.  There haven’t been very many tournaments since then, though we have held two of our largest.  People are finding it and using it.  So hurrah for that.

On Downtimes & Slowdowns

I thought I had oversupplied Tabroom with compute capacity, but we were hit with two slowdowns during Nationals anyhow.  Tuesday midday we just had too much traffic for the machines, so I doubled the memory on the existing ones and spun up two more.  It was overkill, but it worked.

Monday morning, it slowed down but that had nothing to do with capacity.  Sometimes database queries do fine when they return 2,000 records for normal tournaments, but slow down out of proportion when they return 20,000.  Or, 200,000.  On Monday, we discovered 4 such queries, and each of them was being hit dozens of times.  The most significant was the public list of judges at the tournament & their paradigms.  The query behind that page usually takes .04 seconds.  At Nationals it required 2 minutes.

2 minutes sounds like not much time, but in terms of a computing operation it’s deadly slow. The average query to the Tabroom database takes milliseconds.  A 2 second query is usually unacceptable by my standards, unless it runs rarely and is very complex.

A 2 minute query also avoids another capacity-saving technique.  Public facing pages that don’t change often, like that judge list, will be calculated and laid out on the server, with the code for your browser sent off at the end.  For many pages, the server will then save that resulting display code in a cache.  The next person to ask for that page just gets the same display code without the intermediate computation having to be redone. Everyone so often, depending on the page, the cache is invalidated and the page is recalculated on the next request, but for pages that don’t change often like a judge list, that’s usually just once every 30 minutes or so. We save a lot of server capacity and cost that way.

However, if the first request comes and takes 2 minutes to complete, then the next request, and several hundred after that, come in before the cached result is ready.  So they too are sent to the database instead.  A bunch of 2+ minute queries hitting at once become a problem real fast. Soon, the database server gets bogged down with hundreds of these difficult and long running queries, and has few resources left for more ordinary data requests, and the site eventually goes kerplop.

That’s a technical term.

We also  went from using 2 giant machines to a bunch of smaller machines.  That helps us in a lot of ways — when one machine gets knocked out by a hardware fault, we lose less of our total capacity. But,  each machine has its own cache. So instead of an inefficient query hitting 2 times per cache period, they now run 16 times.

For next season I’m on a mission to make sure that you and I do not have to deal with this type of problem again.  I’m going to measure out what the traffic at each slowdown was and what type of machine profile was necessary to meet the demand.  In other words I’ll be revising upwards my idea of what “overkill” means.  Also, a lot of this mess is because we’re still largely using legacy code that takes a lot of memory and CPU to run; the more progress I make replacing it with rewritten JS code the faster things will be with the same compute power.  And finally Hardy has reconfigured our gateway machine to redirect traffic away from Tabroom when the servers are overloaded.  That will help us recover faster from these types of things when you all start refreshing the page every heartbeat; most of those requests will fail, but the ones that do get through will be served properly and we’ll be better able to log in and fix things.

Next Season

So next year I’m no longer tabbing any NDT/CEDA tournaments, and will prune out a fair few others that will be fine without me. It’s difficult to choose which ones to skip, since I don’t tab tournaments for people I dislike.  So I must tell people I like that I’m out.  I do enjoy travel and interacting with the community like that. But I can’t sustain what I’ve done this year, and focus on the long term needs of Tabroom and the code rewrite.  I’m also concerned about my health. I’ve gained back enough of the weight I lost during Covid to be alarmed by it, and this summer I need to reverse that trend.

We’ve also dropped a lot of the sustainability lessons I had thought we’d learned around the pandemic.  Tournament schedules start too early, in the face of a growing mountain of evidence about how badly that affects teenagers and young adults, much less me.  You’d think we’d value the students thinking at their best to get the most out of debate, but instead most tournament directors let their anxiety about what might happen at the end of the day push them into earlier starts.  I don’t truly understand that; in order to make sure we don’t stay to late and lose sleep, we therefore make everyone get there too early and … lose sleep?

And for me, when I’m tired, I have much less willpower to order the salad instead of the burger.  The tired also lasts longer; when I get home from the west coast it can take days until I right my schedule, and by then I’m off on another jaunt.

Maintenance Mode for Tabroom Legacy Code

By August 1st or thereabouts, Tabroom’s legacy Perl code will be declared maintenance only.  That means I will only apply bug fixes and updates to the existing legacy Tabroom code, and will not be coding any new functionality into legacy Tabroom but instead will focus exclusively on the front end for the new system in Javascript & Svelte.  New features will wait until they can be deployed in JS.  That will require focus, discipline and lot of saying no, but it’s the only way we’ll get over this hump of deploying new code while maintaining the system.

This rewrite has been on our horizons for a long long time, and it’s been derailed and delayed by a lot of circumstances, from our failure to work out the original collaboration plan well, to the pandemic.  And it would not have been easy even without those challenges.  Writing a new system while maintaining its predecessor alone is extremely hard, to the point that many such efforts fail.  However, we have made progress behind the scenes, and now we’ve about reached the limits of what I can do with adapting the system over seamlessly.  So now, the seam.

Non Professional Quick Hits

• I missed the Celtics parade for their 18th title while at Nats.  I sent the boss an invoice for one championship parade.  I know he can’t fulfill that.  I just wanted it on the record.  By the looks of this team, we may be parading banner 19 soon enough anyway.
• I have planned out my annual July walkabout.  This year: Alaska. I leave in a few days and am starting to pack.  I’m going to be on a lot more guided tour type things this trip than I usually do; more often I prefer to have zero plan and move each day as the spirit hits me.  But in a place as vast and wild as Alaska, where rental car companies are very clear that you may not take their vehicles to some of the best spots, I am erring on the side of the professionals for this one.  Next trip, I’ll go more independent.
• My brother in law’s mother is having some health struggles. Send some positive vibes northwards to Maine. I’m not one for much belief in the ephemeral like that, but sometimes it’s all you can do.
• I may have accidentally bought a new super-telephoto lens for my camera that I don’t actually need for the trip.  Oops.  That said since I’m just a hobbyist I don’t feel the need to buy the huge, expensive Canon lenses with the magical red ring.  That color describes your bank account balance is if you buy too many of them.  Honestly, I prefer the ring-less cheapo line for my hiking and ranging anyway; they’re far lighter and easier to carry, and it’s less terrifying if you drop them.
• I somehow found time to read.  I finished the Strong Towns Housing book, a re-read of the excellent Tony Judt history of postwar Europe, a takedown of the historicity of constitutional originalism by friend-of-friend Jonathan Gienapp, and some fiction that I wouldn’t especially recommend.  Looking for audiobook suggestions now for some long Alaska drives.
• This summer I’m hoping to get serious about taking the massive pile of photos I’ve taken in the last few walkabouts and pulling out the dozens that actually have a sense of composition and style to them to display in a more prominent manner.  But I have to say, even if I never look at them again, the process of finding photos has pulled me into places I appreciate being for their own sake.
• Late summer I’m landing in LA for a Mock Trial workathon and have used the excuse and coast-to-coast plane ticket to slide in a side trip to Mendocino the week before.
• I’ve written about half of a book that I’ve been working on for a while.  It’s all handwritten right now.  I have no idea what I’ll do with it when I’m finished, but I think I owe it to the stuff to at least type it up and finish it, even for just myself.  I’m contemplating whether I should go rent a cabin somewhere woodsy and not let myself come home until I’ve focused a little more on it.
• I’ve had six people in the last three months proactively tell me I should start a travel blog.  I’ll consider doing it if I can manage to sit down and crank out six entries first.  But it’s not a terrible idea.