A running log of what Palmer is up to right now, if you’re curious. Last edit on 6/24/2024.
Tabroom
The season is officially over as of last Friday night. I got home late Saturday and slept for the next 14 hours.
The Tabroom Inbox is an unmitigated hit. 15,323 users have accessed it since I rolled it out without announcement on Friday May 24th, a month ago today. There haven’t been very many tournaments since then, though we have held two of our largest. People are finding it and using it. So hurrah for that.
On Downtimes & Slowdowns
I thought I had oversupplied Tabroom with compute capacity, but we were hit with two slowdowns during Nationals anyhow. Tuesday midday we just had too much traffic for the machines, so I doubled the memory on the existing ones and spun up two more. It was overkill, but it worked.
Monday morning, it slowed down but that had nothing to do with capacity. Sometimes database queries do fine when they return 2,000 records for normal tournaments, but slow down out of proportion when they return 20,000. Or, 200,000. On Monday, we discovered 4 such queries, and each of them was being hit dozens of times. The most significant was the public list of judges at the tournament & their paradigms. The query behind that page usually takes .04 seconds. At Nationals it required 2 minutes.
2 minutes sounds like not much time, but in terms of a computing operation it’s deadly slow. The average query to the Tabroom database takes milliseconds. A 2 second query is usually unacceptable by my standards, unless it runs rarely and is very complex.
A 2 minute query also avoids another capacity-saving technique. Public facing pages that don’t change often, like that judge list, will be calculated and laid out on the server, with the code for your browser sent off at the end. For many pages, the server will then save that resulting display code in a cache. The next person to ask for that page just gets the same display code without the intermediate computation having to be redone. Everyone so often, depending on the page, the cache is invalidated and the page is recalculated on the next request, but for pages that don’t change often like a judge list, that’s usually just once every 30 minutes or so. We save a lot of server capacity and cost that way.
However, if the first request comes and takes 2 minutes to complete, then the next request, and several hundred after that, come in before the cached result is ready. So they too are sent to the database instead. A bunch of 2+ minute queries hitting at once become a problem real fast. Soon, the database server gets bogged down with hundreds of these difficult and long running queries, and has few resources left for more ordinary data requests, and the site eventually goes kerplop.
That’s a technical term.
We also went from using 2 giant machines to a bunch of smaller machines. That helps us in a lot of ways — when one machine gets knocked out by a hardware fault, we lose less of our total capacity. But, each machine has its own cache. So instead of an inefficient query hitting 2 times per cache period, they now run 16 times.
For next season I’m on a mission to make sure that you and I do not have to deal with this type of problem again. I’m going to measure out what the traffic at each slowdown was and what type of machine profile was necessary to meet the demand. In other words I’ll be revising upwards my idea of what “overkill” means. Also, a lot of this mess is because we’re still largely using legacy code that takes a lot of memory and CPU to run; the more progress I make replacing it with rewritten JS code the faster things will be with the same compute power. And finally Hardy has reconfigured our gateway machine to redirect traffic away from Tabroom when the servers are overloaded. That will help us recover faster from these types of things when you all start refreshing the page every heartbeat; most of those requests will fail, but the ones that do get through will be served properly and we’ll be better able to log in and fix things.
Next Season
So next year I’m no longer tabbing any NDT/CEDA tournaments, and will prune out a fair few others that will be fine without me. It’s difficult to choose which ones to skip, since I don’t tab tournaments for people I dislike. So I must tell people I like that I’m out. I do enjoy travel and interacting with the community like that. But I can’t sustain what I’ve done this year, and focus on the long term needs of Tabroom and the code rewrite. I’m also concerned about my health. I’ve gained back enough of the weight I lost during Covid to be alarmed by it, and this summer I need to reverse that trend.
We’ve also dropped a lot of the sustainability lessons I had thought we’d learned around the pandemic. Tournament schedules start too early, in the face of a growing mountain of evidence about how badly that affects teenagers and young adults, much less me. You’d think we’d value the students thinking at their best to get the most out of debate, but instead most tournament directors let their anxiety about what might happen at the end of the day push them into earlier starts. I don’t truly understand that; in order to make sure we don’t stay to late and lose sleep, we therefore make everyone get there too early and … lose sleep?
And for me, when I’m tired, I have much less willpower to order the salad instead of the burger. The tired also lasts longer; when I get home from the west coast it can take days until I right my schedule, and by then I’m off on another jaunt.
Maintenance Mode for Tabroom Legacy Code
By August 1st or thereabouts, Tabroom’s legacy Perl code will be declared maintenance only. That means I will only apply bug fixes and updates to the existing legacy Tabroom code, and will not be coding any new functionality into legacy Tabroom but instead will focus exclusively on the front end for the new system in Javascript & Svelte. New features will wait until they can be deployed in JS. That will require focus, discipline and lot of saying no, but it’s the only way we’ll get over this hump of deploying new code while maintaining the system.
This rewrite has been on our horizons for a long long time, and it’s been derailed and delayed by a lot of circumstances, from our failure to work out the original collaboration plan well, to the pandemic. And it would not have been easy even without those challenges. Writing a new system while maintaining its predecessor alone is extremely hard, to the point that many such efforts fail. However, we have made progress behind the scenes, and now we’ve about reached the limits of what I can do with adapting the system over seamlessly. So now, the seam.
Non Professional Quick Hits
- I missed the Celtics parade for their 18th title while at Nats. I sent the boss an invoice for one championship parade. I know he can’t fulfill that. I just wanted it on the record. By the looks of this team, we may be parading banner 19 soon enough anyway.
- I have planned out my annual July walkabout. This year: Alaska. I leave in a few days and am starting to pack. I’m going to be on a lot more guided tour type things this trip than I usually do; more often I prefer to have zero plan and move each day as the spirit hits me. But in a place as vast and wild as Alaska, where rental car companies are very clear that you may not take their vehicles to some of the best spots, I am erring on the side of the professionals for this one. Next trip, I’ll go more independent.
- My brother in law’s mother is having some health struggles. Send some positive vibes northwards to Maine. I’m not one for much belief in the ephemeral like that, but sometimes it’s all you can do.
- I may have accidentally bought a new super-telephoto lens for my camera that I don’t actually need for the trip. Oops. That said since I’m just a hobbyist I don’t feel the need to buy the huge, expensive Canon lenses with the magical red ring. That color describes your bank account balance is if you buy too many of them. Honestly, I prefer the ring-less cheapo line for my hiking and ranging anyway; they’re far lighter and easier to carry, and it’s less terrifying if you drop them.
- I somehow found time to read. I finished the Strong Towns Housing book, a re-read of the excellent Tony Judt history of postwar Europe, a takedown of the historicity of constitutional originalism by friend-of-friend Jonathan Gienapp, and some fiction that I wouldn’t especially recommend. Looking for audiobook suggestions now for some long Alaska drives.
- This summer I’m hoping to get serious about taking the massive pile of photos I’ve taken in the last few walkabouts and pulling out the dozens that actually have a sense of composition and style to them to display in a more prominent manner. But I have to say, even if I never look at them again, the process of finding photos has pulled me into places I appreciate being for their own sake.
- Late summer I’m landing in LA for a Mock Trial workathon and have used the excuse and coast-to-coast plane ticket to slide in a side trip to Mendocino the week before.
- I’ve written about half of a book that I’ve been working on for a while. It’s all handwritten right now. I have no idea what I’ll do with it when I’m finished, but I think I owe it to the stuff to at least type it up and finish it, even for just myself. I’m contemplating whether I should go rent a cabin somewhere woodsy and not let myself come home until I’ve focused a little more on it.
- I’ve had six people in the last three months proactively tell me I should start a travel blog. I’ll consider doing it if I can manage to sit down and crank out six entries first. But it’s not a terrible idea.