People catastrophically underestimate how much the world starves for software; even a plain, simple CRUD app and a few for loops could significantly increase the planet-wide utility function and make humans happier. Even the ugliest piece of code with 4 nested for loops and 3 nested if statements can end up upstream of something like $3 M in net present value or $7 M in lifetime earnings if it changes which labor market a stellar student gets to compete in.

You want to see how I got those numbers? Buckle up.

What is arbitration?

In high school, I participated in chemistry olympiads, which basically were an alchemical athanor in which all my current work ethic skills were forged. Point being, they were quite instrumental in shaping my career, so it’s not surprising that after I graduated I wanted to give back and I ended up helping with national olympiad organization in Kazakhstan for ~5 years.

For those not familiar, here’s how a typical final stage of the national olympiad looks like: first day you write a theoretical round (solve a bunch of problems), second day you chill, third day you do experimental round, and fourth day you participate in arbitration: a session when you can come to the jury who graded your work and argue that you should get more points than you have received.

A common misconception among people not familiar with olympiads is that arbitration is not necessary if you have a sufficiently competent jury; that it’s only needed if you worry that the members of the jury will make mistakes. However, even the most competent professor (who are often invited to the jury) might misgrade a work if basically he has a day to go through 150 works, usually grading 3-4 open-ended problems with multiple subparts where you don’t just give points for the correct answer; you also have to check if there’s good reasoning (and many problems can be solved in slightly different ways). But even if somehow your jury gets supernatural powers and makes no mistakes in grading, a student can come and easily argue his way for as much as 3 percentage points. How?

See, when you design a problem, you suffer from the bias of knowing the correct or intended solution. So the most you can guarantee is that your solution is logically correct, every reasoning step is justified and supported by the immediately preceding ones. But unless you put yourself in the condition of seeing the problem for the first time under time constraint, you have no idea if there are other reasonable ideas someone can come up with and get railroaded into a different solution that satisfies 70-80% of your problem, and so a student gets stuck at the remaining 20% not realizing that he’s on the completely wrong branch of the solution space.

And if you happen to be such a student, you might easily argue “how could I have known that at this branching point in logic I was supposed to do X and not Y? the problem is underspecified.” Which is why for many olympiad participants arbitration is one of the most memorable and fun parts of the whole process. It starts at 9am and can easily continue all the way until 10pm. Members of the jury usually sit in the same room until every single student who wanted to talk to them has had a chance to do so (which, when combined over all problems, might easily add up to 30-60 min per student).

Arbitration in the age of COVID

In 2022, even though COVID was on the decline, the national olympiad was still happening remotely. As such, olympiad organizing committee (which is a bunch of bureaucratsto be fair, some of them were good people from the Ministry of Education) came up with a brilliant idea: to avoid chaos, let’s require students to fill some Microsoft form indicating which problems they want to arbitrate the day before arbitration,so technically before they even see their scores, which usually are posted the morning of arbitration simply because they’re not ready earlier and then create a rigid schedule, which, given the number of students, amounted to 5 min per participant.

Like hop on a Zoom from 9:35 to 9:40am, you talk to the full jury committee at the same time, and you have to do that from a “special anchor school”whatever the fuck that means. Mind you, different jury members grade different problems, so there’ll be 5 jury members who will sit idly and do nothing while a student argues with one particular jury member. Why not allow separate schedules for separate jury members? Only because it’d be too much work to create such a schedule. After all, you have to read through the form, do a mapping between students and jury members, and then try to create a conflict-free schedule. And the arbitration schedule has to be sent to the participants in 2.5 hours, so yeah, suboptimal, but WHAT ARE YOU GOING TO DO?

Well, I gather you understand how bad of an idea this is given my extensive aside into how arbitration is supposed to work, so I, only having some elementary coding training (by that time I had only taken 6.009 at MIT, i.e. before even my first algos class), decide to speedrun a simple script to parse the form, do the mapping, create the schedule, and export it as an Excel file that organizers can share. I managed to finish in those exact 2.5 hours, here’s the code; nothing fancy, I didn’t do any smart bipartite graph coloring approaches, no, it’s a brute-force 4 nested for loop check for conflict:

def _is_there_a_conflict(self, data):
    for grade, juryToStudents in data.items():
        for jury, students in juryToStudents.items():
            for i, student in enumerate(students):
                for jury2, students2 in juryToStudents.items():
                    if jury2 != jury:
                        if len(students2) > 0 and i < len(students2):
                            if students2[i] == student:
                                return True
    return False

and a simple “move the conflicting student to the end of the queue”

students = students[:i] + students[i + 1:] + students[i : i + 1]

I run the script, get the Excel file, and send it to the organizers and ask them to let us do it this way. It probably did help that I had earned some good reputation and it definitely helped that the ultimate people in charge were rational, so they said ok, but you take full responsibility. Fine.

If you wonder, arbitration went remarkably well, we haven’t had to turn around a single student, we had a publicly facing live Excel sheet that jury members updated to indicate who was currently in the Zoom room, who’s in the waiting room, and who’s done. We didn’t impose a time limit on our interactions, and yet we were so efficient we finished early and so gladly accepted arbitration from people who didn’t fill the formsome of them didn’t know they were supposed to.

A 20-million-dollar bureaucratic decision

The difference between a gold and a silver medal that year was 1.02 pp in 11th grade and 1.5 pp in 10th grade. A gold medal gets you invited to the national camp. A silver medal doesn’t. If you get invited to the national camp, you participate in the selection process for the Mendeleev olympiad. Which opens the door for selection to IChO (the ultimate international olympiad). IChO is an incredibly powerful signal of your competence to any self-respecting university. You can’t say that an IChO medal increases your chances of admission to MIT, but it’s definitely true that the kind of people who get a medal at IChO are more likely to be the kind of people that have a good chance of being admitted.

Let’s do a rough estimate of the difference in career outcome for a silver and a gold medal. For an upper bound of the delta, we can assume the gold medalist gets into MIT (or another top-tier US college), which opens a path to a career in big tech, biotech, or finance, while a silver medalist goes to a local university and has a strong career at home.

The fourth column is anchored at MIT’s published $145,820 average starting salary for graduates entering industry, while the fifth is an estimate for someone who lands in top-end US tech or finance.See MIT’s return-on-investment page . The other trajectories are illustrative back-of-the-envelope paths in nominal pre-tax USD.

Lifetime earnings are obtained by adding up yearly wages, with linear interpolation between those milestone years, and to get a net present value (NPV), we have to discount future dollars:

Compared to “strong Kaz,” the “MIT / strong US” outcome results in a gap of about $7.25 M in lifetime earnings and $2.96 M in NPV, while the “elite US tech / finance” outcome pushes that to about $19.34 M in lifetime earnings and $7.20 M in NPV.

And you only get even a chance of playing for numbers like that if you get invited to the national camp, and that might end up depending on whether you get a chance to defend your work in arbitration and get the 2 pp you deserve (which would move you from the silver category to the gold).

A simple mundane bureaucratic decision to impose an artificial time limit and cram all juries into a single Zoom meeting, purely because nobody could manually create individual schedules fast enough, can literally sit upstream of a multi-million-dollar fork in the lifetime earnings of a 17-year-old.

Every single scalable societal change is bottlenecked by code

If you follow my recent work, you might find that I have a particular OCD manifesting in the creation of websites. For example, I wrote an academic review article and couldn’t help but accompany it by an interactive portal with key ideas, reading paths, interactive tables. Or I wrote an academic paper establishing a new benchmark and accompanied it by a web portal with leaderboards, inspection of all routes and comparison to reference experimental routes. And it’s not just downstream of me having access to LLM coding tools; I learned to create websites even before LLMs.

See, back in 2020, I decided it’s not worth wasting a year of my undergrad on remote studies, so I took a leave of absence and started a nonprofit in Kazakhstan aimed at creating resources that help students who are interested in learning science beyond the curriculum. I quickly discovered that pretty much anything scalable I wanted to do required writing code.

• You wrote a comprehensive guide on olympiad prep and you want to disseminate it? You need a website. Sure, you can use WordPress, and for a single blog post it works, until…
• You also think it’d be a fantastic idea to create a central source of past olympiad problems, and when you have thousands of them, if you want good user experience and curator experience, you pretty much have to write a custom website.
• Oh, and wouldn’t it be nice to have an archive of olympiad results, so that students 5 years from now can look up how many points they needed to score to get a certain medal. Again, a website.
• Oh, you want to create some low-stakes but realistic practice of an olympiad and you wrote some problems? Cool, if you want to grade stuff efficiently, you need software. Gradescope is a good example, and it was an extraordinary quality of life improvement, until the COVID promotion ended and you had to pay $20 per seat per month, and your only source of revenue was $200 donations each month (which is actually a lot for a grassroots nonprofit in Kazakhstan).

I was coming up with brilliant ideas every week, and every single one of them ended up being a message to a full-stack web dev guy, who was taking care of some of our projects in his free time. The good and bad part at the same time is that he was a great web dev: he had no shortage of options to earn incredulous (even for an adult) amounts of money creating websites, so he didn’t have much free time, and every hour spent on creating websites for us was incurring a real opportunity cost.

Basically, even though we were an educational nonprofit, every. single. initiative. was bottlenecked by code. Can you imagine how much more we could’ve done if we had LLM agents back then?