*A joint reflection from the PSF Infrastructure team* In 2022, I (Ee) was the only person who knew how to restart PyPI's database if it went down at 3am. That's not a brag—it's a confession. The Python Software Foundation's infrastructure had grown organically over two decades. PyPI alone serves billions of requests monthly. And for most of that history, "the team" meant whoever had SSH access and was awake when something broke. Jacob joined the PSF as Infrastructure Engineer in July 2024. His first week, he asked where the runbooks were. I laughed. He didn't. ## The state of things Let's be honest about where we started: Our monitoring was a mix of Datadog checks and hoping someone on Slack would notice if something looked off. Incident response meant "Ee's phone rings." Documentation existed in scattered Google Docs, half-finished wikis, and my head. We run Cabotage—our in-house PaaS on Kubernetes—alongside a Salt stack managing DigitalOcean VMs. Some of those VMs have been running since Ubuntu 14.04. The mail server is one of them. Nobody wants to touch it because migrations are terrifying when you're the only person who understands the system. We had volunteers who wanted to help. But onboarding them meant hours of screen-sharing, explaining context that should have been written down. Most burned out before becoming effective. ## What we actually built The first real change was boring: we wrote things down. Not a grand documentation initiative. Just a rule—if you fix something, document how. If you make a decision, write a one-pager explaining why. The runbook format wasn't sophisticated: ```markdown # Runbook: PyPI Database Failover ## Prerequisites - Access to AWS console (infra-admin role) - VPN connection established - Slack channel #infra-incidents open ## Steps 1. Verify primary database is unresponsive 2. Check replica lag: SELECT pg_last_wal_replay_lsn() 3. Promote replica via CloudNativePG ... ``` Within six months, we had runbooks for Cabotage, the Salt stack, DNS, SSL renewals, and most of the legacy systems. The on-call rotation took longer. We implemented PagerDuty with a simple alternating schedule—Jacob and I trade weeks. Clear escalation paths. No more "whoever's awake." | Level | Response Time | Who | |-------|---------------|-----| | P1 | 15 minutes | On-call primary | | P2 | 1 hour | On-call primary | | P3 | Next business day | Team queue | The hardest part wasn't technical. It was convincing ourselves that "good enough" was acceptable. For years, the culture had been: if you see a problem, you fix it, no matter the hour. That's unsustainable. We had to learn to file tickets for tomorrow. ## The team today We went from one person to a real engineering team: - **Director of Infrastructure** ()—team lead, partner relationships, PyCon US tech, and still in the on-call rotation - **Infrastructure Engineer** ()—Salt, Kubernetes, DigitalOcean, and keeping python.org/docs.python.org running - **Security Developer in Residence** ()—CPython security roadmap, vulnerability coordination - **PyPI Safety and Security Engineer** ()—malware detection, supply chain security, package review - **PyPI Support Specialist** ()—user support, account recovery, abuse handling Funding comes from multiple sources. supports our security roles. The PSF core budget covers infrastructure and support. It's a patchwork, but it works. ## The systems we maintain Infrastructure isn't glamorous. Here's what we're responsible for: **** (our in-house PaaS): Runs on Kubernetes with HashiCorp Vault for secrets. Hosts PyPI, python.org, us.pycon.org, and a dozen other services. CloudNativePG handles PostgreSQL. ****: Configuration management for our DigitalOcean fleet. mail.python.org, docs.python.org, the buildbot infrastructure, downloads servers, and a collection of legacy systems—bugs.python.org (Roundup), hg.python.org (yes, Mercurial still exists), the old wiki. **CDN**: Fastly sits in front of everything. When PyPI goes down, it's usually our fault, not theirs. **Vendor relationships**: We manage accounts with AWS, DigitalOcean, Gandi, Fastly, Datadog, Sentry, 1Password, and about 40 others. Each one has billing, renewals, and the occasional "why did this stop working" investigation. ## What we got wrong We over-documented at first. Created templates for everything, which meant people stopped writing because the process felt heavy. Scaled back to "just write it down, format doesn't matter." I'll admit it: I tried to Terraform the mail server. Spent two weeks writing modules for a system we touch once a year. In my defense, it *would* have been beautiful. Eventually. The runbook Ee wrote took an afternoon. And it actually got used. Lesson learned. Tried to move too fast on Terraform. Spent two months building modules for something that breaks once a year. Should have written a runbook and moved on. Underestimated how much communication overhead grows with team size. Four people need different coordination patterns than two. Five people need yet another adjustment. ## Where this goes The team has grown, but we're not done. We're actively looking to add: **A Product Engineer for PyPI.** Organizations and enterprise features need dedicated attention. PyPI's revenue potential—through premium features for companies—could help sustain the PSF long-term while ensuring amazing features remain free for our community users. We have grant funding to start this work, but we need someone to build it. **A Security Engineer to work with Seth.** CPython security is too much for one person. We have to bring on someone who can support vulnerability disclosure, security tooling, and cross-ecosystem initiatives. These aren't nice-to-haves. PyPI handles over 1 billion package downloads per day. CPython powers most of the world's Python. The community deserves infrastructure that doesn't depend on heroics. And with , building sustainable revenue and diverse partnerships matters more than ever. ## For other projects If you're running open source infrastructure with a small team: 1. Write it down, even badly 2. On-call doesn't work without clear boundaries 3. New contributors need a path that doesn't require pairing with a senior person for weeks 4. "Not my shift" is healthy, not selfish 5. Funding is always a patchwork—embrace it Python's community deserves infrastructure that works without heroics. We're not there yet, but we're closer than we were. --- *Want to help? The PSF is always looking for infrastructure volunteers and sponsors. Reach out to infrastructure@python.org.*

lazy from blog import posts