Howdy, nerds. Grab a coffee. We're going to discuss rewriting part of our data pipeline from TypeScript to Python, and this time we cover the actual mechanics, because "we used a library" is a boring sentence and you deserve better.

Quick disclaimer: nobody on this team is a licensed Python therapist. If reading about the GIL gives you flashbacks, go pet a cat and come back later.

Here's the problem that started all of this. Every card we track, whether it's MTG, Pokémon, Lorcana, One Piece, or Riftbound, exists on several marketplaces at once, and none of those marketplaces agree with each other. Scryfall has one ID for a card. TCGPlayer has another. Cardmarket has a third. CardTrader has a fourth. Reprints, alternate art, and foil variants all tend to share nearly identical names, so somewhere in that mess, we need one card in our database to point at the correct listing on every single marketplace, every time, no exceptions.

We should say up front that we love every one of these games, unreasonably so. Somewhere on this team there's a person who will fight you about the color pie, someone who owns more Charizards than is medically advisable, a person who has genuinely gotten misty-eyed over a Disney Lorcana pack pull, at least one soul convinced One Piece will have more printed variants than atoms in the universe by 2030, and somebody keeping a very close eye on Riftbound as the promising new kid at the table. It's exactly why getting this right matters to us and not just to a spreadsheet.

That's a real data-matching problem with a measurable error surface, not a "call an API and store the JSON" problem, and for a while, we treated it like the second one anyway.

Here's the confession part. When two nearly-identical reprints showed up with no clean way to tell them apart by number or name alone, somebody had to actually open the product photos and look, then bake whatever they found into a written-down exception, because there wasn't yet an automated way to make that same call at sync time. That's a perfectly fine thing to do once. It's a rough way to scale, because the list only grows and the person doing the looking doesn't get any faster at it.

That folder is the number that actually moved for us. It used to hold about 3,000 lines spread across roughly 19 separate files, one per game per marketplace quirk, each one basically a fossil record of a mistake somebody eventually caught. Pixel matching handles the overwhelming majority of that same ground now, comparing actual product photos automatically instead of relying on a name and a number, and what's left for One Piece specifically is a single 65-line file, each entry dated and annotated with the real reasoning behind it, not just a bare ID dropped into an array by someone hoping nobody would ask why. Ambiguity didn't disappear entirely. It got small enough to actually read in one sitting, which might be the most satisfying sentence we've written all year.

The photo-matching itself has been quietly great too. On our first full validation run, it correctly matched 203 out of 204 cards, and the one it didn't resolve, it flagged as unconfident instead of guessing and strolling off like nothing happened. That's the part we're genuinely proud of, a system that knows when to shrug instead of pretending it's sure, which is more self-awareness than the old approach ever managed.

Once that piece was working, we kept finding reasons to build more around it. Nothing reaches production anymore without landing in a holding area first, getting diffed against what's currently live, and waiting for an actual human to approve it, which is a real safeguard that, credit where due, simply did not exist before. The old pipeline wrote straight to the live database the moment a sync finished, no pause, no second look, the data equivalent of handing someone the car keys and calling it a road test. If a source fed us garbage on a bad day, customers got garbage on that same bad day. Now it stops at review instead.

Each game's scraper turned into its own small independent service too, so a bug or an outage in, say, the Pokémon source doesn't take MTG or Lorcana down with it, and our CI only rebuilds whichever service actually changed instead of re-shipping the whole pipeline for a one-line fix. The infrastructure itself got smarter along the way as well: containers spin up when they're actually needed and shut themselves down again once idle, instead of running around the clock regardless of whether anyone's using them, which turned out to be a real cost saving and not just tidiness. We even ended up running our own self-hosted model that other services can call on demand, no third-party bill, no data ever leaving the building.

There's a longer list of smaller things too, containers that auto-update, an hourly job that quietly clears unused disk space before it becomes a problem, MTG syncs that only pull as much data as they actually need, registry logins that renew themselves so a release never breaks over an expired token nobody remembered. None of it is glamorous. All of it means fewer pages at 2am.

None of this came free, and this next part is really just for the engineers who want the honest texture of it. Type hints in Python are more of a suggestion than a bodyguard. mypy catches a lot, but in the early days, people still shipped a None where a string was expected and found out about it in the logs, and everyone develops a cautiously paranoid relationship with Optional[] after that happens once. Packaging has main-character energy too, pip, venv, poetry, uv, requirements.txt, pyproject.toml, and everyone on the team has an opinion about which combination is correct. We landed somewhere that works, though there's a Slack thread from that particular week that honestly deserves to be framed. And asyncio asks you to actually understand what you're doing in a way Node's event loop never really did. We learned that one properly, on a Friday afternoon, as one does.

None of that changed our minds though, and here's the actual reason why. The libraries we needed already existed. Image hashing, data wrangling, fuzzy matching, this has been Python's home turf for over a decade, and we got to use well-tested tools to automate a comparison that used to mean someone opening a product photo and squinting at it, one card at a time. A huge amount of this work is also genuinely exploratory, pull some data, inspect it, check an edge case, try a different matching rule, repeat, and Python's write-and-run loop just suits that kind of poking around better than TypeScript's compile-first habits do. It's also the language the next upgrade will need anyway. If we ever want smarter matching, real anomaly detection on pricing, anything more ML-flavored, that whole world already speaks Python by default, so we're not one language bridge away from leveling up, we're already standing in the room. And a straight-line pipeline, fetch, normalize, reconcile, store, just doesn't need a type defined for every single step the way a long-lived interactive app would. Type-safety anxiety was real for about a week. It was also the only real cost on the whole list, and everything above is what we got back in exchange for it.

We ended up with a lot more than a rewrite in the end. A card-matching system that knows its own limits, a review layer with an actual paper trail, a fleet of services that manage their own lifecycle, and a pile of small automation that keeps the whole thing healthy without anyone babysitting it. Python earned its spot here, and honestly, so did admitting the old approach was held together with hope and regex.

Thanks for reading. If you ever want to stress-test a card-matching system, hand it a full shelf of One Piece alternate art reprints and see what happens. Ours didn't flinch. The scrapers are staying in Python. Now if you'll excuse us, several of us have decks to build.