🍔🧠 How Uber Upgraded 2M Spark Jobs (Saved $MMs/Year)

.bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
.bh__table_cell { padding: 5px; background-color: #FFFFFF; }
.bh__table_cell p { color: #2D2D2D; font-family: ‘Helvetica’,Arial,sans-serif !important; overflow-wrap: break-word; }
.bh__table_header { padding: 5px; background-color:#F1F1F1; }
.bh__table_header p { color: #2A2A2A; font-family:’Trebuchet MS’,’Lucida Grande’,Tahoma,sans-serif !important; overflow-wrap: break-word; }
Today’s issue of Hungry Minds is brought to you by:Happy Monday! ☀️Welcome to the 140 new hungry minds who have joined us since last Monday!If you aren’t subscribed yet, join smart, curious, and hungry folks by subscribing here.📚 Software Engineering ArticlesThe last app for technical diagramsReact Native CPU spike reveals critical mobile app optimization lessonsHow Infobip scaled to handle 10 billion daily messagesMaster SQL window functions with practical examplesLoad balancers explained with real-world architecture examplesBreaking free from tutorials with this Python roadmap🗞️ Tech and AI TrendsApple’s new iPhone chip architecture prioritizes AI workloadsAndroid for PC confirmed by Google executivesTesla hints at new Cyber SUV and compact Cybertruck👨🏻‍💻 Coding TipWhat is edge storage?Time-to-digest: 5 minutesBig thanks to our partners for keeping this newsletter free.If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you love. 💚Unify Your Engineering Workflows in One Visual Workspace with Miro 🎯Stop context-switching between fragmented tools. Miro brings your entire technical workflow into one unified visual platform:Map Complex Systems: Powerful diagramming with 2500+ shapes and AI generation for architecture & flows.Run Async Collaboration: Lead effective stand-ups, retros, and sprint planning with async video and tools.Track Sprints Visually: Visualize dependencies, timelines, and backlogs with native Jira/Linear integration.Document & Align: Create living technical docs that stay in sync with your team’s work.Integrate Your Stack: Works seamlessly with GitHub, AWS, Azure DevOps, and more. Get started free How Uber Migrated 2 Million Daily Spark Jobs to Version 3.3 ⚡Uber operates one of the world’s largest Apache Spark deployments with over 2 million applications running daily across 20,000+ scheduled workflows. Moving from Spark 2.4 to 3.3 meant upgrading every single job without breaking production systems or affecting millions of users.The challenge: Scale manual migration across 40,000+ Spark applications with no staging environment, no existing test cases, and zero tolerance for production data corruption.Implementation highlights:Smart code rewriting: Extended their open-source Polyglot Piranha tool to automatically parse ASTs and apply Spark 3 compatibility transformations across Java, Scala, and Python codebases Iron Dome framework: Built custom interceptors for Spark’s Catalog interface and Hadoop’s File Output Committer to safely redirect production paths during shadow testingDependency chain overhaul: Systematically upgraded Python 2→3, Scala 2.11→2.12, and resolved monorepo conflicts while maintaining backward compatibility with custom shuffle manager ZeusAutomated shadow testing: Created Cadence-powered workflows that automatically shadow production runs, validate output data, and mark jobs ready for migrationRuntime path translation: Implemented filesystem-level guardrails that transform /db/tbl to /stgdb/tbl at runtime, ensuring zero risk to production dataResults and learnings:Massive automation win: 85% of jobs migrated within 6 months through automated tooling vs manual developer effortSerious performance gains: 60% of jobs saw 10%+ improvement with 50% overall reduction in runtime and resource usageDeveloper time savings: Saved thousands of engineer hours while unlocking millions in infrastructure cost savingsUber’s approach proves that even the most complex infrastructure migrations can be automated when you build the right tooling first. Their framework now powers other major upgrades, showing that good automation pays compound interest. FREE Technical Diagrams Online Make technical diagrams at ease with Miro’s many diagramming tools. Share perspectives, clarify complexity, and automate tasks — all for building the next big thing. Stop grinding leetcode for data engineer interviews! Landing a role in big tech and “grinding leetcode” have gone together like peanut butter and jelly for the last ten years. How Kafka Works #91: Learn Everything About Apache Kafka’s Architecture, Including Brokers, KRaft, Topic Partitions, Tiered Storage, Exactly Once, Kafka Connect, Kafka Schema Registry and Kafka Streams What All Can A Load Balancer Do? One tool…many features We Thought React Native Was the Answer — Until Our App Hit 190% CPU How we scaled Motion’s mobile team, rewrote our calendar in native code, and found a hybrid approach that actually works How Indexing Works in Distributed Databases In a single-node database, indexing is relatively simple. You Won’t Be Promoted Unless You Ask Don’t wait for your manager to bring it up; you should start the conversation. It is perfectly reasonable to ask for a promotion; many people feel shy about it, but you don’t need to. Pitch your case How AI Is Impacting Engineering Leadership Recap of the talk I did at the Tech Lead conference this week! ESSENTIAL (strategy-ninja)Good Strategy / Bad StrategyARTICLE (networking-for-nerds)Your Network is Your Asset as an Engineering LeaderARTICLE (sql-battle-royale)Row_Number() vs. Rank() vs. Dense_Rank() Easily Explained!ARTICLE (pixels-gone-wild)Building Conway’s Game of Life in Go with raylib-goARTICLE (geo-data-party)Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizenESSENTIAL (rest-in-pieces)GraphQL 101: API Approach Beyond RESTARTICLE (message-tsunami)How Infobip’s Infrastructure Team Handled 10 Billion Messages in a DayARTICLE (escape-tutorial-prison)The Python Roadmap That Finally Gets You Out of Tutorial HellARTICLE (rpc-superhero)Cap’n Web: a new RPC system for browsers and web serversWant to reach 190,000+ engineers? Let’s work together! Whether it’s your product, service, or event, we’d love to help you connect with this awesome community. WORK WITH US 🔲 Apple Achieves Full In-House Chip Control, Signals Major AI Push (3 min)Brief: Apple completes transition to full in-house chip production for iPhones, marking strategic shift to optimize AI capabilities while reducing dependence on external suppliers.🤖 Google Cloud and Datadog Release Guide on AI’s Future with 7 Key Insights (2 min)Brief: Google’s Director of AI and Datadog’s VP of Engineering share strategic insights on AI implementation, LLMs, and observability practices in a new comprehensive guide for technical leaders on Google Cloud.🚙 Tesla Hints at Cyber SUV and Smaller Cybertruck in Product Pipeline (2 min)Brief: Tesla’s Chief Designer suggests company is considering Cyber SUV and compact Cybertruck models, with a potential mock-up design already spotted in promotional material, despite earlier statements about not using stainless steel exoskeleton in new vehicles.💻 Google’s Android PC Push: Desktop Expansion Plans Revealed (2 min)Brief: Google is reportedly developing an Android-based desktop operating system, signaling a major push to expand beyond mobile and challenge Windows and macOS in the PC market.🤖 Perplexity Launches Email AI Assistant for Gmail and Outlook Users (2 min)Brief: Perplexity introduces Email Assistant that helps with meeting scheduling, email drafting, and priority labeling for Gmail and Outlook, available exclusively to Max subscribers as part of their AI productivity suite.This week’s coding challenge: Build Your Own BitTorrent Real-world proficiency projects designed for experienced engineers. Develop software craftsmanship by recreating popular devtools from scratch. This week’s tip:Cloudflare Workers Durable Objects provide globally consistent state with SQLite-like transactions across edge locations. Each object instance gets its own isolated JavaScript context with persistent storage that automatically handles consistency and failover.Wen?Real-time collaborative apps: Building document editors where each document is a durable object handling concurrent editsGame state management: MMO games where each room/lobby maintains a consistent state across global playersRate limiting with memory: Per-user rate limiters that persist across requests and maintain state during traffic spikesIf I cannot do great things. I can do small things in a great way.Martin Luther King, Jr.That’s it for today! ☀️Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦Thanks for spending part of your Monday morning with Hungry Minds.See you in a week — Alex.Icons by Icons8.*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).Powered by beehiiv