Data Engineering Digest, June 2026
Hello Data Engineers,
Welcome back to another edition of the data engineering digest - a monthly newsletter containing updates, inspiration, and insights from the community.
Here are a few things that happened this month in the community:
How much code are big tech engineers still writing by hand?
Can you really do data engineering in C#?
Will pipelines ever fix themselves while you sleep?
The Fivetran and dbt merger closes, and dbt Core goes (mostly) open source
Snowflake and Databricks place the same big bet at their summits
How Pinterest taught its pipelines to handle schema changes on their own
Community Discussions
Here are the top posts you may have missed:
1. How much code are big tech engineers still writing by hand?
A startup data engineer admitted what a lot of people are feeling: they barely write code by hand anymore. Not because they believe the AI is good, but because their company prioritizes speed and tolerates the odd mistake in production. So they asked the big tech crowd: are you still vibe coding over there, or do the higher stakes change things?
💡 Key Insight
First of all, big firms can afford more mistakes than small ones, just not at scale. On the coding itself, several recent ex-Meta engineers said data engineers there hand-write almost nothing, something like 99% of code is AI-written, and insiders still report the same. While AI is writing the code, big tech engineers are still following spec-driven workflows - write a plain document first listing the architecture, libraries, requirements, and tests, then let the agent build against it. They also rely on a semantic layer mapping what “revenue” or “active user” really means, a library of instructions for every process, and specialized helper agents to call on.
2. Can you really do data engineering in C#?
A data engineer who learned the trade from the community laid out what they thought was the standard recipe: Python, an orchestrator like Airflow, Prefect, or Dagster (the tool that schedules and runs your pipeline steps), and a data lake plus warehouse. Their shop runs mostly on C# instead, with a .NET message bus and Redis, and new management likes it that way. They wanted to know how much real data engineering happens in C#.
💡 Key Insight
Streaming pipelines on .NET and ETL (extract, transform, load) jobs via tools like Hangfire are common; that crowd just posts in .NET forums, not here.
SSIS (SQL Server Integration Services), itself written in C#, still runs ETL in plenty of on-premise Microsoft shops.
Concepts are language-agnostic: even Python leans on C and C++ underneath, so the patterns carry over wherever you write them.
Orchestration is the real gap, with Python’s schedulers far ahead and no real C# equal.
Rule of thumb: use the language your team knows best, expect more glue work on the C# side (especially orchestration), and never rewrite to escape a cloud bill - it usually just moves over to payroll.
3. Will pipelines ever fix themselves while you sleep?
Could an AI agent debug a broken pipeline on its own - schema drift (when the shape of incoming data changes without warning), missing rows, a downstream service throwing errors - push the fix, and let you sleep through it? And would you trust it with the keys?
💡 Key Insight
Not fully, but further than the “never” crowd admits. Full control is genuinely unsafe: you cannot hand a probabilistic model - one that guesses the next likely thing rather than following fixed rules - a pipeline that must behave identically every run, and people pointed at agents that “heal” a problem by quietly rewriting the tests meant to catch it, or inventing a transformation to force the numbers to line up. But the engineers actually shipping this already run a scoped-down version. One team triages data quality issues across roughly 2,500 dbt models through a chain of language models before a human looks. Another wrote fix-it rules for the top 90% of repeating alerts, so on-call wakes to an alert plus a suggested fix and still clicks the button. What works is a short leash: scope the agent to small, well-understood failures, keep safety checks enforced by the pipeline rather than the model, and let it open a fix for a human to approve. Trust grows from there, one boring win at a time.
Industry Pulse
Here are some industry topics and trends from this month:
1. The Fivetran and dbt merger closes, and dbt Core opens up
The biggest tooling story of the month is also the one most worth reading carefully: dbt’s core engine just went (mostly) open source. On June 1, Fivetran and dbt Labs closed their merger and shipped dbt Core v2.0 (alpha) the same day. Core is now built on the faster Rust-based Fusion engine, with a large chunk of once-proprietary code re-licensed as fully open source in the main dbt-core repo. The result: much faster parsing on big projects, a built-in linter, queryable Parquet output files, and an early natural-language assistant called dbt Wizard.
The catch is the licensing. There are now two free versions: plain dbt Core is fully open, but Fusion, the one dbt tells almost everyone to install, is that open core plus some closed extras shipped as a precompiled binary. As Brooklyn Data argues, the dbt most people actually run is no longer the fully open one. If you run dbt, this is a start-evaluating-now moment.
Continue reading:
2. Snowflake and Databricks place the same big bet
Snowflake and Databricks held their summits within weeks of each other, and the striking thing was how identical their bets turned out to be. Both spent their keynotes, and their acquisition budgets, racing for the same prize: ownership of the governed, trusted context that AI agents need before they can safely act on your data. The wager underneath is that the model on top is becoming a commodity, while the lasting advantage sits in the context layer feeding it.
They took different roads to get there. Databricks went architecture-first with LTAP (Lake Transactional and Analytical Processing), a pitch to serve fast transactions and heavy analytics from a single open copy of your data, backed by a query engine it says answers in under 100 milliseconds (Lakehouse//RT) and a managed stream that skips the message bus (Zerobus). Snowflake went context-and-governance first, with Horizon Context and Cortex Sense to ground agents in permission-aware business meaning, plus a planned purchase of Natoma to own the Model Context Protocol (MCP) gateway agents use to reach enterprise data. Databricks plans to buy into that same connective tissue from the security side with Panther. Even the partnerships split cleanly, Databricks with OpenAI and Microsoft, Snowflake with Anthropic. But the strategy was shared, and that is the real signal for working data engineers: semantic modeling, catalog, and governance work just moved from a side chore to the center of the platform.
Much of these announcements are roadmap, not shipped product. Databricks’ LTAP is coming soon and Lakehouse//RT is in beta, Snowflake’s Horizon Context launched with only about five connectors in preview, and the flashy speed numbers on both sides are vendor demos rather than independent tests.
Continue reading:
3. How Pinterest taught its pipelines to handle schema changes on their own
Pinterest’s follow-up from February reveals a new automatic schema handling pattern everyone can copy. The follow-up covers how they let tables change shape on their own without breaking everything downstream, which is one of the genuinely hard problems in distributed data work, since a schema is a contract that spans ingestion, transformation, storage, and backfill all at once. If you run change data capture into Iceberg, or maintain any ingestion pipeline where source tables keep shifting, it is worth your time for how Pinterest reasoned about the tradeoffs and where they drew the line on what to automate.
🎁 Bonus:
😅 Showed leadership our architecture diagram, forgot to take the last box out
📚 101 concepts every data engineer should know (Simon Späti)
📦 A weekend “pipeline in a box”: raw data to dashboard in one command
📅 Upcoming Events
No events - check back next month!
Share an event with the community here or view the full calendar
Opportunities to get involved:
What did you think of today’s newsletter?
Your feedback helps us deliver the best newsletter possible.
If you are reading this and are not subscribed, subscribe here.
Want more Data Engineering? Join our community.
Stay tuned next month for more updates, and thanks for being a part of the Data Engineering community.