Data Engineering Digest, July 2026
Hello Data Engineers,
Welcome back to another edition of the data engineering digest - a monthly newsletter containing updates, inspiration, and insights from the community.
Here are a few things that happened this month in the community:
Prefect buys Dagster, and the Airflow alternatives become one
Postgres and the lakehouse meeting at storage, not inside one engine
Can Trino stand in for your data warehouse?
The semantic layer gets a vendor-neutral home at Apache
Databricks raises at a $188 billion valuation
How Meta got large language models writing its pipelines
Community Discussions
Here are the top posts you may have missed:
1. Prefect acquires Dagster
Prefect is buying Dagster Labs, putting the two most widely adopted alternatives to Apache Airflow under one owner. Terms were not disclosed. Both products live on for now: Dagster keeps its name and its open source license, Dagster+ stays a supported paid product, and pricing on both clouds is unchanged. The combined company takes the Prefect name in August.
For anyone who chose Dagster specifically because it was not Prefect, the open question is what one owner does with two competing roadmaps over the next few years.
💡 Key Insight
The community did not buy the reassurance. Support commitments hold only until the new owner changes its mind, the announcement itself admits the two products overlap heavily, and nobody is going to run both. Everyone expects the usual acquisition sequence: prices go up, one platform quietly winds down, and the open source version drops to bare maintenance. On top of that, Dagster’s founders are leaving rather than joining.
If you still want what Dagster does, self-host the open source version rather than commit to the paid cloud, or lean toward foundation-governed projects like Airflow.
2. Thoughts on the new LTAP/Lakebase architecture
“Read a Databricks piece on Lakebase/LTAP and wrote a short note on the idea that clicked for me: maybe OLTP and OLAP should meet at storage, not inside one engine.”
OLTP, or online transaction processing, is the database behind your app, tuned for tiny reads and writes. OLAP, online analytical processing, is the warehouse, tuned for scanning millions of rows. Most teams connect the two with change data capture, a pipeline that copies every row change from one to the other, and that pipeline is where the delays and the 3am alerts live. Databricks calls its answer LTAP, lake transactional and analytical processing, which we covered when it was announced at the June summit.
💡 Key Insight
A similar but older idea, HTAP (hybrid transactional and analytical processing), asked one database engine to handle both jobs. LTAP gives each job its own engine and has them share the same storage.
The design comes from Neon, the Postgres company Databricks bought last year. A storage layer sits between Postgres and the cloud, keeping the busiest data close by so everyday app lookups stay fast.
When that layer saves data to cloud storage, it writes it in column form, the shape analytics engines scan fastest, so the lake copy stays current without a separate pipeline.
You end up with one copy of the data in two shapes, one for the app and one for analytics, synced automatically.
Analytics queries read from cloud storage over the network and pay per request, so refreshing dashboards and recurring reports add up fast.
Delta and Iceberg keep old versions of every table, but that history gets cleaned up after days and is slow to query, so you still need to build real history tables.
All of that buys one fewer copy pipeline to babysit, though it is managed cloud only for now and nobody reported running it in production yet.
3. TrinoDB as an alternative data warehouse
Can Trino stand in for your data warehouse? A data engineer planning a move off Google BigQuery, chasing lower cost and a leaner open source stack, needed one layer over a business intelligence (BI) tool, the product’s backend database, Elasticsearch, buckets on Google Cloud Storage and Amazon S3, and BigQuery itself.
💡 Key Insight
A warehouse like BigQuery bundles three things: storage, a query engine, and the housekeeping that keeps both healthy. Trino is only the middle piece. It takes your SQL, fans the work out to wherever the data already lives, whether that’s Iceberg tables on cloud buckets, Postgres, or Elasticsearch, and joins the results, all without moving or storing anything itself.
Choosing it means signing up to assemble the rest of the warehouse yourself. Storage becomes Iceberg or Delta tables on your buckets, dbt handles transforms, a client like DBeaver or Power BI faces the users, and housekeeping like compaction runs on your schedule. The tradeoffs with distributed query systems like this are typically speed and freshness. Joins across different systems drag data over the network and can crawl, and Trino does not ingest streams, so sub-second dashboards need a real-time analytics database alongside.
Rule of thumb: reach for Trino when you want one SQL layer across many systems. If the goal is a smaller bill, tune what you already have first.
Poll Results From Last Time
How much of your production code is AI-written today? (28 responses)
Industry Pulse
Here are some industry topics and trends from this month:
1. The semantic layer gets a neutral home: Apache Ossie
The Open Semantic Interchange (OSI) is now Apache Ossie, accepted into the Apache Incubator this month. The spec and the community are unchanged; the governance just moved. Ossie defines a vendor-neutral format, written in plain text files, for expressing business metrics, dimensions, and how they relate, so “monthly active users” stops meaning three different things in the warehouse, the customer system, and the BI tool. The founding coalition has grown from 17 partners to more than 50 organizations, with contributors from Snowflake, Salesforce, Databricks, and dbt Labs, and converters already merged for the dbt Semantic Layer, Apache Polaris, and Snowflake’s semantic model. The spec is version 0.1 and incubating, so it may be too early to migrate but keep an eye on this space.
2. Databricks raises at a $188 billion valuation
Databricks signed a term sheet valuing the company at $188 billion, led by existing investor Coatue. The round closes later this summer and adds roughly $3 billion to the balance sheet, according to Wall Street Journal reporting. The money goes toward the AI strategy, naming Unity AI Gateway, Genie, and Lakebase, plus further acquisitions. The numbers released alongside it are the useful part: Databricks SQL has reached $1.5 billion in annual recurring revenue and is growing more than 100% year over year, and the company expects to exit the first half of its 2027 fiscal year at roughly $6.9 billion. For scale, $188 billion against about $5.4 billion in yearly revenue prices Databricks at roughly 35 times revenue. Snowflake trades publicly at around 18 times. Ali Ghodsi repeated that an initial public offering is on track for possibly as soon as next year, with nothing filed yet.
3. How Meta built DEmate, its data engineering assistant
Five months from experiment to 3,500 weekly users, with 80% of suggested code changes accepted. Meta’s analytics stack is homegrown, with its own orchestrators, query engines, and hundreds of in-house functions, so a model that writes flawless standard SQL still produces code that will not run. The write-up covers how Meta got around that without training a custom model, and the most interesting part comes later: what broke once engineers across the company started contributing to the system, and how Meta unblocked it. Every company’s stack has in-house quirks that trip up AI coding tools, and this is one of the most detailed public accounts of getting past that.
🎁 Bonus:
📅 Upcoming Events
8/31-9/2: Airflow Summit 2026, Austin, Texas
Share an event with the community here or view the full calendar
Opportunities to get involved:
What did you think of today’s newsletter?
Your feedback helps us deliver the best newsletter possible.
If you are reading this and are not subscribed, subscribe here.
Want more Data Engineering? Join our community.
Stay tuned next month for more updates, and thanks for being a part of the Data Engineering community.
Very nice post. I'm on Social Security otherwise I would subcribe to. your channel.