Data Engineering Digest, December 2024 & Annual Recap

Hello Data Engineers,

Welcome to the end of year edition of the data engineering digest - a monthly newsletter containing updates, inspiration, and insights from the community. This edition contains a special end of year recap of highlights from the community. We’ll be back to our regular format in the new year.

Data Engineering 2024 Recap

Top 10 Articles of the Year

We sorted through thousands of posts to bring you the best articles from 2024. Be sure to bookmark these if you haven’t already!

1. How Stripe Processed $1 Trillion in Payments with Zero Downtime

2. This is How Discord Processes 30+ Petabytes of Data

3. Is S3 Becoming a Data Lakehouse?

4. Data engineering projects: Airflow, Spark, dbt, Docker, Terraform (IAC), GitHub Actions (CI/CD), Flink, DuckDB & more runnable on GitHub codespaces

5. The Numbers behind Uber’s Data Infrastructure Stack

6. Why use dbt

7. Analyst to Engineer

8. How we built a 70% cheaper data warehouse (Snowflake to DuckDB)

9. What I’ve Learned After a Decade of Data Engineering

10. What DuckDB really is, and what it can be

11. (Bonus) The State of Data Engineering in 2024: Key Insights and Trends

The State of Data Engineering in 2024: Key Insights and Trends

“Best of” Lists

• 💻 Best Data Engineering Blogs

• 📚 Books to Start, Grow, or Deepen Your Knowledge as a Data Engineer

• 🪵 Best resources for data architecture, data management, and data modeling

• 📝 Data Testing Cheat Sheet: 12 Essential Rules

Ask Me Anything Events (AMAs)

• Joseph Machado - I am a data engineer(10 YOE) and write at startdataengineering.com - AMA about data engineering, career growth, and data landscape!

• Andreas Kretz - I’ve taught over 2,000 students Data Engineering – AMA!

• Great Expectations Engineering - We’re the Great Expectations team. We just launched GX Core 1.0 and we’re here to answer all your data quality questions. Ask us anything!

Acknowledgements and Thank Yous

Open Source Contributors

Thank you to the people who improved our community’s open-source projects by contributing their time and knowledge. Want to get involved? Get involved here.

• ykdojo

• cbergh

• genegc

• lbg-oju

• MartinRst

• royendo

• dataoculus

• JPHaus

• Benjameep

Sponsors & Backers

A special thanks to our sponsors and backers who support our community.

• Great Expectations

• Starburst

• Data Engineering Jobs

• davidkhala

You

And thank YOU for being here. There are countless members who make our community great through sharing high quality content, answering questions, and generally helping out however they can. You are appreciated.

Going into 2025, let’s take a pulse check on how Data Engineers feel about what is most likely the most important topic in tech right now:

---

---

📅 Upcoming Events

• 1/19-1/22: CIDR 2025

Share an event with the community here or view the full calendar

---

Opportunities to get involved:

• Volunteer to speak at/organize a Data Engineering meetup

• Share your story on how you got started in DE

• Help improve the DE Salary app

• Contribute to the wiki

Want to get involved in the community? Sign up here.

What did you think of today’s newsletter?

Your feedback helps us deliver the best newsletter possible.

• ⚡⚡⚡ Awesome!

• ⚡⚡ Pretty Good!

• ⚡ Could be better.

If you are reading this and are not subscribed, subscribe here.

Want more Data Engineering? Join our community.

Want to contribute? Learn how you can get involved.

Stay tuned next month for more updates, and thanks for being a part of the Data Engineering community.