Yetunde Dada is Senior Director of Product Management at Astronomer — the primary custodian of Apache Airflow. Astronomer leads roadmap and engineering for the world's largest open-source data orchestration project, originally built at Airbnb and now part of the Linux Foundation, and runs a managed-Airflow business alongside it.
Yetu has previously led product strategy at QuantumBlack, where she helped create and scale Kedro — an open-source framework for production-ready data and ML pipelines, now used by thousands of organisations globally and also stewarded by the Linux Foundation.
We talked about:
What "the 10,000-hour problem" looks like inside a company shipping AI features — and why the convergence of designer, PM and engineer into a single "builder" only works when there's craft behind it
Why data foundations didn't go away with AI — "everyone talks about context, but context is just data" — and what that means for orchestration
Inside Otto and the Astro IDE: how Astronomer encodes public, vendor-specific and customer-specific knowledge into the agents data engineers actually use
The chatbot as a research channel — what you learn when every customer prompt becomes product feedback
What an "agent harness" is — and why CLIs have been quietly beating MCPs as the interface agents prefer
The hot take: AI isn't creating enough new jobs to absorb the ones it's replacing — and the open question of how juniors will learn the craft
We need to entirely reframe how junior people are taught. They need to learn this editorial skill set — to prompt the agent in the right way — and also know what craft excellence looks like, even though they're not being put through the 10,000 hours. How do you condense 10,000 hours into 100? Because these people need to become more proficient at their job.
James Mulligan
Yeah. Yetu welcome — amazing to have you here today. I feel like we've done this before, in maybe a slightly different environment, but probably a very similar topic. Why don't we start with who are you, what do you do, and where do you work?
Yetunde Dada
Sure. So I'm Yetu — Senior Director of Product Management at Astronomer. Astronomer, more broadly, is the primary custodian of Apache Airflow, which is the world's largest open-source data project at this point. Airflow largely focuses on data — moving data from point A to point B, doing it scheduled, doing it monitored. Astronomer's relationship with the Airflow project is kind of like Databricks to Spark — we drive the roadmap and have a very successful commercial business built on top of it.
James Mulligan
And Airflow has been around for a while, right?
Yetunde Dada
Yes. Airflow evolved out of Airbnb. Astronomer's relationship with the project really picks up about six years ago — we decided to become primary contributors and drive forward development. Our impact has been significant. The growth of Airflow and its dominance in industry is largely because we've been driving the roadmap, convening everyone toward solving some of the major challenges. Our product focus is split in two: managed Airflow — your experience of actually running and setting up the infrastructure, we do all the plumbing — and the data-engineering experience, how do we make data engineers very productive. That's the part I lead, and it looks at how we embed AI more broadly to help data engineers with the things they need to do.
James Mulligan
I guess it's a misnomer to describe Astronomer and Airflow as an AI product, because it's been around a lot longer than the current flavour of AI. Data orchestration has existed and been required for decades. Do people often ask, is it primarily for AI? And I guess the answer is no. How do you distinguish the use case of a decade ago versus what the demand is today?
Yetunde Dada
Two ways of looking at it. First, recent investments in the Airflow project itself. When you're writing a pipeline, you can put anything in it — that's the extensibility of Airflow. You could be moving data, or you could be a major retailer using the same pipeline analogy to pay people's payslips. In this world, it means you can also orchestrate pipelines for agents — which is where we see the project evolving. There's work now with a common AI provider that would allow you to use Airflow to set up pipelines for agentic workflows.
James Mulligan
Interesting.
Yetunde Dada
This is where the project is evolving — as data engineers' work is changing and they focus more on agentic workflows, we're trying to meet them where they are. But importantly, even if we look at our large-scale customers, the ones that got their AI strategy right are ones that really invested in their data foundations being correct. Everyone talks about context, context, context with AI — but it's data. Making sure it's structured, well-labelled, arrives on time, and everyone understands who owns it. All those fundamentals of your data platform — if you did them right, and you were probably using Airflow to scale governance across your business — you are well on your way to adopting AI more broadly across the org.
James Mulligan
It's fascinating because a decade ago, when people were starting to do advanced analytics and machine learning, this whole concept of a data foundation was common law — everyone knew you needed it. Now everyone, regardless of role or level, has access to this magic interface called AI. Do you see a lot of companies consequently thinking they can skip a lot of that stuff?
Yetunde Dada
What's happening is how they change their investment approach — where they are allocating money. There's still work to be done on data foundations, but everyone's like, let's do the AI part first, then figure out the foundation. We are seeing, though, that AI is creating more visibility on the gaps in data foundations. So we hope more companies really take it seriously when trying to repair it. We do think orchestration is at the heart of making sure those data foundations are set up correctly.
James Mulligan
Structurally within Astronomer, there's a series of product teams working on various features and components of Airflow, plus the managed service. You've been building products for a long time — before the LLM craze and coding agents came along. How do you see that structurally changing? What are you seeing happen within your team, and what do you see happening with product management generally with the advent of AI?
Yetunde Dada
Within my team I currently work closely with two people. Taylor Murphy kind of leads — you'll actually see a major launch coming out today regarding Otto, our engineering agent. Otto is the pinnacle of a lot of our AI investments over the past year. I specifically worked on things like the Astro IDE — our AI code-authoring editor — which created an in-browser web experience for working with Airflow code: building data pipelines using an agent that was really good at working with Airflow code and understanding your codebase, all the way through to testing those pipelines. Otto is the advent of us being able to assemble what we call public domain knowledge around Airflow, Astronomer-specific domain knowledge around Airflow, and our company's data as well. We generate so much data when our customers are using the platform — they should be able to use that data for things like root-cause analysis: why did my pipeline fail, how can I find what's going on. Astronomer-specific context — we have playbooks on how to upgrade people's codebases when moving from Airflow 2 to 3, on upgrading dependencies, on the tools you're using. We've embedded that into the agent to really make some of those workflows efficient. So when we look at the evolution of AI — at least on my team — we have a part of the organisation specifically focused on how to make sure data engineers have a great time at work, and AI is one of the tools to enable that developer experience.
James Mulligan
Can I ask a few questions on the data engineering piece? Because I think this is a really interesting starting point. Data engineers are your end user of the Airflow products. And your starting point is: how can we leverage AI to make their experience better? That's really cool — because rather than it being, okay, we've got an engineer that's booted up Claude Code and now we're doing things faster, the product-management best practice being applied here is: how can we use that technology to improve our product for our end users?
Yetunde Dada
It actually should always be that way. Because remember, AI on its own is just a hammer — and you shouldn't be looking for nails just to bash in order to solve the problem.
James Mulligan
A lot of nails getting bashed on LinkedIn at the moment.
Yetunde Dada
Many. The Astro IDE has been in preview with our customers since September last year. It's probably the only real chatbot or prompt-box entry point on the Astronomer platform. We've been able to look at the prompts people have been sending — the questions people are asking — and now you see us effectively building for the use cases the agent couldn't do well.
James Mulligan
Wow. Okay — this is super meta, because now we're talking about an entirely novel feedback channel from your end users. Gosh, now I'm rethinking all the times I've screamed at AI on the various tools I'm trying to get it to control — thinking about the poor product manager of that product who's now reading my screams. Have you seen an uptick in novel feature requests because of that channel?
Yetunde Dada
We've set it up so that summaries pop into a Slack channel — we have LLMs looking through and summarising the intent of each session, categorising these things, which makes our analysis a little easier. That's where we've started to see themes. For instance, the Astro IDE didn't really support a good debugging or troubleshooting workflow. So Stephanie, who works on my team, is building an agent specifically focused on being really good at investigations. This is also a remarkable point for organisations. To give context: in some organisations you have 100 data engineers every day building pipelines, and 20 engineers specifically staffed to maintain that growing number of pipelines. They have a certain number of pipelines per person they can maintain. When those pipelines go down, it's a big point of drama for those organisations. Having tooling that helps them be efficient — helps them do more pipelines per person — is something our customers really need. So combining what would otherwise be a product need with seeing what people are actually asking the agent to do and not getting a sufficient response — yeah, we have space to address those use cases.
James Mulligan
I'm thinking more and more about this agentic piece you're describing. I think a lot of engineers and product builders will come to the realisation this year that there's probably a benefit in their product being configured in a way that it can be agentically controlled. But it's not as straightforward as just pointing an LLM at your product and asking it to start pressing buttons. This is maybe a conversation for an hour rather than 20 minutes — but what advice would you have for people trying to make that step? Going from their existing tool to it being controlled by an agent — what are the things in between they should be thinking about?
Yetunde Dada
Good question. The space is constantly evolving. When you look at the concepts around an agent harness — all the supporting things the agent needs around it, the system prompt, the tools it has access to — there was a lot of work earlier last year where we talked about MCPs being the end-all for how you should pass that to the agent so it would know what to do. But since time has gone by, we've actually found that agents really like CLIs — command-line interfaces — and working with those things. So when you're thinking about the agentic workspace — and this is sometimes a challenge I have with some of my teams — the agent is acting on behalf of the user. What is still to be done by the user? And therefore focus on the tools that help them do that. That helps limit, or at least surface, which parts of your interface with the product you need to make available to the agent. You shouldn't be making everything available — but you do need to make the most important things available so the agent can do the job on behalf of the person working with it.
James Mulligan
Got it — product-management best practice for a new age of product management coming out of this conversation already. The guiding principle being: you should already know what your user wants to do. That's their need. Therefore, how do you translate that into an agent being able to do it — rather than going the other direction, giving it access to the entire API and seeing what happens. Can you explain the harness bit? I don't think everyone will be familiar with the term.
Yetunde Dada
Agent harness — I guess it's a catch-all term — is all the things the agent needs to be successful to complete its job. Obviously the system prompt is its description of why it exists, what its job is supposed to be, what it's supposed to be helping the end user do. All the way through to: what context does it have available to solve the job? Is it referring to your product documentation? Is it looking at maybe skills that you've actually written out for it, so it can be successful? All the way through to: does it have access to MCPs and tools it can call to execute on its job? This isn't a new concept, but it's all the things you need to capture — because these agents on their own are actually pretty good, these models are pretty good. The real differentiating factor is what context they have to execute the job they're instructed to do really well. So you think of the agent harness as the differentiating factor as to whether it's a general-purpose model or it is really successful at data-engineering work.
James Mulligan
So I guess this is an optimistic view, maybe — a rose-tinted view of AI and how it's changing the way we think about building products. But there do seem to be entirely new paradigms spawning. This is the second time I've spoken to someone about harnesses, and that as a concept didn't exist a year ago. At least from my perspective — and maybe this is biased, having worked in AI companies — I don't see the role of a product team going away. It's just what they focus on that's changing and evolving based on this entirely new paradigm and how you would interact with software you're building. Do you see that too? Is that your reflection as you build these things out?
Yetunde Dada
I'm going to have a hot take on this one.
James Mulligan
Okay.
Yetunde Dada
Relates to the future of work.
James Mulligan
Love it. Let's do it. What's the hot take? We're all doomed?
Yetunde Dada
I should have been a plumber. I told my mom when I was 16.
James Mulligan
Well, Airflow is a type of plumbing, you know.
Yetunde Dada
Well, that's one way of looking at it. When we talk about the future of work, there is going to be significant impact in the way AI influences the tech world. I am a big user of AI in my workflows. It's helped me do things all the way from problem discovery and synthesis of customer issues — the ability to comb through all our customer transcripts in such an easy way and pull out insights means problem discovery is very focused for me. Prototype development — the fact that I can spend time vibe-coding a concept and bringing it to life. Over time I've worked with designers and engineers and kind of know how to shape these things and at least bring the first initial versions to the front, so I can have a very constructive conversation with engineers to take it forward. It's shortened the amount of time it takes to bring value to my customers — and to test concepts with my customers as well, to make sure we're on the right track. The fact that I'm able to work significantly faster is very exciting. But I do think — maybe I'm a little more pessimistic about what this means for the future of work — because we really talk about the convergence of roles happening right now. In the tech space, software engineering has been the role most impacted by AI. We see all the layoffs happening. It's caused a bit of a crisis on the software-engineering front — around what do they do, how do they stay relevant. So they've started to move into the designer role, into the product-management space. That will cause a convergence in how we all have to work. You might see companies start to hire for what we call a builder role — someone who can do all three roles end-to-end with the power of agents supporting them. But even with that, part of the reason at least older people, like us, can do work like this is because we spent so much time learning the craft and learning principles.
James Mulligan
Right.
Yetunde Dada
So it does make you wonder — what happens when we see this?
James Mulligan
When people can't learn that craft, right?
Yetunde Dada
I hope the AI is good then — because it's going to need to be.
James Mulligan
I completely agree with this convergence point. I see it on a daily basis where the product builder — it's becoming about outcomes rather than very specific, constrained roles on how to get to an outcome that matters. I also have fear around this craft piece, because those product builders are typically people that have been doing it for 10, 20 years and know what good looks like across each stage of a product-development lifecycle. They might not be an amazing engineer, but they have the principles to understand what is good and what is bad. Likewise from a design perspective and a product perspective — they've had exposure to it. I guess what you're identifying is that unless you've done that grind — unless you've got your 10,000 hours — what is the next generation going to do?
Yetunde Dada
Exactly. Because they're not necessarily being given opportunities to even learn it.
James Mulligan
How do you solve for that? Does it mean investment into junior training? Investment into roles you might not think you need right now? How do you solve for it?
Yetunde Dada
I'm actually not sure — there are two things that need to happen. One: we need to entirely reframe how junior people are taught, because they need to somehow learn this editorial skill set — prompt the agent in the right way — and also know what craft excellence looks like, even though they're not being put through the 10,000 hours. How do you condense 10,000 hours into 100? Because these people need to become more proficient at their job. The second thing is — I don't know if industry is concerned with the junior people. This one I feel has to be a regulatory, government-style intervention. But they've obviously got their hands full. I foresee some future challenges with the future of work, specifically in our area, because AI is amazing — I'm a big user of it. But when I look at the future of work, I do see things that worry me about the destruction of jobs in the space — the fact that AI is not really creating so many new jobs to absorb people, which is quite different from other technology revolutions we've had before. We have to live through it. I'm a millennial — we've lived through our challenges. We're going to live through another one in future.
James Mulligan
Well, the millennials, the product builders that are doing all of these roles, they're going to age out.
Yetunde Dada
Yeah, we're going to age out.
James Mulligan
They'll want to retire and stop working. And then who will be left to make these companies money?
Yetunde Dada
Agents will.
James Mulligan
I guess then it becomes a question of accountability. If there's no one accountable for the outcome on behalf of the customer, you can't blame an agent.
Yetunde Dada
Yeah.
James Mulligan
But maybe that's a little utopian to think people will worry about that. Well — regardless, rallying cry: this technology is transformative. It's fundamentally changing the way we all build software products. But there needs to be serious conversation about tactical interventions. What does the next generation learn?
Yetunde Dada
What does the next generation learn? What do we do in the interim period as well, when they can't work? That was also a worrying one — because if we're not going to be having that many roles in the technology space, where do these folks go?
James Mulligan
Yeah. I guess we'll figure that out in the next conversation. Thank you so much. I've learned so much, as always, talking to you — and hopefully we'll get to speak again very, very soon.
