I think analytics engineering's potential as a role can be systematically limited by how it's located in data teams. From my perspective, it usually sits in an ankward role between data engineering and data analytics, and that can limit the scope of what they are able to do (due to how, in order to undertand the business context, one must sit close to the business, and pretty often that role is owned by the data analytics team). It also can become a rebranding of the business intelligence usual role, not changing much beyond that.
I've been wondering how a change in the team's structure might help analytics engineers flourish. I think embedding them in teams / making them owners of certain domains is inevitable if you want them to actually go beyond being the "dbt folks" (as that's more of a consequence of them not being able to carve out an space, squeezed as they are between data engineers that own the platform and data analysts that own the answers to the business questions), however that creates a new problem to tackle, which is how you make sure the dimensional model doesn't take a hit due to lack of centralized governance.
Maybe it shouldn't be a role, but a skill expected from data engineers, although I've found many data engineers don't really care about analytics engineering per se.
Overall this post had me nodding the whole time. I wonder whether a rebranding could be useful, too. "Decision engineering" comes to mind. Logic behind is that, if we frame the role around making better decisions faster (be it through reverse ETLs, or alerting, or dashboards, or LLM bots, or metric trees, or better processes such as WBRs, or whatever), then understanding the business becomes a tool on itself, as does semantic and dimensional modeling. Nobody woud expect a "decision engineer" to just own the dimensional layer, it'd be a means to an end, just like the making of a dashboard, which would be measured against whether the problem identified around a decision has been tackled or not.
Thanks for taking the time to read and sharing your perspective here! This is really astute and I appreciate the depth of the comment.
The "squeezed between data engineering and data analytics" framing nails it. That tension is a big part of why the role's scope stopped growing, and I think organizational structure determines more about what analytics engineers can become than most people realize.
On the rebranding question: I like the thinking behind "decision engineer," and "context engineer" maps well too. But I've stopped caring about the title. Data engineering is already such a wide term that it's almost as generic as "software engineer," and analytics engineering carved out a focused subset of that breadth. The people who are great at this work thrive because they go deep on business logic rather than trying to own the full data engineering surface area. I want those people empowered, regardless of what we call the role.
The centralized governance funkiness you're raising is worth a longer conversation. Embedding creates ownership but risks fragmentation. I don't have a clean answer for that one yet. 'preciate you, dude!
There's so much to say about this, but I think you're on the correct vector. I have been working independently to capture all of this on a the 'small data revolution' scale. I've been a practitioner at the OLAP and DW scale and ultimately when it comes to human decision-making that's all the data it ever takes.
For example, I recall when our folks were called in to solve a merchandizing problem for Gap. At the time, we had Sun's largest server, an E10000 working on a model with two, half-million member SKU dimensions. The problem was that the management philosophy of the company maximized store owner flexibility. So Manager A and Manager B could call T-Shirt X whatever they liked. We called it 'SKU Skew' and even though we only had 3GB of source data, finding the duplications was a nightmare. Just getting the aggregations was more than the machine could handle. The entire problem was one of semantics forced by company policy which we demonstrated gave a huge calculation problem. A few people there understood this and they had been kicking the can waiting for the next evolutions in compute and methodologies.
I've got all of these memories from a career of over 30 years designing and building through multiple generations of hardware, networks, storage, software, architectures and methodologies. It remains fascinating, especially in fields like aviation.
I have been bending, however as I alluded to the single expert worker, using myself as a model. I have been using AI to build a virtual practice based on the decisions I've made in my designs, choices and decision points. The best way I would describe this is what is the architecture you would give to a CIA analyst with a PhD in X who would not just RAG an existing book, but write the referential book. IE, what kind of system do you build for the guy who knows everything? It's working at a philosophical level about decision making rather than from a corporate paradigm.
The meta-knowledge working with new tools and techniques will generate the next generation of the consulting industry's approach to serving the next generation knowledge worker. Matters of governance will be key, but also self-governance.
----
My current personal project is building a tool that will help me manage trust for thousands of people, agents & services. It will hopefully answer questions like "Which of my 5,000 subscribers would I trust and therefore encourage to comment on my Substack, and which of them live near me so we could meet for a drink?"
I could see SWEs owning more of this, especially the upstream fixes you're describing. The closer the fix is to the source, the less cleanup downstream. And SWEs have a real advantage in understanding the systems generating the data.
Where I'd push back is on the business logic side. Knowing why a status column has five values is different from knowing which three count as "active" for sales vs. finance. That interpretive work requires proximity to the business, and it's what AEs specialize in. SWE is even broader than DE (which is arguably already a specialization within SWE), and analytics engineering carved out focused depth on exactly that layer. When you hand it to someone with a broader scope, they're either stretched too thin or they're doing analytics engineering under a different name.
Curious whether you've seen orgs where SWEs owned this end to end. I'd love to know what made that work.
I’ve seen it at a growth stage company where founding engineers maintain an incredible understanding of the analytics needs adjacent to the core app stack, but only because those analytics needs were so fundamental to the business (revenue driving). It does require some level of polymath inclination, and business sensitivity which are both rare enough to not expect it as a default.
But there is always the “important but not business critical” like finance that must run as best it can without consuming SWE time.
I think that dynamic now changes with the repricing of code. The AE can right code. They become SWE on the margin.
What I’m effectively saying is that AE doing dbt is a symptom caused by lack of SWE, and coding agents might meaningfully change this. In that case I think AE / DE / SWE overlap even more.
(Another argument is that analytics is a core capability of a good “Product Engineer”)
I think analytics engineering's potential as a role can be systematically limited by how it's located in data teams. From my perspective, it usually sits in an ankward role between data engineering and data analytics, and that can limit the scope of what they are able to do (due to how, in order to undertand the business context, one must sit close to the business, and pretty often that role is owned by the data analytics team). It also can become a rebranding of the business intelligence usual role, not changing much beyond that.
I've been wondering how a change in the team's structure might help analytics engineers flourish. I think embedding them in teams / making them owners of certain domains is inevitable if you want them to actually go beyond being the "dbt folks" (as that's more of a consequence of them not being able to carve out an space, squeezed as they are between data engineers that own the platform and data analysts that own the answers to the business questions), however that creates a new problem to tackle, which is how you make sure the dimensional model doesn't take a hit due to lack of centralized governance.
Maybe it shouldn't be a role, but a skill expected from data engineers, although I've found many data engineers don't really care about analytics engineering per se.
Overall this post had me nodding the whole time. I wonder whether a rebranding could be useful, too. "Decision engineering" comes to mind. Logic behind is that, if we frame the role around making better decisions faster (be it through reverse ETLs, or alerting, or dashboards, or LLM bots, or metric trees, or better processes such as WBRs, or whatever), then understanding the business becomes a tool on itself, as does semantic and dimensional modeling. Nobody woud expect a "decision engineer" to just own the dimensional layer, it'd be a means to an end, just like the making of a dashboard, which would be measured against whether the problem identified around a decision has been tackled or not.
Does this resonate with you?
Thanks for taking the time to read and sharing your perspective here! This is really astute and I appreciate the depth of the comment.
The "squeezed between data engineering and data analytics" framing nails it. That tension is a big part of why the role's scope stopped growing, and I think organizational structure determines more about what analytics engineers can become than most people realize.
On the rebranding question: I like the thinking behind "decision engineer," and "context engineer" maps well too. But I've stopped caring about the title. Data engineering is already such a wide term that it's almost as generic as "software engineer," and analytics engineering carved out a focused subset of that breadth. The people who are great at this work thrive because they go deep on business logic rather than trying to own the full data engineering surface area. I want those people empowered, regardless of what we call the role.
The centralized governance funkiness you're raising is worth a longer conversation. Embedding creates ownership but risks fragmentation. I don't have a clean answer for that one yet. 'preciate you, dude!
There's so much to say about this, but I think you're on the correct vector. I have been working independently to capture all of this on a the 'small data revolution' scale. I've been a practitioner at the OLAP and DW scale and ultimately when it comes to human decision-making that's all the data it ever takes.
For example, I recall when our folks were called in to solve a merchandizing problem for Gap. At the time, we had Sun's largest server, an E10000 working on a model with two, half-million member SKU dimensions. The problem was that the management philosophy of the company maximized store owner flexibility. So Manager A and Manager B could call T-Shirt X whatever they liked. We called it 'SKU Skew' and even though we only had 3GB of source data, finding the duplications was a nightmare. Just getting the aggregations was more than the machine could handle. The entire problem was one of semantics forced by company policy which we demonstrated gave a huge calculation problem. A few people there understood this and they had been kicking the can waiting for the next evolutions in compute and methodologies.
I've got all of these memories from a career of over 30 years designing and building through multiple generations of hardware, networks, storage, software, architectures and methodologies. It remains fascinating, especially in fields like aviation.
I have been bending, however as I alluded to the single expert worker, using myself as a model. I have been using AI to build a virtual practice based on the decisions I've made in my designs, choices and decision points. The best way I would describe this is what is the architecture you would give to a CIA analyst with a PhD in X who would not just RAG an existing book, but write the referential book. IE, what kind of system do you build for the guy who knows everything? It's working at a philosophical level about decision making rather than from a corporate paradigm.
The meta-knowledge working with new tools and techniques will generate the next generation of the consulting industry's approach to serving the next generation knowledge worker. Matters of governance will be key, but also self-governance.
----
My current personal project is building a tool that will help me manage trust for thousands of people, agents & services. It will hopefully answer questions like "Which of my 5,000 subscribers would I trust and therefore encourage to comment on my Substack, and which of them live near me so we could meet for a drink?"
This is a nice articulation of what an AE does and why it wasn’t a very effective role.
I’m increasingly convinced the problem it is solving is something a software engineer would do a better job of
I also think it will prove itself to be the case, likely in DEs and AEs pushing fixes upstream
Thanks for reading and for pushing on this.
I could see SWEs owning more of this, especially the upstream fixes you're describing. The closer the fix is to the source, the less cleanup downstream. And SWEs have a real advantage in understanding the systems generating the data.
Where I'd push back is on the business logic side. Knowing why a status column has five values is different from knowing which three count as "active" for sales vs. finance. That interpretive work requires proximity to the business, and it's what AEs specialize in. SWE is even broader than DE (which is arguably already a specialization within SWE), and analytics engineering carved out focused depth on exactly that layer. When you hand it to someone with a broader scope, they're either stretched too thin or they're doing analytics engineering under a different name.
Curious whether you've seen orgs where SWEs owned this end to end. I'd love to know what made that work.
Yes we are agreement on the business logic side.
I’ve seen it at a growth stage company where founding engineers maintain an incredible understanding of the analytics needs adjacent to the core app stack, but only because those analytics needs were so fundamental to the business (revenue driving). It does require some level of polymath inclination, and business sensitivity which are both rare enough to not expect it as a default.
But there is always the “important but not business critical” like finance that must run as best it can without consuming SWE time.
I think that dynamic now changes with the repricing of code. The AE can right code. They become SWE on the margin.
What I’m effectively saying is that AE doing dbt is a symptom caused by lack of SWE, and coding agents might meaningfully change this. In that case I think AE / DE / SWE overlap even more.
(Another argument is that analytics is a core capability of a good “Product Engineer”)
context layer? specifically ontology
we are putting together a course for ontology driven modeling
you can sign up to our edu newsletter here https://dlthub.com/events where we will announce it in a few weeks
more info on our blog. we think this is where the field is going in a few months, because the technology to do it arrived in december.
https://dlthub.com/blog/ontology
-adrian