Data teams shrinking, but their impact continues to grow

Data analytics hasn’t lost its appeal.

---

Just a decade ago, data scientist roles were considered the sexiest jobs in the world. It was revenge of the geeks on a major scale. Virtually every business wanted to tap into the advantages of data-driven insights, seen almost as a magic wand waiting to transform a frog into a prince, if not a unicorn.

As with every shiny new thing, the novelty has worn off, but data analytics hasn’t lost its appeal. Business leaders still value data insights, but now they are seen as a basic need for every organisation, and as with all operational departments in times of austerity, there’s less eagerness to pour money into data science departments.

Data analytics is having something of an adolescent moment, going through a transitional period where companies want to do more with less. No one is about to give up on their data-driven decision-making, but at the same time, they want to cut budgets and trim down data science teams.

Here’s what this trend could mean for data scientists, and for data analytics as a discipline.

The initial explosion of big data caused the number of data-related positions across industries to mushroom. It seemed that every company was looking for database managers, data analysts, analytics engineers, data scientists, data visualisation experts – if the title began with “data”, it was in demand.

However, that trend is now reversing. Data-related roles have partly become a victim of their own success, with a wide range of positions now participating in the process of producing data insights. But we're also seeing a change in attitude towards data management.

At the same time, organisations are refocusing on using data to drive business growth, investing in platforms, methodologies and tools that increase productivity and deployment rates for data management. The tighter economy is pushing companies to cut budgets and shrink data team sizes, with dbt Labs’ 2024 State of Analytics Engineering survey revealing that data team head counts are more likely to shrink than to grow.

It would be a mistake to think that this shift indicates a lack of reliance on data. On the contrary – recent research from Oracle shows that 97% of business leaders want data to help them make decisions, and a Gartner study predicts that two-thirds of B2B sales organisations will move from intuition-based to data-based decision-making by 2026. Pressure on budgets also means that businesses have to optimise their strategy, which requires even more reliance on data for predictive forecasting, model simulations, etc.

One solution is to use self-service GenBI solutions to increase the accessibility of data insights across the company, so that it’s not only the preserve of data science teams. One study reported that 78% of US business leaders see data literacy as the most important skillset for employees’ daily operations.

“Successful business intelligence today features democratised access to data, distributed across the organisation in an open but governed manner,” says Omri Kohl, CEO of Pyramid Analytics, a platform that supports data conversations using natural language.

“There is greater accessibility to insights, driven by data through purpose-built user interfaces that non-technical employees can use,” he continues. “Today’s successful BI removes barriers to employee adoption and increases the likelihood that employees across the organisation will access insights from the same data sets.”

Accessibility for non-techies doesn’t remove the need for data scientists, but it does transmute their role within the company.

“It’s much more intuitive for someone to ask AI a question than it is for them to develop a dashboard,” observes Megan Dixon, VP of data science at Assurance IQ. “AI can create data visualisations and answer basic questions, meaning teams can make data-driven decisions or measure progress without relying on constant support.”

But like all AI tools, today’s cutting-edge GenBI solutions still need to be told how to do things and supervised for bias, flaws or hallucinations. The way we collect and access data might be changing, but skilled professionals are still vital to operate and oversee AI-powered data tools.

This is leading to a diffusion of data experts to collaborate across business units and departments. Data scientists are now more likely to work as an addition to an existing team, like marketing, product development or finance, rather than in their own data-infused silo.

The twin factors of a need for data insights and a rise of self-service GenBI tools are giving rise to a new set of archetypes. On the one hand, larger companies in particular are seeking out specialised talent who are expert at one specific element of data analysis, to enhance the general responses delivered by GenBI solutions.

On the other hand, Alteryx research shows demand for data generalists who can manage the entire data stack, albeit with less sophistication. Close to three-quarters of respondents are seeking multi-skilled employees, while only 28% want those with just one skill.

With the right mix of generalists and specialists, even a very lean data team can still have a strong impact on company strategy and culture.

Organisations of all sizes and in all industries strongly rely on data to drive decision-making and strategy. While the number of data professionals is still in flux, and their position is still adjusting, there’s no chance that the data management role will go away.

Data experts will remain in demand to enhance the impact of self-service GenBI solutions, and enable their company to maintain a competitive edge at lower costs.