Becoming data-driven: Lessons from tackling Durban’s water crisis

What is a data-driven organisation? What lessons can leaders learn from other organisations that use data to make step changes in performance? In this press release, we answer these questions by exploring how the municipality of eThekwini in South Africa is using data to address water scarcity issues through a transformation of people, processes and technology. Harnessing the data scattered throughout the municipality, AWS is helping eThekwini leaders solve a pressing problem, ensuring that all their residents have a reliable supply of water and helping them jumpstart a data-driven culture for further innovation. Through their story, we can see the benefits of being a data-driven organisation and how to get there. 

The challenges of becoming data-driven

According to recent research, 99% of blue-chip companies are currently spending money to apply data to derive better insights from their customers and businesses. And leveraging data is not simply a matter of buying hardware and software. Ninety-two percent of these organisations cite people, business processes and culture as their greatest challenges; only 8% say it’s technology. Unfortunately, only 24% have successfully created a data-driven organisation, even though there is many a reason to try: research shows companies that leverage data perform much better than their peers. MIT professor Erik Brynjolfsson found companies that embrace data-driven decision-making enjoy 5% to 6% higher output and productivity. Forrester also estimates data-driven businesses are growing at an average of more than 30% annually.

But even those companies had to start somewhere.

The Durban water crisis

eThekwini is a municipality in South Africa’s KwaZulu-Natal province. The province includes Durban, the third-largest city in South Africa with a population of 3.9 million. Like many places, Durban wrestles with ageing water infrastructure, a growing population as more people migrate to big cities, and the threat of not having enough water to support this growth.

Durban loses 50% of its municipal water supply to leaks from its ageing pipes, poor metering and theft. Many of the municipality’s 500 000 meters aren’t reading water volumes correctly because they are broken or out of calibration. Plus, they must be read manually, so readings are always dated. Without good data, officials don’t know where water has gone, can’t charge for it and can’t be sure they have enough to meet the future demands of the rising population.

In 2018, after a three-year drought, the water in Cape Town’s dams dropped to 26% of capacity. The city was three months away from Day Zero – the day the government would have to turn off taps in homes and businesses until the rains came. Cape Town ultimately avoided that fate with strict water restrictions and heavy rains that started in June. In Durban, officials foresaw the potentially catastrophic impact on the area’s citizens if they couldn't better manage the municipality’s water supply.

To do this, they needed to overcome five challenges around their water data:

1. They knew modern data tools and methods could help but weren’t sure how.
2. They had internal organisational boundaries that impeded collaboration.
3. They were too busy managing operational issues to make time for creating new solutions.
4. They lacked many analytical skills needed to interpret data.
5. They had a legacy technical architecture filled with fragmented and duplicate data that couldn’t combine data from diverse sources.

As Dr Sandile Mbatha, senior manager of Research and Policy Advocacy at eThekwini’s Office of Strategic Management, explained: “As a city, we generate tons of data. But it is stored in static formats and in various servers and hard disks, it’s dis-integrated and it’s not giving us the insights we need for strategic decision-making and long-term planning.”

The challenges described by Dr Mbatha are hardly unique; they are common to organisations large and small, in industries from manufacturing to retail, across the globe.

Five steps to become data-driven

We define a data-driven organisation as one that “harnesses data to drive sustained innovation and actionable insights to improve the customer experience”. The municipality of eThekwini took five crucial steps that put it on the path to becoming more data-driven – steps that other organisations with similar aspirations can take.

1. Build diverse teams and leverage their experience

In 2020, Dr Mbatha visited Arizona State University in Phoenix, Arizona, through the U.S. Department of State International Visitor Leadership Program (IVLP). His delegation met with the President’s Office, which referred them to the University’s Smart City Cloud Innovation Center (CIC) powered by AWS. The CIC uses Amazon’s innovation processes and AWS Cloud services to solve community challenges such as reducing homelessness and preparing students for the workforce. While there, Dr Mbatha explored with the CIC the idea of improving the municipality’s data capabilities, starting with water supply.

Like eThekwini, Phoenix is arid, and ASU is home to the Kyl Center for Water Policy – one of the world’s leading centres on water conservation. Because ASU is a public school with a philanthropic mission, the CIC provides its technology innovation services freely to communities in need far beyond Arizona.

To kick off the partnership, the CIC spent a few months working interactively with eThekwini’s water sector partners, data custodians and data users to understand the data they had and identify the problem they wanted to solve. This was the beginning of the kind of partnerships all organisations need to forge before embarking on a journey to become data-driven. At the outset, few companies have all the skills and technology they need in-house, and they can learn a lot from experts in other organisations and domains.

2. Work backwards from the customer

During the first conversations between the municipality and the CIC, the eThekwini team said their goal was to improve the water agency’s data capabilities. But that objective was too broad. The CIC and eThewikini teams worked together to narrow it down to a specific customer challenge: To reconcile the amount of water the municipality purchased with the amount it sold. That would help it figure out where it was losing water. Once it had that nailed down, the eThekwini team would know where to intervene to stem the losses. Ednick Msweli, Head of Water & Sanitation at eThekwini Municipality, told us: “We need R8.2 billion (USD $566 million) to replace our ageing water infrastructure, but we can spend only R500 million (USD $34 million) a year. With limited resources, data is critical to investing in the right places.”

Armed with a new focus, the CIC led a two-day virtual Amazonian Working Backwards engagement with the eThekwini officials. The team began by asking key questions to create a vision for a solution that would provide the most value to the municipality’s customers, including: Stakeholders in eThekwini’s Water and Sanitation Unit, South Africa’s Water Research Commission, the Center for Scientific and Industrial Research, water suppliers, and, of course, the end-users – the residents and businesses of eThekwini.

To aid in framing both the questions and the solution, the CIC brought a range of senior experts to the project, including Dominic Papa, vice-president for Smart State Initiatives at the Arizona Commerce Authority. “When AWS came with the opportunity to partner with eThekwini around the water challenge, I jumped at it because it's a challenge that's very important also to Arizona and our municipalities here,” Papa shared with us. His sponsorship of, and participation in, the project encouraged leaders, including from the water unit at eThekwini and the Water Research Commission, to openly share their data and dedicate their time to working on the problem. “In any organisation, whether that's public or private, approaching the challenge is difficult. What I think the Amazon Working Backwards process does so well, is it instills what I like to call public entrepreneurship in individuals. It drives them to think in a new way that might not be practiced every day in their organisations,” Papa said.

By the end of the workshop, the team had created a description of a solution it thought would best address the use case: A data platform to support the essential functions for water operations such as reporting, modelling, benchmarking and alerts. They called it SHANA, short for “ukushintshana”, which means “exchange” in Zulu.

Knowing what the eventual solution should look like, and following the Working Backwards engagement, the team started small and created a “minimum viable product”, a pilot that took about four months to develop. The pilot was designed to get early feedback and learnings from customers, with the goal of building on it and scaling it later. It helped that the 15 to 20 people who participated in the workshop had a broad range of skills, which is critical to getting good results. And by working backwards from the customer need, not forwards from the technology, the team had a much better chance of creating a product that added real value. “The Working Backwards process was a safe space to co-create with colleagues. It had a huge impetus on the data journey that we’ve taken as a city,” Mbatha said.

3. Push responsibility to the edge

The goal of the SHANA project was to centralise data from three sources: The municipal staff that buys the water (revenue management), those that sell the water (suppliers), and those responsible for maintaining the system (fault management). Each group stored different versions of the data in multiple siloed systems, so CIC data scientists started by setting up a repository to collect data from all of them.

The SHANA solution typifies the modern data community, comprising three roles: Data producers, platform, and data consumers. Within the eThewikini setting, the producers are the water suppliers, the finance team (running the numbers on the water sold), and the maintenance team, producing data on project backlogs. In a modern data community, the producers share their data to the exchange for the benefit of everyone.

The consumers are the individuals, teams, and machines that use the data. In the eThekwini project, they are the leaders and staff in various departments within the municipal Water and Sanitation Unit, including water suppliers and finance and maintenance teams who produce data but previously could only see their own piece of the total picture. Among the consumers are also ASU CIC data scientists who use the data to derive new insights.

Between the producers and consumers is the team responsible for operating the platform they use to collect, view and analyse the data. In the eThekwini project, the platform team comprised a small group of ASU CIC engineers who created the SHANA pilot, ensuring the right infrastructure and services are available.

In a modern data community, the IT department is no longer responsible for data ingestion, quality or insights. Instead, responsibilities are pushed deeper into the organisation, many of them delegated to data producers and consumers themselves, making it faster and easier for organisations to generate value from data.

4. Encourage organisation-wide data literacy

Before SHANA, eThekwini’s water unit, like many organisations, kept its data in spreadsheets. It was incomplete and in a format that made it difficult for anyone who wasn’t a data analyst to understand. Management knew there was a better way of doing things, but they had never seen it, so the staff couldn’t imagine what it might look like.

The SHANA project showed them what was possible. The water unit staff experimented with the platform and learned by doing, and in the process, became more data-literate. The staff was then able to show government leaders outside the water unit why and where budget was needed to invest in tools, like smart meters, to stem the water loss.

Having a use case like this is vital for getting buy-in within an organisation. And when projects produce outcome-focused data products, they help build a data-driven culture. In the SHANA project, eThekwini leaders saw what was possible through data. This, in turn, drove demand for more insights. It also created an appetite among senior leadership for future data initiatives.

According to Sibusiso Makhanya, deputy city manager for Trading Services at the eThekwini Municipality: “Raw data does not assist decision-making. This programme assisted a great deal in analysing the data so that information is ready for decision-making. People tend to support technology when they see the benefits. That’s what this programme has achieved.”

5. Build a modern data platform

Any organisation that wants to become data-driven needs to have a data architecture that is scalable, provides seamless data access and has unified governance. It should also be able to quickly access new capabilities for analysing and presenting data.

In the SHANA project, several features of the data architecture made this possible. First, the platform is cloud-based, so it could be set up immediately and scaled as needed. SHANA started with data from only three sources, but its data producers can add data from other sources in the future. For example, they can upload population data to track how fast cities are growing or contracting and assess how that relates to their water usage.

Second, the ASU CIC uses 95 AWS data services, so the team had access to advanced tools like Amazon QuickSight, which can read and consolidate data in different formats (like spreadsheets and flat files). The CIC data scientists used QuickSight, Amazon EC2 and Amazon S3 to combine and organise water data for easy integration, visualisation and interpretation.

Once the data is in SHANA, it can be operated on directly in the platform without having to copy or move it. When consumers find their datasets, they can use their preferred tools to perform their analysis. Built-in data standardisation gives SHANA users confidence in the integrity of the data because they know everyone is working off the same set of information. And because AWS Cloud services had just launched in South Africa, eThekwini’s data could be housed there in compliance with local data privacy regulations.

Results now and a platform for the future

Because of SHANA, eThekwini officials now better understand the municipality’s water losses and can take action to reduce them. The revenue management department can track its purchases without relying on suppliers to tell them how much water they’ve bought. For example, SHANA shows water consumption by suburb over time. When one suburb showed a dramatic decrease in consumption, the revenue team could determine that it was because of an intermittent supply (not because people mysteriously used less water).

SHANA also supports rapid proofs of concept for further sustainability initiatives like smart water metering. Once the team adds pipe network mapping and reservoir data to SHANA, it can pinpoint where to put smart meters to monitor consumption and detect losses. Then the municipality can build a proof-of-concept for a new water meter network to increase the accuracy and completeness of its water data.

Such a network will enable eThekwini’s water agency to identify where it’s losing water, why and how to fix it. As Dr Mbatha told us: “The Working Backwards workshop created a ripple effect among my colleagues, to start thinking more about the data and innovating with data as a strategic tool.” This is what all data-driven organisations aspire to, creating an initial compelling example of data-driven insights that inspires the rest of the organisation to use data more. It creates an organic momentum, linked to outcomes, that can often be more successful than top-down mandates from leadership to make the transformation.

Another benefit of SHANA is that it has become a springboard for other smart city initiatives. The water unit is now beginning to scale the data platform and extend it beyond water, leveraging SHANA for multiple city functions. Not only do they aspire to use SHANA to get a complete picture of water supply and use but also to understand workforce productivity, electricity consumption and traffic patterns. As Makhanya noted: “Right now, we have high water losses. But the moment we sort out the problem, SHANA will help us improve on other areas that will lead to better customer service. For instance, SHANA is assisting us a great deal towards better consolidation of electricity and water. And, in the process, introducing newer technology like smart metering.”

Leveraging data for your organisation

By working backwards to identify a compelling customer-focused solution, building skills, and leveraging agile cloud technologies, the eThekwini Municipality is gaining valuable insight into its water conservation challenges. The goal is to never turn off the taps. And it now has a platform ready to help it address other municipal challenges.

The data complexities confronted by eThekwini’s leaders are common across organisations. Like the municipality, other organisations have taken initiative and used data to solve problems, unlock new revenue opportunities, lower costs and risks and enhance their customer experience. As in eThekwini, they have understood that applying a modern methodology that incorporates mindset, people and process transformation is just as important as adopting leading-edge technology.

Top five challenges for data leaders and how they can overcome them

Challenge 1: Inadequate understanding of how data tools and methods can help.

Lesson for data leaders:

• Build diverse teams and leverage their experience via communication and sharing.
• Experts from other organisations and domains can help.

Challenge 2: Functional boundaries that inhibit collaboration.

Lesson for data leaders:

• Work backwards from the customer challenge.
• The process of working together breaks down boundaries.
• Leverage cloud technologies to remove the undifferentiated work from your teams.

Challenge 3: Time absorbed by operational issues leaves little opportunity for creating new solutions.

Lesson for data leaders:

• Push responsibility to the edge and remove the undifferentiated heavy lifting.
• This increases autonomy, ownership and speed.

Challenge 4: A lack of skills to interpret data.

Lesson for data leaders:

• Encourage organisation-wide data literacy.
• A minimum viable product teaches people what’s possible and leads to more initiatives.

Challenge 5: A legacy technical architecture that does not support insights.

Lesson for data leaders:

• Build an agile modern data platform.
• Then users can easily derive their own insights.
• The modern data platform can scale to meet evolving needs.

Learn more about how data can push your organisation forward.

Start and scale your data-driven organisation through an AWS Data-Driven Everything (D2E) or AWS Data Lab programme. For more information, reach out to your account executive.

Collaborate with an AWS Cloud Innovation Center.

About our guests

Steve Cooper

Worldwide Lead, Data and Analytics Program, AWS

Steve Cooper is the worldwide lead for Data-Driven Everything (D2E) at Amazon Web Services. D2E provides a use-case driven framework to help customers create a compelling data-driven vision, accelerate the flywheel for their most challenging use cases, build experience and scale with a “go-big” plan and six- to nine-month roadmap of priority projects. Prior to joining AWS, Cooper led Accenture's big data platform and AI/ML studio services, driving Accenture's transformation to an intelligent enterprise. He is passionate about bringing data solutions to meet the most compelling challenges, building data communities, sustainability and mentoring.

Ben Butler

Global Lead, AWS WWPS Cloud Innovation Programs

Ben Butler is the global leader for the World Wide Public Sector (WWPS) Cloud Innovation Centers (CICs) Program at AWS. He develops the programme to advance public sector innovation, digital transformation, workforce development and the value of cloud computing by solving innovation challenges for societal good. Previously, Butler was the VP of Marketing and Partners for REAN Cloud, a start-up that Butler helped become an AWS Premier Consulting Partner in 2015. Butler’s prior roles at AWS include Global Sr Marketing Manager for Big Data and HPC, a Senior Solutions Architect and AWS Certified Trainer. Butler was awarded the AWS Worldwide Public Sector Solutions Architect of the Year for 2012.