The rise and fall of data architecture

There is a long way to go to regain trust in architecture. For this to be possible, we need business to change how it perceives data architecture.
Read time 5min 10sec

I have been an architect for 15 years now, specialising in data and solution architecture. In the beginning of my career, architecture was a well-respected discipline, revered even.

The industry has changed over the years though. We have encountered new technologies, such as cloud and big data, and explored new development methodologies, such as agile and micro-services.

Somewhere along this journey, based on my own experiences, I feel that we have lost our reverence for architecture. I have seen many architects become disconnected from their true calling and purpose.

The result? Delivery teams not hiring architects, or entire companies removing the architect job title from their vocabulary. And in some cases, architects ruling and dictating, rather than serving and enabling.

Where did it go wrong?

For many of us, myself included, we did not even realise it was happening. My own moment of self-realisation came at the behest of a client of mine. I had been tasked with the formulation of the roadmap and architecture for the next five years.

I was having great fun putting it all together, lots of great pictures and bundles of rules and standards. And it all came together in a glorious, epic document!

It didn't end there. Once completed, I got involved in further amendments. I was also drawn into meetings with the rest of the enterprise architects for other initiatives. Meetings and design sessions daily, and conferences at remote locations every month. All very important, and all very necessary.

Or so I told myself.

And then my manager came to me quite concerned. All told, I had been away from the team for close to a year by this stage. I was essentially working as a remote solution architect − one that was only creating documents, and not helping the team in their day-to-day deliverables.

What did it boil down to? This process was not adding value.

The watershed moment

I had a great relationship with this client, so we had a serious heart to heart, and meeting of minds. In this discussion, I realised that as an architect, I had become too far removed from my team.

I was no longer current with the technology and programming languages. I was unfamiliar with their challenges and unaware of the burdens that governance and architecture placed on them.

I have seen many architects become disconnected from their true calling and purpose.

I realised that as an architect, I needed to be visible on the floor. I needed to engage with the team on a regular basis. Most importantly, I had to once again be responsible for delivery. By taking ownership of delivery, I was once again accountable. You will be amazed at what this does to your perspective, when you are accountable for something.

Having re-joined the delivery team again, I realised what I as an architect had been doing wrong. I was making rules and standards that I, myself, did not have to follow. Or suffer the burden of.

I became aware of various flaws in my architecture and processes. Issues that slowed down delivery and hampered development. Issues caused by me being out of touch with my developers, and the approach they had to follow.

You only realise the impact of rules if you yourself are bound by them. By being bound to my own rules, I was quickly able to see that some of it was not workable and placed too much burden on the delivery team.

Giving up control

Of course, there was a downside. I could not do it all. We had a team of close to 40 developers and analysts working on different platforms and technologies.

As an architect, I was all about control, trusting in the rules and processes, and not the people. So, it was anathema to me to give this up and bring other people into my ‘kingdom’.

We had to compromise, and I had to let go. We brought in a new architect, to take over ownership of a large swathe of my original domain. I brought up and trained team leads to fulfil some of my functions.

This allowed me to have a very clear focus on a stream of the delivery, one that I was responsible for. I then worked with the new architect and team leads.

The result? I ended up developing a fantastic relationship with the new architect. We both learned an enormous amount from each other and leveraged each other’s strengths and bolstered our weaknesses.

We now had more time to attend some of the necessary enterprise meetings and initiatives. We were able to do this, without risking our newfound involvement with delivery.

Our teams were also benefitting. Since the architects were now available, decisions were instantaneous. Developers would raise problems in a meeting and the architect would solve it in the same meeting. As architects, we adapted and responded in-transit and our delivery gained momentum!

The way forward

I believe that the role and approach of architecture is changing. Architecture should now be based on the concept of servant leadership. We are changing how we engage with teams. Where once we were remote, we are now actively engaged and this is having a noticeable benefit to our delivery. However, the battle is not yet won.

There are many who have lost trust in architecture. We have a long way to go to regain that trust. For this to be possible, we need business to change how it perceives architecture.

Business must allow for the appointment of more architects, allowing architects a narrower focus. We need more architects discarding the ways of the past, and embarking on this new approach of servant, and responsible leadership.

Julian Thomas

Principal consultant at PBT Group

Julian Thomas is principal consultant at PBT Group, specialising in delivering solutions in data warehousing, business intelligence, master data management and data quality control. In addition, he assists clients in defining strategies for the implementation of business intelligence competency centres, and implementation roadmaps for a wide range of information management solutions. Thomas has spent most of his career as a consultant in South Africa, and has implemented information management solutions across the continent, using a wide range of technologies. His experience in the industry has convinced him of the importance of hybrid disciplines, in both solution delivery and development. In addition, he has learned the value of robust and flexible ETL frameworks, and has successfully built and implemented complementary frameworks across multiple technologies.

See also