Has software development become too complex?

Jaco van der Merwe, CEO of DVT.

---

When I started my career in IT, I was fortunate enough to build and deploy several custom enterprise software solutions. In my first proper job as a software developer, I coded in Delphi using an Oracle database. It was a relatively large system with a fairly complex data structure, yet with a very small team, we were able to get it delivered into production in less than six months. These days, similar-sized systems take considerably longer to develop, often with much larger teams than before.

It is not as if our small team of Delphi developers back then was that special. I saw the same with teams of Visual Basic developers, Oracle Forms developers and many others.

So why does it take so long to build software? Or more appropriately, a question many of our clients ask: why is software development so expensive?

With the growth of available libraries, cloud-based services, open source frameworks and advances in software development tools, one would be forgiven for assuming the time taken to develop software would decrease, not increase, as would seem to be the case over the last decade or so.

I can't help but wonder if software development has become too complex? The reality is that software development today is orders of magnitude more complex than one or two decades ago.

Why has software development become so complex?

The short answer to why software development has become so complex is because of the additional demands we have from software these days compared to one or two decades ago.

These additional demands from our software were first described by computer architect Fred Brooks in 1986, in a paper called "No Silver Bullet", in which he observed that software engineering wasn't producing the same productivity gains compared to hardware engineering. Brooks described that in software development, there were two major barriers to overcome: essential complexity and accidental complexity.

Essential complexity vs accidental complexity in software development

Essential complexity is, in essence, the business problem that needs to be solved.

It describes what the system must do, or the "scope" of the business problem. Although there may be some flexibility in the scope of the business problem, we must solve the essential complexity to have a system that solves the business problem. Although the actual business problems have changed over the years, they have not become more complex. Essential complexity is not our challenge.

Accidental complexity is the complexity developers unintentionally make for themselves as a result of trying to solve a problem. For example, if a team of developers is highly skilled and experienced in using Angular as a Web framework, but that same team needs to now develop a system using React, it adds considerable accidental complexity to the software development project. Another example could be a team that is used to developing Web-based solutions now having to develop a mobile device interface.

The hard life of a software developer

Learning a new development framework or solving the challenges of producing software for a new type of device is not exactly a new challenge in software development. I have learned to code in several programming languages during the years of developing software. So, what exactly changed from my simple two-tier, client/server architected software development days to today?

The answer is that the demands we have from software today have resulted in additional accidental complexity in software development that has gone through the roof.

Here are just some of the accidental complexities that a developer has to contend with today that weren't there 10 or 20 years ago:

Multiple devices and platforms

Developers today need to develop software that runs on multiple devices and platforms on form factors that include desktops, laptops, tablets, phones, watches and IOT devices. Although there are various frameworks out there to assist with this challenge, it still adds considerable accidental complexity to a software development project.

The first time I experienced this complexity was when my Microsoft ASP-based Web system ran perfectly in Internet Explorer but completely misbehaved in Netscape!

Containerisation

Containerisation is great for encapsulating or packaging up software code and all its dependencies so that it can run uniformly and consistently on any infrastructure, but boy does it add complexity to a project! The first time I saw this was with the introduction of J2EE.

When I first started working with J2EE, I understood the value of an application server but I saw just how much the speed of getting software in production decreased.

User experience (UX) and user interface (UI) design

As little as 10 years ago, most developers had not heard of UX/UI design. It did not exist as a software development practice.

It meant that software was generally functionally great but sucked from a user experience or user interface point of view! Today, great software has professional UX/UI designers behind it – to create designs for the software that makes it super easy for users to use. UX/UI design is essential to software development but implementing UX/UI design adds more complexity.

Cyber security

Today, all systems are vulnerable to cyber security threats. It means developers need to design with cyber security in mind and have their software tested for security vulnerabilities.

More complexity.

Source control

Source control is by no means new to software developers, but it hasn't been this widely adopted in the past or dealt with the complexities of multiple code branches and the complexities of continuous deployment and integration.

The ability to respond to changing business demands

Continuous deployment and integration and more regular deployments help to create software that can respond to business needs quickly and deliver business value faster. No debate.

Just a lot more complex.

Scalability

Software that can handle tens of thousands of users sometimes nearly concurrently was unheard of two decades ago. Add to that the ability to operate with virtual hardware in cloud-based environments that scale up or down as the demand requires, and you have some very impressive software. That very impressive software requires developers to deal with a lot of added accidental complexity to understand and work with the tools and software development architectures and design patterns to support that level of scalability.

Defect-free software

These days, all users expect defect-free software. It results in a momentous effort to reduce software defects. Test-driven development, test automation, performance testing, etc, all add to higher quality software, but certainly has an impact on how quickly software gets deployed to users.

New technology

From my experience, each time a software development team starts with a new project, they have to deal with and learn a large portion of new technology (languages, frameworks, design patterns, tools, vendor software, etc).

Developers love to learn new tech, but the pace at which it is rolled out presently slows down software development.

How to deal with the complexity of software development

There are many more examples of how software development has become complex than those cited above. Does that mean software development will become ever more complex to the point where it is not feasible or worth the cost?

I believe that we are in a time where software development complexity is increasing but the current complexity will be reduced through the development of new technology. Low-code software development tools are already making it easier to solve certain types of business problems and AI will automate some of the complex tasks that developers are faced with.

But for now, spare a thought for the ever-growing accidental complexity that software developers must deal with to meet the demands we have for software.

Finding a software development team that knows when to reduce complexities for greater efficiency, productivity and quality means looking towards the right people for guidance. Look to people who have had multiple successes, not just one-off hits. The company also needs to have established and good remote practices and reliable technology distribution across the engineering team.