We've solved monitoring. So why are major incidents still taking hours to resolve?

Not long ago, one of the biggest challenges facing infrastructure teams was simply knowing when something had gone wrong. A server stopped responding. A network link failed. A UPS switched to battery. A cooling unit tripped. An application became unavailable.

For many organisations, the response was relatively simple. A monitoring system generated an e-mail or SMS and someone was expected to notice it and take action.

Things have changed considerably.

Today, infrastructure can tell us almost everything about itself. We collect information from servers, switches, routers, UPS systems, generators, cooling equipment, environmental sensors, industrial controllers, cloud platforms and thousands of other devices. We have dashboards for almost everything.

We have monitoring platforms, observability platforms, DCIM systems, BMS platforms, network management tools, IT service management systems and security platforms. We have never had more visibility into our infrastructure.

Yet major incidents can still take hours to resolve.

The reason is not necessarily that we cannot detect the problem. The problem is what happens after we detect it.

We have become very good at finding problems. We have not been as successful at co-ordinating the response.

Monitoring has matured. Response hasn't.

Walk into almost any enterprise today and you will find multiple monitoring platforms working side by side. Network management systems monitor switches and routers. DCIM platforms monitor power and cooling. Building management systems monitor environmental conditions. Cloud platforms monitor applications and services. Security platforms monitor networks, endpoints and access. UPS systems monitor power conditions. Generators monitor their own operating status.

Each system does its job.

The problem starts when something goes wrong across more than one system.

Consider a network switch at a remote site. The switch fails, and the network monitoring system reports that the switch is unreachable. The servers connected to it report connectivity problems. Applications begin reporting timeouts. A UPS or power monitoring system may report a change in load. Environmental monitoring may report that several devices have stopped communicating. The service desk may receive additional alerts.

Suddenly, one physical failure has become 10, 20 or even more notifications.

Every monitoring system has done exactly what it was designed to do. But now an engineer has to determine what is actually happening.

Which alert is the cause?

Which alerts are symptoms?

What is the impact?

Who owns the problem?

Who needs to respond?

How urgent is it?

And, most importantly, what needs to happen next?

This is where organisations start losing time.

We often call this alert fatigue. But I think the problem is bigger than that.

We have become very good at generating alerts. We have invested considerably less effort into deciding which alerts matter, who needs to respond and how that response should be co-ordinated.

Finding the problem has become easier. Managing the response hasn't.

Twenty years ago, many organisations managed a relatively small number of data centres and branch offices. Infrastructure was more centralised. The people managing it were often relatively close to the equipment.

Today, infrastructure is everywhere.

Banks operate thousands of branches and ATMs. Retailers depend on technology across hundreds or thousands of stores. Utilities operate substations and infrastructure across enormous geographic areas. Telecommunications providers maintain thousands of network sites. Mining companies operate remote facilities where the nearest engineer may be several hours away. Manufacturing companies increasingly connect operational technology with enterprise IT.

And almost every organisation now has some combination of cloud, hybrid infrastructure, edge computing and remote operations.

This has changed the problem.

When infrastructure was concentrated in a few locations, sending someone to investigate was relatively straightforward. When infrastructure is distributed across hundreds or thousands of locations, the response becomes much more complicated.

If a remote site goes down, someone needs to know what happened. Someone needs to own the incident. Someone needs to acknowledge it. Someone needs to determine whether a technician is required. Someone needs to establish whether the problem can be resolved remotely. And if the primary engineer doesn't respond, somebody else needs to be contacted.

Every minute spent figuring this out adds to the outage.

The technical failure may have happened in seconds. The operational response can take much longer.

This is something I have seen repeatedly in infrastructure environments.

The actual fault is not always what takes the longest to resolve. Sometimes the equipment can be repaired in minutes. The delay happens before anyone gets to the equipment.

An alert is generated. Nobody sees it. Someone eventually sees it. They aren't sure who owns the equipment. They send an email. They wait. Someone else is contacted. They need access to the system. The normal network path is unavailable.

Now someone has to work out how to access the equipment.

By the time the engineer starts troubleshooting, a significant amount of time has already passed.

The monitoring system may have detected the fault within seconds. But detection is only the beginning.

A useful way of looking at incident response is:

Detect → Decide → Communicate → Access → Recover → Verify

If any one of those stages introduces unnecessary delay, the incident takes longer to resolve.

This is why simply adding another monitoring platform does not necessarily solve the problem. The bottleneck may no longer be detection. The bottleneck may be everything that happens afterwards.

Most infrastructure architectures separate responsibilities. Monitoring systems collect information. Analytics systems process information. IT service management systems manage incidents and changes. Security platforms enforce security controls. Communication systems deliver messages. Remote access systems provide connectivity.

All of these are important.

But there is often a gap between the systems detecting the problem and the people responsible for dealing with it.

Monitoring tells us that something has happened. It may even tell us how serious it appears to be. But somebody still has to decide what happens next.

That decision is often left to engineers who are already dealing with hundreds of alerts, tickets, messages and requests.

This is where I believe infrastructure architecture needs another layer.

At Tektonics, we refer to this as the operational resilience layer.

It is not another monitoring platform. It is not intended to replace the monitoring systems an organisation already has. It sits between those systems and the operational teams responsible for responding.

Its purpose is to take technical events and turn them into co-ordinated operational action.

That means receiving information from different systems, removing unnecessary duplication, applying rules and context, determining who needs to know and making sure that the incident is followed through to resolution.

The goal isn't to send more notifications.

The goal is to send fewer, more meaningful notifications.

One of the first things an operational layer needs to do is understand that not every alert represents a separate incident.

If 20 devices become unreachable within a few seconds, we should not automatically create 20 independent incidents. There may be one problem causing all of them.

It could be a failed switch. It could be a power problem. It could be a network circuit. It could be a router. It could be something at the site itself.

The important thing is to understand the relationship between the events.

There is a big difference between suppressing alarms and understanding them.

Suppressing alarms can hide problems. Correlation should provide context.

Instead of presenting an engineer with:

Device A unreachable

Device B unreachable

Device C unreachable

Application unavailable

UPS communication lost

Environmental controller unavailable the operational system should be able to present something closer to:

Critical site connectivity failure – multiple downstream systems affected.

That is a much more useful starting point for an engineer.

The next part is understanding that not every alarm has the same importance.

A failed temperature sensor in an office is not necessarily equivalent to a cooling controller failure in a critical data hall. A network device in a test environment is not equivalent to the router supporting a critical production facility.

A UPS operating on battery at a remote telecommunications site may require immediate attention because there may only be a limited amount of battery runtime available.

The alarm itself doesn't tell the whole story.

We need to know:

• Where is the equipment?
• What does it support?
• Who is responsible for it?
• What services depend on it?
• Is the site critical?
• Is there redundancy?
• Is maintenance currently taking place?
• How long can the equipment continue operating?
• What is the likely impact?
• Who needs to respond?

This is where operational context becomes important.

It also allows organisations to capture knowledge that has traditionally lived in the heads of experienced engineers. The engineer who has worked on a particular site for 10 years often knows exactly which alarms matter and which ones don't.

The challenge is making that knowledge available to the wider operational team.

Another area that deserves more attention is communication.

Sending an SMS or e-mail is not the same as managing an incident.

The important question isn't:

"Did we send the message?"

The important question is:

"Did the right person receive it and take responsibility?"

Those are very different things.

A proper operational communication process should support acknowledgement and escalation. For example:

1. A critical event is detected.
2. Related events are combined.
3. The incident is classified.
4. The responsible engineer is notified.
5. The engineer acknowledges the incident.
6. If there is no acknowledgement, the incident escalates.
7. The next person is notified.
8. Management or the service desk can be informed if required.
9. The incident remains tracked until recovery.
10. The response is recorded for later review.

The communication channel can change depending on the environment.

It might be SMS. It might be email. It might be a collaboration platform. It might be another messaging system.

The important part is not the channel. It is the workflow around the communication.

An organisation should know whether the notification was delivered, whether somebody acknowledged it and whether the escalation process worked.

This is where operational resilience becomes more than communication.

Imagine a critical site loses its primary network connection. The monitoring system detects the outage. The alert is sent. The engineer receives it.

But the engineer cannot access the equipment because the management network uses the same connection that has just failed.

Now the organisation knows exactly what is wrong. It has communicated the problem. But it cannot fix it remotely.

This is one of the reasons secure out-of-band management is so important.

An independent management path allows authorised engineers to access critical infrastructure even when the primary production network is unavailable.

This can be extremely valuable during:

• ISP failures
• Routing problems
• Firewall changes
• Network configuration errors
• Equipment failures
• Remote site outages
• Major infrastructure incidents
• Cyber security incidents

The same principle becomes even more important during a security incident. Organisations may need to isolate production networks. That is often the correct response.

But isolation can also remove the management path engineers need to investigate and restore the environment.

A resilient architecture therefore needs to consider both sides of the problem. We need to be able to isolate systems when necessary. We also need to maintain a secure way to manage and recover them.

Detecting an outage is valuable.

Maintaining the ability to respond during that outage is far more valuable.

The move towards distributed infrastructure makes this problem harder.

A central data centre is relatively easy to reach. A remote site may not be.

A technician may be hundreds of kilometres away. The site may be unmanned. Physical access may require security clearance. The nearest spare equipment may be several hours away.

In those environments, remote operational capability becomes critical.

Consider a remote utility site.

A power event occurs. The monitoring system detects it. The site then stops communicating.

The operations team needs to determine:

• Did the site lose power?
• Did the communications equipment fail?
• Is the equipment still operating?
• How long can it continue operating?
• Is there another communications path?
• Can the equipment be accessed remotely?
• Does somebody need to travel to the site?
• Who is responsible?
• At what point should the incident be escalated?

These are not monitoring questions.

They are operational questions.

Over time, I have found that resilient operations consistently depend on five capabilities.

1. Detect

We need to know what is happening.

That means collecting reliable information from IT, OT, facilities, power, environmental, network and other infrastructure systems.

Detection remains the foundation.

But it is only the first step.

2. Decide

We need to determine what actually matters.

This means filtering noise, correlating related events, applying operational context and understanding potential impact.

The objective isn't simply to reduce the number of alarms. It is to reduce the amount of unnecessary decision-making required from engineers.

3. Communicate

The right people need the right information at the right time.

Critical incidents need acknowledgement. If there is no response, they need to escalate.

Communication needs to be measurable.

We should be able to answer:

Who was notified?

When were they notified?

Did they acknowledge the incident?

Who did it escalate to?

When did someone accept responsibility?

4. Recover

Once we know what is wrong, we need a way to fix it.

That means secure remote access, out-of-band connectivity, appropriate tools, documented procedures and, where appropriate, automation.

Detection without recovery capability gives us visibility without resilience.

5. Learn

Every major incident should improve the next one.

We should measure:

• Time to detect
• Time to acknowledge
• Time to assign
• Time to access
• Time to recover
• Escalation performance
• False alarms
• Recurring incidents
• Communication failures
• Resolution performance

The purpose isn't simply to create another report.

It is to find where the organisation is repeatedly losing time.

There is no doubt that AI will become increasingly useful in infrastructure operations. It can help identify patterns, compare events and assist engineers with troubleshooting.

But I don't believe the answer is simply to generate more intelligent alerts.

If we already have too many alerts, making those alerts more sophisticated doesn't necessarily solve the operational problem.

The more useful application is helping determine what should happen after an event occurs.

For example:

• Is this a new incident?
• Is it related to another event?
• What systems are affected?
• What is the likely cause?
• Who owns the equipment?
• How critical is the site?
• Who should be contacted?
• What happened the last time this occurred?
• Has someone acknowledged the incident?
• Has the incident escalated?

These are practical operational questions.

Technology should help answer them without adding another layer of complexity for the engineer.

I think we also need to change how we measure infrastructure operations.

We often measure whether monitoring systems are available. We measure whether alerts are generated. We measure whether messages are delivered. We measure whether tickets are created.

All of these metrics are useful.

But they don't necessarily tell us how quickly the organisation can recover from an incident.

The real measurement should be end-to-end.

From the moment the problem occurs until the service is restored.

How long did it take to detect?

How long did it take to understand?

How long did it take to reach the responsible person?

How long did it take to establish access?

How long did remediation take?

How long did verification take?

This gives us a much better picture of operational performance.

The question should not simply be:

"Is our monitoring working?"

It should be:

"Can our organisation respond quickly when something goes wrong?"

I believe operational resilience needs to become an architectural discipline.

We already design networks for availability. We design security architectures around protecting systems. We design applications for performance and scalability.

We should also design infrastructure around how it will be operated when something goes wrong.

That means considering five things from the beginning:

Detection

How will we know something has happened?

Decision

How will we determine what matters?

Communication

How will we reach the right person?

Recovery

How will we access and restore the infrastructure?

Learning

How will we improve the next response?

This becomes particularly important in critical infrastructure, utilities, telecommunications, data centres, mining, manufacturing and any environment where assets are geographically distributed.

The more distributed the environment becomes, the harder it is to rely on manual co-ordination.

We have spent decades building increasingly sophisticated ways of seeing infrastructure. That investment has delivered enormous value.

We can now see more of our infrastructure than ever before.

But visibility is not the same as resilience.

The next challenge is co-ordination.

The organisations that recover fastest will not necessarily be the organisations with the most monitoring tools. They will be the organisations that have built a reliable operational process around those tools.

They will be able to move quickly from:

Something happened

to

We know what it is

to

The right person has been notified

to

Someone has accepted responsibility

to

We can access the affected infrastructure

to

The service has been restored

to

We understand what happened and how to prevent it happening again.

That is operational resilience.

Monitoring tells you what is wrong. Operational resilience determines what happens next.

Monitoring is still essential.

We need good monitoring. We need good telemetry. We need good network management. We need good DCIM. We need good IT service management. We need good security.

But these systems should not operate as isolated islands.

They need to work together.

At Tektonics, we believe the operational resilience layer provides a way of connecting these existing systems to the people responsible for responding to them.

It is about taking the information already being generated by infrastructure and turning it into something operationally useful. It is about reducing noise. It is about applying context. It is about getting the right information to the right person.

It is about acknowledgement and escalation. It is about maintaining access when the primary network is unavailable. And it is about creating a record of what actually happened during an incident.

Most importantly, it changes the question we should be asking.

Instead of asking:

"Did our monitoring detect the problem?"

We should be asking:

"How quickly can we respond when something goes wrong?"

Because detecting a failure in five seconds is impressive.

But if it still takes two hours to get the right person involved, establish access and begin recovery, then we haven't solved the operational problem.

We have solved monitoring.

Now we need to solve what happens after the alarm.

That is where the next generation of infrastructure resilience will be built.

About the author

Tshepo Chiloane is the Founder and Managing Director of Tektonics Systems, specialising in operational resilience, intelligent edge computing and critical infrastructure management. With more than two decades of experience across enterprise IT, operational technology, utilities, telecommunications and data centres, he works with organisations to improve operational response through intelligent communications, secure remote management and resilient infrastructure design.