Field report: secondary circuit validation

Onsite attendance to observe and validate an intermittent secondary WAN connection. The production network was fully operational on arrival. No work to the primary circuit, virtualisation estate or internal switching was included in the requested scope.

Assignment record

08:42 — Arrival and baseline

The site was operating normally when the engineer arrived. Users were authenticated, internet access was available and no active alarms were visible on the production firewall. Both VMware and Hyper-V management consoles reported their hosts online. The local Active Directory domain controllers, DNS, DHCP, file services and line-of-business virtual machines were responding within their normal tolerances.

The sole outstanding concern was the backup WAN interface. Historic firewall events showed brief carrier loss followed by successful recovery, although the timestamps did not align precisely with the carrier portal. As the secondary connection was not carrying traffic, a controlled reset was considered non-impacting.

09:07 — Secondary circuit reset

The backup interface was administratively reset and the carrier termination equipment was restarted. Link state returned, dropped, and returned again without completing a stable failover health check. The behaviour was consistent with the reported intermittent condition and therefore confirmed that attendance had been necessary.

A second reset produced a different sequence of indicator lights but no materially improved result. The primary connection continued to carry all production traffic throughout this stage.

09:51 — Substitute firewall assessment

A replacement firewall was retrieved from the engineer’s van to separate the carrier service from the installed firewall hardware. The unit was unused stock of the nearest available model. On boot it required online activation before exposing the full diagnostic interface. It also reported that its installed firmware could not import the production configuration revision.

The backup service could not provide the reliable upstream connection required to activate or upgrade the replacement unit. The remaining option was to present the known-working primary connection to it briefly. To minimise disruption, this was deferred until the office lunch period, when the majority of users would be away from their desks.

12:04 — Lunchtime validation

After confirming that desk activity had reduced, the primary carrier handoff was moved from the production firewall to the replacement unit. The substitute firewall obtained a connection, completed activation and began downloading its required firmware. External reachability was confirmed briefly from its diagnostic page.

During the download, the primary connection began alternating between connected and disconnected states on the replacement firewall. The firmware transfer restarted twice and the management session became unavailable for approximately forty seconds. As this introduced uncertainty into a previously known-good circuit, the validation was stopped and the primary handoff was returned to the installed firewall.

12:19 — Primary service does not return

When reconnected to the production firewall, the primary interface reported physical carrier but did not restore its upstream session. Reseating the handoff caused the carrier state to disappear entirely. The carrier equipment still displayed its normal service indication, while the firewall alternated between negotiation and timeout.

The backup circuit was then selected to preserve external access, but it reproduced the intermittent condition for which the visit had originally been arranged. At this point both WAN paths were unavailable, although local servers, authentication and workstation access remained operational.

12:33 — High-availability state change

The active firewall was restarted to clear a suspected interface-driver state. Its peer correctly attempted to assume the active role. The peer’s configuration, however, contained an older internal interface map and identified two current VLAN interfaces as pending objects. This difference had not been visible while the appliance remained passive.

The virtual gateway addresses moved between appliances several times while the pair attempted to agree an active member. Internal clients began losing access to DNS and application networks. The firewalls eventually settled with the original unit active, but the internal core retained address-resolution entries associated with both appliances.

12:58 — Core adjacency investigation

A refresh of the core switch stack was undertaken to eliminate the stale gateway and virtual MAC state. One stack member returned normally. The second member declined to rejoin because its firmware revision differed from the elected master by one maintenance release. The mismatch was pre-existing and had not affected service before the refresh.

Links connected to the unavailable member were redistributed across the surviving stack member. Several aggregated uplinks came back with only half their expected members. The VMware distributed switch interpreted the changing uplink state as repeated path failure, while the Hyper-V switch-embedded teams reported inconsistent reachability across VLANs.

13:21 — Storage and cluster impact

The site’s storage paths shared the affected switching fabric. Loss and restoration of individual trunks caused the iSCSI sessions used by the VMware hosts to enter an all-paths-down condition. Hyper-V cluster shared volumes moved into redirected access before the cluster witness became unreachable.

Virtual machines initially remained visible in their respective management consoles but stopped responding to network checks. Automatic recovery attempts increased storage and cluster traffic on the remaining links. Two VMware hosts then isolated themselves, and the Hyper-V nodes paused a number of guests to protect disk consistency.

13:47 — Directory services unavailable

One Active Directory domain controller was hosted on VMware and the other on Hyper-V. The VMware controller was inaccessible with its datastore path unavailable; the Hyper-V controller had paused with its cluster volume. DNS registration, Kerberos authentication, DHCP policy processing and access to centrally managed workstation profiles consequently ceased within the same interval.

Existing user sessions began failing as cached service tickets expired. Workstations could no longer resolve internal services or obtain a usable lease after reconnecting. By 14:05, all occupied desks were reporting loss of network access despite the lunch period having formally ended.

14:16 — Recovery materials inaccessible

The current switch-stack configuration backup was recorded as being held on the infrastructure file share, which was hosted inside the unavailable VMware estate. A removable copy in the communications cabinet was located, but its label referred to a switch model retired in 2022. The cloud configuration repository could not be reached because neither WAN circuit was operational.

An attempt to restore the second switch member independently was postponed when its console requested an image file that was also stored on the inaccessible file share. Meanwhile, the firewall pair resumed exchanging state and copied the presently incomplete interface status between members.

15:02 — Full site outage

The remaining VMware management path was lost when the surviving core member placed the storage trunk into a protective inconsistent state. Hyper-V cluster communication then failed completely and its remaining guests entered a stopped or isolated condition. Environmental monitoring remained reachable only from its directly connected display.

At 15:18 the mini datacentre had no available production virtual machines, no Active Directory service, no functional DNS or DHCP, no working primary or secondary internet route, and no authenticated workstation connectivity. The original intermittent backup-line investigation was therefore suspended pending a wider recovery activity.

15:24 — UPS switching anomaly

As the virtualisation hosts continued attempting local recovery, the changing server load coincided with an unexpected state transition on the communications-room UPS. Its display reported that an inverter-to-static-bypass transfer had been requested, cancelled and then completed. No transfer had been initiated through the management interface.

The load remained powered, but the UPS was now supplying the room through bypass rather than conditioned inverter output. The network card that would normally provide detailed event history was unreachable because it depended on the failed switching and directory environment. The front-panel log contained only a switching error and an event number not present in the manual kept onsite.

15:27 — Return to inverter operation

Operation on bypass left the entire datacentre load exposed to raw mains without battery protection. The onsite manual listed a front-panel transfer to inverter as a standard operator function, and the display reported the inverter as available. The transfer was therefore selected in order to restore protected output before any further recovery work.

The UPS acknowledged the command and displayed a synchronising message for several seconds. It then completed the transfer. The static bypass switch, however, did not release as the inverter connected, which appears to have been the same switching condition logged at 15:24. For a brief interval the inverter output and the raw bypass supply were connected to the same output bus while out of phase with one another.

15:28 — Upstream protection operates

A loud report was heard from the distribution board in the second-floor corridor. The inverter protected itself within milliseconds; the fault current flowing back through the bypass input was instead cleared by the 63 A fuse supplying the UPS. The unit had been installed with a single shared input, so the same fuse fed both its rectifier and its bypass. The circuit schedule on the board door was handwritten and attributed this way to “Kitchen / misc”.

The fuse also supplied the second-floor office sub-board. Lighting and small power across the floor were lost at the same moment. Staff reported a power cut to the building management company, which confirmed that the other floors, the ground-floor café and the street remained supplied normally.

15:28 — Transfer to battery

With both rectifier and bypass inputs now unavailable, the UPS transferred the load to its internal battery string. This was the first transfer of the day to complete in the manner described by the manual. The display estimated eleven minutes of runtime, falling to four minutes within the first thirty seconds as every host simultaneously resumed its recovery routines at full fan speed.

The UPS log showed that a “replace battery” notification had been acknowledged from the front panel on a number of occasions since 2019, most recently the previous Tuesday. A scheduled battery self-test had been deferred on each of the same dates. A sharp chemical odour was noted but was not initially distinguished from the warm-dust smell already present in the room.

15:31 — Battery overtemperature

The UPS reported battery overtemperature, followed by a battery fault. It continued supplying the load, as it had no other available source. To reduce the discharge current, the virtualisation hosts were shut down individually using their front-panel power buttons, the management network remaining unavailable. By the third host, the side panel of the battery cabinet was visibly deformed and an audible hissing could be heard from its vents.

15:34 — Emergency power off

The room emergency power off button was operated. UPS output ceased immediately and the remaining equipment fell silent. The emergency power off disconnects the UPS output only; the battery string remained connected to its own DC bus through a breaker inside the battery cabinet, behind the side panel that was now bulging.

The hissing continued. Grey-white vapour began to leave the upper cabinet vents and collect beneath the ceiling tiles.

15:36 — Battery fire

Flame became visible at the cabinet vents as one battery block entered thermal runaway and the condition extended to the blocks on either side. The room’s gas suppression system had been placed in manual-only mode following an unplanned discharge in 2021. Its manual release station was mounted on the wall beside the UPS and could not be approached safely.

A carbon dioxide extinguisher from the corridor was discharged at the vents from the doorway. The flame subsided and re-established itself within approximately one minute. The communications-room door was closed. Smoke was by then passing through the cable penetrations into the ceiling void above the office floor.

15:39 — Evacuation

The building fire alarm was activated from the corridor call point and all floors were evacuated to the car-park assembly point. A complete roll call could not be taken: the visitor record and staff attendance list were held on the reception PC, which had been unable to authenticate since 13:47. The building management company evacuated the adjoining units and the ground-floor café as a precaution.

Smoke was visible from the communications-room window and the second-floor roof vent. All technical recovery activity ceased and the incident was escalated to the emergency services.

Finding and attribution

No single causal event could be isolated. The outcome required the close temporal convergence of a carrier handoff that changed negotiation behaviour, an activation-dependent replacement firewall, an existing HA configuration asymmetry, a dormant switch firmware mismatch, incomplete uplink resilience, an undocumented storage dependency, an unsolicited UPS bypass transfer, a static switch that did not release on retransfer, an unlabelled upstream fuse shared with the office floor, a battery string operating beyond its service interval and a suppression system left in manual mode.

Each condition became observable only after the preceding condition was tested. The onsite actions were individually consistent with normal fault-isolation practice; the return to inverter operation, in particular, was a documented operator function selected to restore protection to the load. The actions were adjacent to, rather than conclusively responsible for, the subsequent infrastructure, power and fire states. On that basis the incident is best classified as a coincidental cascade of latent environmental conditions with no attributable initiating party.

15:54 — Fire brigade attendance

Four fire appliances and an aerial ladder platform attended, and the incident commander assumed control of the building. The road outside was closed and a cordon established around the building. Crews wearing breathing apparatus isolated the electrical intake and entered the communications room. They asked for the battery chemistry, block count and cabinet layout; this information was held in the UPS documentation on the infrastructure file share inside the unavailable VMware estate.

The battery cabinet was disconnected, removed to the car park and cooled with water. It continued to vent for a further forty minutes. The onsite engineering attendance was closed at that point, with the original backup-connection test outstanding.