A thousand devices and no master key

The problem

Around a thousand network devices — switches, access points, routers and wireless controllers — administered through local accounts, one per device, under an agreed naming convention. On paper there was a criterion. In practice there were a thousand doors, each with its own key, and no central way of knowing who went through which.

That way of administering ended the way it usually ends: someone got in and wiped configurations. The operation went down, and bringing it back took far longer than it would have if the real state of the network had been written down anywhere.

Because that was the other problem. The documentation lived in files updated whenever somebody remembered, so the drawn topologies and the actual network had not matched for a long time. Every support request started by working out what was there, which turned a diagnosis of minutes into one of hours.

The decision

Centralising administration and access control with Cisco is what anyone who knows the problem would recommend. The hard call was a different one.

What was decided was to stop working as an outside supplier. To come into the organisation as one more member of the team, to take ownership of the whole solution instead of delivering an implementation and invoicing it, and to stay inside until the new way of operating was simply the way of operating.

And a masterclass for the client’s team, so they would understand what was being done and why. Not a vendor product training: an explanation of the reasoning behind each decision, so that afterwards they could hold it up and argue with it without calling anyone.

That last part runs directly against any supplier’s short-term commercial interest. A client who understands their own infrastructure calls less.

The result

Administration of the thousand devices ended up centralised, and with it control over credentials: the dependence on device-by-device local access was over — the very door the incident had come through.

The network gained visibility of its own. The topology stopped depending on somebody keeping a file up to date, and support stopped beginning each request by working out what was connected. That shortened diagnoses and, above all, made them repeatable: they stopped depending on who happened to be on duty.

On the internal team’s side, the masterclass did what it was meant to do. The decisions behind the new architecture stopped being a box someone from outside had assembled and became something they could explain themselves.