I solved a Ubiquiti networking equipment problem yesterday.
The Ubiquiti system is very powerful and versatile, yet marketed as "just plug it together", so there are hidden and unclear pitfalls.
The initial problem statement was something like "device firmware updates are reported as available, but update process fails, no other known issues".
Eventually I located and fixed two separate problems:
Each hardware device in a Ubiquiti network participates fully in the network, having IP address, routing, etc. This... also includes DNS. Due to some re-addressing work all the Ubiquiti devices had the former IP addresses for DNS servers. This had no apparent impact on day to day performance, until firmware upgrades were attempted. It didn't seem to matter if the firmware was cached on the controller or not. I'd theorize the firmware is going out to the internet to verify firmware keys and refusing to boot if unable to verify authenticity. The symptom was attempts at firmware updates would take seemingly forever (at least more than five minutes per attempt per device), then fail and roll back to the old firmware. Anyway, regardless of hypothesis about "how" it works, every individual device on the Ubiquiti network needed to be manually updated to the correct, new, working DNS server address.
Generally speaking, in the Ubiquiti web interface, you select a device, "Settings", "Network", and edit "Preferred DNS" and "Alternate DNS".
There's always some give and take in a centralized configuration system, or in this case, push and pull. Take for example, Puppet, which has a remote agent periodically pull configs from a central server, vs Ansible, where a central server pushes configs out to remote devices. Sure, in a simplistic way, Ubiquiti pushes: You configure stuff in the web interface such as perhaps DNS server info, then "Apply Changes" and in a disturbingly long several minute process the device goes offline, then "Getting Ready" status, then comes back online. However, how does it work under the surface? There is a system wide setting, mysteriously named "Override Inform Host", that is pushed to every Ubiquiti device, asking it to pull from a new controller IP address. I've used this many times in the past when installing new controllers. I've upgraded from old to new cloud key to a much faster Docker installation of the controller in the past. Anyway, for whatever reason this site setting had the old IP address for the inform host. Not entirely sure how/why the system was working, but it was working perfectly... except for no upgrades are possible.
In the Ubiquiti web interface, select "Settings" then "System" then scroll to the very bottom "Override Inform Host" and if thats enabled, it should be the IP address of the current (or new?) controller.
One of the many advantages of a Docker install of the controller is access to the "unifi/log/remote" directory. There should be a file for each device's IP address and the logs are interesting to watch during a firmware upgrade attempt.
So, the lesson learned, is a Ubiquiti network can operate perfectly normally with an incorrect IP address for DNS servers in each device AND an incorrect IP address as the system-wide "Override Inform Host", but either mistake will prevent successful firmware upgrades. Who could have guessed?
Anyway, in the end, it all worked!