I've been doing upgrades for quite a few years and after running into numerous issues upgrading into the 5.0.X version, I thought I'd reach out to the community and see if anyone else has come across a good upgrade plan to migrate services to new hardware and upgrade the software. 

In the past I would upgrade the Admin Console by running a backup and removing the backup file from the server. I would shut down the Admin Console then fire up a new VM (with newer OS) with the same host name and IP address, install the originating Access Manager version, perform a restore and do any necessary upgrades to get to current or which ever version we needed to get to. 

This had worked great until 5.0.X and RHEL 8 came around. I could not install older version of AM onto RHEL 8 and upgrading on the original Admin Console to 5.0.X would 'forget' to back pieces up and subsequently not restore leaving corrupt CAs, missing App Marks and visual errors when going to the Access Gateways cluster pages.

For testing purposes, I would also add a new server into the IDP and AG cluster and use host file redirects to do the testing before either upgrading the original or deprecating the old one from the cluster all together. This has been a crap shoot now and is explicitly called out in the documentation as something that should never be done, at last without making any changes. With that said, how does one go about testing 150+ federations when 80% do not have a non-production equivalent? I am to believe we are just going to upgrade all the IDPs and 'hope' that everything works on the other side? There needs to be some migration path to take and I'm just not seeing one from the official documentation other than the big bang/pull the band-aid off - that is unless I'm missing something. 

Anyone have any input on a better direction for a hardware migration/version upgrade?