Firepower Management Center HA
Configuring HA for FMC is pretty straight forward but how exactly does it work and how can we troubleshoot HA if it is not working correctly? In this post I will show you what FMC HA is doing behind the scenes and tools we have available to take a deeper look into the system and uncover issues.
FMC High Availability
High Availability is available on physical Firepower Management Center appliances (and FMCv since 6.7.0). It lets you create an active/standby HA solution which does not require layer 2 adjacency (making it possible to have real HA over multiple sites without extending layer 2 across sites).
FMC HA will create a second “manager” registration on your sensor resulting in two sftunnel connections. One to your primary FMC and one to your secondary FMC.
It is a cold-standby solution that does not failover without a manual interaction. All events are being logged to both FMCs so in case of device failure you should not lose any events sent from the sensor to FMC. Just promote the passive FMC and it will start up all the necessary processes and become the active unit, allowing you to perform configuration changes from the secondary unit.
Behind the scenes the HA procedure consists of a sybase database mirror and a transaction framework that will make sure data is being replicated from the active to the passive management center.
Before configuring FMC HA make sure that…
- Hardware is identical (no mix and match between virtual and/or physical form factors)
- Software release is identical on both FMCs
- There are no sensors registered to the secondary FMC
- You have a working backup (I havent seen a case where HA initialization caused any issues that would require re-imaging… but you never know :)
To configure High Availability login to FMC, navigate to
Integration > High Availability and define a secondary peer. Then grab some coffee and be patient. It will take some time and you will see some warnings that might be misleading, but before jumping into a CLI to start troubleshooting wait about 20 minutes for the procedure to finish. Depending on the number of sensors managed the process might only take a few minutes, but from experience it’s more in the 15-30 minutes range.
After some minutes the status of HA Synchronization should change to “OK” like this:
At this point your sensors should be registered with your secondary FMC and should be listed on the device management page
To upgrade an FMC in HA you will have to follow the following instructions:
- Manually stop HA synchronization
- Upgrade the passive FMC
- Wait for the upgrade to finish (HA state may change to degraded, which is normal)
- Upgrade the active FMC (Upgrade cannot be started while standby FMC is not finished)
- Wait for active FMC to reboot and all processes to start up
- Promote the primary FMC to become active
- Deploy configuration to your sensors to verify everything is working as expected
Make sure to always check the current release notes for additional information!
Device not registered to secondary FMC after HA configuration
In case the device registration failed you will have to remove the sensor from your active FMC and login into your sensor. You will need to use the
configure manager delete command followed by the
configure manager add command to add your sensor to FMC again. At this point I would advice you to open up
pigtail on both your sensor and FMC and re-add the sensor on the active FMC.
Using pigtail you will log all necessary output to find any issues in case the registration fails again.
Synchronization stopped during FMC backup
This behavior is by design and is not an issue. If a backup of FMC is being performed the HA synchronization will be stopped. During this timeframe you can continue configuration on your active FMC.
Events related to FMC HA are being logged to
/var/log/syncd.log. In case you have any issues that cant be solved via the UI make sure to check this logfile for further details.
FMC ships with two perl scripts that can be used to query for high availability information. I would recommend not using these tools to change any configuration parameters but only to verify the current state of FMC HA. In the past I have used
manage_HADC.pl to switch roles, break HA etc. and did not encounter any problems.
manage_HADC.pl provides a command line interface to query the current HA state and execute management tasks that are also available on the UI.
Option 1 is probably the only option you want to select. It will display the current state of HA with some additional details
As the name of the script indicates you can use troubleshoot_HADC.pl to verify some additional information not available using manage_HADC.pl
Option 1 will display the current status of the sybase database replication
Option 2 will connect to the local sybase database to verify that connectivity is working
Option 4 will display the status of the peer (other FMC). Use this option to verify software version, ip address, etc. are correctly set