Firepower Management Center HA

Configuring HA for FMC is pretty straight forward but how exactly does it work and how can we troubleshoot HA if it is not working correctly? In this post I will show you what FMC HA is doing behind the scenes and tools we have available to take a deeper look into the system and uncover issues.

FMC High Availability

High Availability is available on physical Firepower Management Center appliances (and FMCv since 6.7.0). It lets you create an active/standby HA solution which does not require layer 2 adjacency (making it possible to have real HA over multiple sites without extending layer 2 across sites).

FMC HA will create a second “manager” registration on your sensor resulting in two sftunnel connections. One to your primary FMC and one to your secondary FMC.

It is a cold-standby solution that does not failover without a manual interaction. All events are being logged to both FMCs so in case of device failure you should not lose any events sent from the sensor to FMC. Just promote the passive FMC and it will start up all the necessary processes and become the active unit, allowing you to perform configuration changes from the secondary unit.

Behind the scenes the HA procedure consists of a sybase database mirror and a transaction framework that will make sure data is being replicated from the active to the passive management center.

Configuration

Before configuring FMC HA make sure that…

• Hardware is identical (no mix and match between virtual and/or physical form factors)
• Software release is identical on both FMCs
• There are no sensors registered to the secondary FMC
• You have a working backup (I havent seen a case where HA initialization caused any issues that would require re-imaging… but you never know :)

To configure High Availability login to FMC, navigate to Integration > High Availability and define a secondary peer. Then grab some coffee and be patient. It will take some time and you will see some warnings that might be misleading, but before jumping into a CLI to start troubleshooting wait about 20 minutes for the procedure to finish. Depending on the number of sensors managed the process might only take a few minutes, but from experience it’s more in the 15-30 minutes range.

After some minutes the status of HA Synchronization should change to “OK” like this:

At this point your sensors should be registered with your secondary FMC and should be listed on the device management page

Upgrade Procedure

To upgrade an FMC in HA you will have to follow the following instructions:

• Manually stop HA synchronization
• Upgrade the passive FMC
• Wait for the upgrade to finish (HA state may change to degraded, which is normal)
• Upgrade the active FMC (Upgrade cannot be started while standby FMC is not finished)
• Wait for active FMC to reboot and all processes to start up
• Promote the primary FMC to become active
• Deploy configuration to your sensors to verify everything is working as expected

Make sure to always check the current release notes for additional information!

Troubleshooting

Device not registered to secondary FMC after HA configuration

In case the device registration failed you will have to remove the sensor from your active FMC and login into your sensor. You will need to use the configure manager delete command followed by the configure manager add command to add your sensor to FMC again. At this point I would advice you to open up pigtail on both your sensor and FMC and re-add the sensor on the active FMC.

Using pigtail you will log all necessary output to find any issues in case the registration fails again.

Synchronization stopped during FMC backup

This behavior is by design and is not an issue. If a backup of FMC is being performed the HA synchronization will be stopped. During this timeframe you can continue configuration on your active FMC.

Log Files

Events related to FMC HA are being logged to /var/log/syncd.log. In case you have any issues that cant be solved via the UI make sure to check this logfile for further details.

CLI Tools

FMC ships with two perl scripts that can be used to query for high availability information. I would recommend not using these tools to change any configuration parameters but only to verify the current state of FMC HA. In the past I have usedmanage_HADC.pl to switch roles, break HA etc. and did not encounter any problems.

manage_HADC.pl

manage_HADC.pl provides a command line interface to query the current HA state and execute management tasks that are also available on the UI.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

****************  Configuration Utility  **************
 1   Show HA Status
 2   Register as Secondary HA FMC
 3   Register as Primary HA FMC
 4   Switch HA roles
 5   Pause Mirror
 6   Re-establish Mirror
 7   Set as Active for Full UI
 8   Force as Active to resolve split brain
 9   Break HA - keep devices
 10  Break HA - delete devices
 0   Exit
**************************************************************

Option 1 is probably the only option you want to select. It will display the current state of HA with some additional details

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

(...)
STATUS:
Active HA DC: 1 
HA_SYNC: SYNCHRONIZATION_HADC 
LastFailover: Success, Mon Jun 12 19:54:23 2017 
Sybase_State: 1 
blocking: 0 
connected: 1 
degraded: 0 
degraded_message: ARRAY(0xabd5e78) 
mirroring: 1 
peer_connected: 1 
status: ARRAY(0xacdb880) 

troubleshoot_HADC.pl

As the name of the script indicates you can use troubleshoot_HADC.pl to verify some additional information not available using manage_HADC.pl

1
2
3
4
5
6
7
8

****************  Troubleshooting Utility  **************
 1   Show HA Info Of FMC
 2   Execute Sybase DBPing
 3   Show Arbiter Status
 4   Check Peer Connectivity
 5   Print Messages of AQ Task
 0   Exit
**************************************************************

Option 1 will display the current status of the sybase database replication

1
2
3
4
5
6
7
8
9

Enter choice: 1

HA Enabled: Yes

This FMC Role In HA: Active - Primary
Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)
Sybase Database Connectivity: Accepting DB Connections.
Sybase Database Name: csm_primary
Sybase Role: Active

Option 2 will connect to the local sybase database to verify that connectivity is working

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Enter choice: 2
Sybase Database Name: csm_primary
$VAR1 = [
          'Mirror Server => csm_primary',
          {
            'stderr' => undef,
            'stdout' => 'SQL Anywhere Server Ping Utility Version 16.0.0.2271
Type       Property                  Value
---------  ----------------          ------------------------------
Database   MirrorRole                primary
Database   MirrorState               synchronizing
Database   PartnerState              connected
Database   ArbiterState              connected
Server     ServerName                csm_primary
Ping database successful.
',
            'rcode' => 0
          }
        ];

Option 4 will display the status of the peer (other FMC). Use this option to verify software version, ip address, etc. are correctly set

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Enter choice: 4
Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected

$VAR1 = {
          'vip_local' => '',
          'priority' => '0',
          'ip' => '10.1.1.2',
          'model_id' => 'F',
          'uuid' => 'f1fa29d2-7b4f-11e6-b5a3-ca59e03bf0e4',
          'sw_version' => '6.2.0.2',
          'upgrade_version' => '',
          'persistent' => '0',
          'mgmt_mac_address' => '00:62:EC:42:EE:F2',
          'vnet' => undef,
          'primary_mgr' => '0',
          'vip' => '',
          'model_number' => '66',
          'ipv6' => undef,
          'reg_state' => '0',
          'name' => 'fmc-secondary',
          'active' => '1',
          'uuid_gw' => '',
          'reg_key' => '',
          'last_changed' => '1491849682',
          'role' => '0'
        };