Troubleshooting Firepower User Identity

Troubleshooting user identity issues on firepower systems can be quite daunting at times if you do not know where exactly you should look for issues. After encountering many different problems I thought it might be a good idea write down what I have learned from various troubleshooting sessions since unfortunately the amount of available documentation is quite lacking.

Architecture

Before diving into the various tools we can use to troubleshoot identity issues lets focus on how exactly passive user identity is working and how we are able to accurately map users to ip addresses.

In most scenarios we are using a LDAP directory to fetch user to ip mappings. Because users are using a central point to authenticate to the corporate network we can query our directory service (e.g. Active Directory) to find out which ip address generated a logon event. To fetch this information we can use two different approaches.

The first approach is using a dedicated agent that must be installed on a domain-joined windows server. The Firepower User Agent (FUA) will be used to query the active directory security log for logon events and send this information to FMC using a direct connection to the FMC Mysql database.

The second approach is using Cisco ISE integration. ISE joins a active directory domain to query the security log for logon events using WMI. This information is then propagated to FMC using pxGrid.

Apart from fetching ip mappings using either FUA or ISE, Firepower Management Center will also directly query the ldap directory to download user to group mappings. A so called “Realm” is configured on FMC to periodically download this information to FMC to keep track of any changes that might occur (e.g. new user added to group HR, users being deleted, etc.)

The last step in the chain is propagating the received user to group mappings and user to ip mappings to the sensors. To ensure the identity information is sent to the correct sensors, we must associate a Identity Policy to our Access Control Policy in FMC. The Identity Policy will define which Realm to use and for which network ranges we want to send identity mappings to our sensors (e.g. only send user:ip mappings to sensor if ip address is within a certain private ipv4 range).

After the Access Control Policy with the associated Identity Policy has been deployed to our sensor, FMC will forward identity information to the sensor using a tls encrypted tunnel (sftunnel) that is used for communication between FMC and the sensor. SFTunnel is used for communication between the two devices and is utilized for various purposes (monitoring, configuraton deployment, updates, etc.). A snapshot of the currently available mappings is pushed by FMC to the sensor over sftunnel, which is then processed by the SFDataCorrelator process.

Summary

1. User identity integration is implemented centrally using FMC
2. FMC connects directly to the directory service to download user to group mappings (realm configuration)
3. FMC connects to either Firepower User Agent or ISE pxGRID to receive user to ip mappings
4. An Identity Policy must be assigned to the Access Control Policy to send user:ip mappings to sensors
5. FMC pushes identity information to sensors using a tls encrypted tunnel named sftunnel

Basic Troubleshooting

This section contains various troubleshooting steps that can be performed to solve common user identity related issues.

Verify Connectivity between FMC and ISE

Under the Identity Sources menu in the FMC UI, the connection status between FMC and ISE can be tested. Since pxGRID will only be active on the Primary Administration Unit (PAN) on ISE, you will see the connection to one ISE failing. This is perfectly fine.

Navigate to System > Integration > Identity Sources > Identity Service Engine to verify the connection status

You may use the Additional Logs section to check for error information. Normally issues can be traced back to connectivity issues between FMC and ISE (e.g. missing firewall rules) or certificate mismatches.

Another method of checking the pxGRID connection is to directly connect to ISE and check the pxGRID Services status. If FMC is listed as Subscriber for identity information you are good to go.

To check the status of pxGRID on ISE navigate to Administration > pxGrid Services > Clients on the PAN.

Additional Troubleshooting information

If you are looking for additional troubleshooting logs for pxGRID you may want to check out /var/log/messages on FMC. Use the following command to check for ISE related log entries:

1

cat /var/log/messages | grep adi.ISEConnection

Verify Connectivity between FMC and Active Directory

You may use the test function in the realm configuration to check the connection to a directory server. Navigate to System > Integratrion > Directory and edit the configured realm. A test button will appear which can be used to connect to the server.

Additional Troubleshooting information

If you are looking for additional troubleshooting logs for directory connections, you may want to check out /var/log/messages on FMC. Use the following command to check for directory related events that include additional information about connectivity issues between FMC and Active Directory.

1

cat /var/log/messages | grep adi.Directory

Verify Access Control Policy Configuration

To make sure that identity mappings are pushed from FMC to your sensor make sure that your identity policy is referenced in your Access Control Policy.

Advanced Troubleshooting

This section contains troubleshooting information for advanced issues. Use the following procedures at your own risk.

Debugging ADI

Firepower Management Center and the sensor use the ADI process for identity related tasks. For example on FMC, ADI is used to connect to ldap directories, establish a connection to your FUA or ISE. To gather additional debug information you may start the process using a debug flag. Keep in mind that you have to kill the process before that, which will stop identity updates to your FMC. If you depend on identity integration dont leave it disabled for too long.

1. Login to FMC using SSH and change to root

   1

   sudo su -

2. Disable adi process using process manager (pmtool)

   1	pmtool DisableById adi

3. Verify adi is not running

   1	ps ax | grep adi

4. Start adi process with debug flag, pipe output to /var/tmp/adi-debug.log and move process to background

   1	nohup /usr/local/sf/bin/adi --debug > /var/tmp/adi-debug.log 2>&1 &

5. Verify adi is up and running

   1	ps ax | grep adi

6. After gathering debug information kill adi again (use ps to find out pid)

   1	kill -9 <adi-pid>

7. Enable adi process using process manager (pmtool)

   1	pmtool EnableById adi

Dump Identity Mappings

Firepower Management Center saves identity mappings to its mysql database. This information is then sent to the sensor using sftunnel. A file containing the current mappings is transferred to the sensor and processed by SFDataCorrelator which will write the current user:ip mappings to the local mysql database instance.

If everything is working corretly you should find the user to ip mapping in three different locations

1. Mysql database on FMC
2. Identity snapshot on sensor
3. Mysql database on sensor

In case a user to ip mapping is not present anywhere down the road you probably hit a bug and should contact TAC.

FMC Mysql database

1. Connect to FMC using SSH and login to mysql database. Use the default password ‘admin’.

   1	mysql -uroot -p sfsnort

2. Search database for the user that experiences issues (note user id after executing query)

   1	mysql> select * from user_identities where username like '%<your-username>%' \G

3. Search database for user to ip mapping for the specified user (use user id for query)

   1	mysql> select inet6_ntoa(ipaddr) FROM current_user_ip_map where id=<user-id> \G

Sensor Identity Snapshot

The user to ip snapshot is not human readable and must be dumped using the u2dump utility. Before dumping the identity mappings verify that a snapshot exists.

1. Connect to sensor using SSH, change to expert mode and change to root

   1 2	> expert sudo su -

2. Verify existance of user snapshot

   1	ls -lah /var/sf/user_enforcement

3. Dump user snapshot to /var/tmp

   1	u2dump /var/sf/user_enforcement/user_ip_map.* > /var/tmp/user-ip-map.dump

4. Open user snapshot using vi(you might also use another text editor or use more but I prefer vi to search)

   1	vi /var/tmp/user-ip-map.dump

EDIT

2019-11-25: Removed incorrect information about communication between Firepower User Agent and FMC. FUA does not utilize FMCs REST interface but a direct database connection (thanks to Alexander V. for bringing this to my attention)