Splunk Enterprise Security

Splunk ES asset and identity merge issues

phoenixdigital
Builder

A quick question about how the asset and identity list is populated for Splunk ES.

I can see it is happening from a Identity Management modular input under (with associated python scripts)

Settings -> Data Inputs -> Identity Management

However sometimes the list used by Splunk fails to populate. Generally if there is an issue with the asset or identity csv.

The problem I have is this failure to parse the asset or identity list csv fails silently. I get no error to indicate that this didn't work.

This is probably the biggest issue I am concerned about as if the auto generation routines for these list puts something in that breaks the ingestion we might not know for weeks that the list is old due to a failure of parsing the newer csv.

Lastly how often does Splunk ES update this list when csv entries are changed?
Sometimes it appears to be immediate other times it appears to take a few minutes.

1 Solution

mdessus_splunk
Splunk Employee
Splunk Employee

Hi,

Splunk checks if there is a modification from time to time (however, I don't remember the interval).
You can always force it with the following command:
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SA-IdentityManagement/bin/identity_manager.py --username admin
(see http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Force_a_merge )
And to check if it worked or not, look in the internal indexes:
index=_internal source=*python_modular_input.log "Updated: target lookup table" OR "No merging required"
(it's documented here: http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Verify_expansion_process )

View solution in original post

andrew207
Path Finder

The answer by mdessus describes how to detect this issue.

Firstly, the ES Identity and Assets are merged every 5 minutes as a modular input, that explains why sometimes it will happen instantly and other times it can take a few minutes: http://docs.splunk.com/Documentation/ES/4.1.1/User/Identitymanagement#Merging_the_asset_and_identity...

What worked for me was the following:

Background: you have a lookup, ad_identity_list that is silently failing to load in to ES. The lookup is populated with good data, you've checked the logs for modular inputs and have seen that the merge is running properly, but no data Identity data is being populated in ES.

  1. Make an interim lookup, called something like ad_identity_interim.
  2. Copy whole ad_identity_list into ad_identity_interim.
  3. Execute the following, to place only a few entries into the Identity lookup ES is trying to merge. | inputlookup ad_identity_interim | head 5 | outputlookup ad_identity_list
  4. Wait until the merge occurs and you should see the five entries in your Identity Center.
  5. Continue adding incrementally until you have the whole list in there, making sure you wait for the merge to occur between each execution.
    | inputlookup ad_identity_interim | head 50 | outputlookup ad_identity_list
    | inputlookup ad_identity_interim | head 100 | outputlookup ad_identity_list
    | inputlookup ad_identity_interim | head 500 | outputlookup ad_identity_list
    | inputlookup ad_identity_interim | head 1000 | outputlookup ad_identity_list

  6. You should now have all your identities in ES.

I'm unsure as to why this works, but the issue has occurred and this fix has worked for me in several completely different architectures. It seems as though once the initial list has populated that the updates to the lookup are loaded properly, so I haven't had to make a chain of saved searches to behave as described above; it works as expected once it's all initially loaded -- noting that I have only ever made minor changes on an ongoing basis.

esix_splunk
Splunk Employee
Splunk Employee

Regarding the merging, by default this is every 5 minutes. This can be changed in the SA-IdentityManagement TA.

mdessus_splunk
Splunk Employee
Splunk Employee

Hi,

Splunk checks if there is a modification from time to time (however, I don't remember the interval).
You can always force it with the following command:
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SA-IdentityManagement/bin/identity_manager.py --username admin
(see http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Force_a_merge )
And to check if it worked or not, look in the internal indexes:
index=_internal source=*python_modular_input.log "Updated: target lookup table" OR "No merging required"
(it's documented here: http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Verify_expansion_process )

splunkDude2015
Explorer

doesn't this throw an ERROR? the first part generates an xml and passing it to python errors out. Is this some sort of bug...

$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python

0 Karma

phoenixdigital
Builder

Thanks heaps for the response. I'm glad there is a way to detect failed imports.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...