Splunk ES asset and identity merge issues

[phoenixdigital]

A quick question about how the asset and identity list is populated for Splunk ES.

I can see it is happening from a Identity Management modular input under (with associated python scripts)

Settings -> Data Inputs -> Identity Management

However sometimes the list used by Splunk fails to populate. Generally if there is an issue with the asset or identity csv.

The problem I have is this failure to parse the asset or identity list csv fails silently. I get no error to indicate that this didn't work.

This is probably the biggest issue I am concerned about as if the auto generation routines for these list puts something in that breaks the ingestion we might not know for weeks that the list is old due to a failure of parsing the newer csv.

Lastly how often does Splunk ES update this list when csv entries are changed?
Sometimes it appears to be immediate other times it appears to take a few minutes.

[andrew207]

The answer by mdessus describes how to detect this issue.

Firstly, the ES Identity and Assets are merged every 5 minutes as a modular input, that explains why sometimes it will happen instantly and other times it can take a few minutes: http://docs.splunk.com/Documentation/ES/4.1.1/User/Identitymanagement#Merging_the_asset_and_identity...

What worked for me was the following:

Background: you have a lookup, ad_identity_list that is silently failing to load in to ES. The lookup is populated with good data, you've checked the logs for modular inputs and have seen that the merge is running properly, but no data Identity data is being populated in ES.

1. Make an interim lookup, called something like ad_identity_interim.
2. Copy whole ad_identity_list into ad_identity_interim.
3. Execute the following, to place only a few entries into the Identity lookup ES is trying to merge. | inputlookup ad_identity_interim | head 5 | outputlookup ad_identity_list
4. Wait until the merge occurs and you should see the five entries in your Identity Center.

5. Continue adding incrementally until you have the whole list in there, making sure you wait for the merge to occur between each execution.
   | inputlookup ad_identity_interim | head 50 | outputlookup ad_identity_list
   | inputlookup ad_identity_interim | head 100 | outputlookup ad_identity_list
   | inputlookup ad_identity_interim | head 500 | outputlookup ad_identity_list
   | inputlookup ad_identity_interim | head 1000 | outputlookup ad_identity_list

6. You should now have all your identities in ES.

I'm unsure as to why this works, but the issue has occurred and this fix has worked for me in several completely different architectures. It seems as though once the initial list has populated that the updates to the lookup are loaded properly, so I haven't had to make a chain of saved searches to behave as described above; it works as expected once it's all initially loaded -- noting that I have only ever made minor changes on an ongoing basis.

[esix_splunk]

Regarding the merging, by default this is every 5 minutes. This can be changed in the SA-IdentityManagement TA.

[mdessus_splunk · Accepted solution]

Hi,

Splunk checks if there is a modification from time to time (however, I don't remember the interval).
You can always force it with the following command:
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SA-IdentityManagement/bin/identity_manager.py --username admin
(see http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Force_a_merge )
And to check if it worked or not, look in the internal indexes:
index=_internal source=*python_modular_input.log "Updated: target lookup table" OR "No merging required"
(it's documented here: http://docs.splunk.com/Documentation/ES/3.3.0/User/Assetmanagement#Verify_expansion_process )

[splunkDude2015]

doesn't this throw an ERROR? the first part generates an xml and passing it to python errors out. Is this some sort of bug...

$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python

[phoenixdigital]

Thanks heaps for the response. I'm glad there is a way to detect failed imports.