There are several things that could be causing this problem.
If pushing the bundle with the updated identities.csv file from the search head cluster deployer , the bundle could be too large therefore hitting the http server max content length of 800MB on the SHC members.
You would see evidence of this on the SHC member you push to in $SPLUNK_HOME/var/log/splunk/splunkd_access.log with logs with status code 413
If that is the case you could increase the max_content_length on the SHC members to work around that:
* Measured in bytes
* HTTP requests over this size will rejected.
* Exists to avoid allocating an unreasonable amount of memory from web
* Defaulted to 838860800 or 800MB
* In environments where indexers have enormous amounts of RAM, this
number can be reasonably increased to handle large quantities of
If pushing the bundle from the deployer and using the preserve-lookups flag, that will not update lookups on the members but preserve their local lookup files instead of take what the deployer is pushing. This feature is somewhat limited becuase it doesn't allow you to get granular and specify which lookups to preserve.
./splunk apply shcluster-bundle -target https://linux01.sv.splunk.com:8089 -preserve-lookups true -auth admin:pwd
lookup files that don't have their inputs or transforms.conf configured properly so the identity_manager.py script is not picking them up
rob1_identities.csv is my file I want merged into identities_expanded.csv
confirm each member has the updated file.
now the next time the identity_manager.py runs (every 5 min) it will merge the change into identities_expanded.csv
Any manual file update on one of the SHC members will not work unless the member is the captain. And in that case only the identies_expanded.csv would get replicated to other members not the rob1_identities.csv so this approach is not advised.
Confirm your file has the correct header format for identities.csv
The most common reason for failure is incorrect formatting or invalid data in the assets.csv or identities.csv lookup files used as the source.
The header must be included in the file and be in this format for identities.csv:
There are several ways to update the identities.csv file on the SHC members and trigger the merge of the file(s) into identities_expanded.csv
a.) option 1: run a search with outputlookup to update the file:
If you run the modular input command manually (as suggested in "a" above) for these merges to try to debug what's going on with them, the results will not wind up in the correct SA-IdentityManagement context. In my tests they wound up in Searching & Reporting OR EnterpriseSecurity contexts. We ended up manually moving the results to SA-IdentityManager so as not to have to run the search again.
Modular input throws an error if there are spaces in any extra field names you pull in.
Errors like #2 may be masked by other problems, like inputs.conf stanza in one context and transforms.conf stanza in another (which may happen if you use the WebUI).