Recurring import of ITSI entities not replacing existing entities

[amartin6]

Bulk entity import worked but trying to understand why recurring ITSI imports aren't occurring:
https://docs.splunk.com/Documentation/ITSI/4.3.0/Configure/Recurringimport
Working on single instance
Under step 5 there is a note: The recurring import search executes as splunk-system-user, which returns entities from datasets that exist in indexes that the user creating the import might not have access to.
No errors when looking in _audit index in regards to splunk-system-user, how can I determine if this is the issue since I can't assign the role capabilities?

Using REPLACE as the update type and log level set to DEBUG and see it run but no errors, my test is deleting an alias in one of the entities to see if it will get added. When running the saved search either scheduled or by itself it returns data as needed, it seems like the replace option isn't working. I don't have another example to compare to so I'm not sure what the logs should look like for a successful import, below is the log file.

2019-08-01 14:00:00,550 DEBUG [itsi.csv_import] [itoa_storage] [is_available] [104610] Querying if KV store is available: True
2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itoa_storage] [wait_for_storage_init] [104610] KV store has been initialized.
2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itsi_csv_import] [do_run] [104610] import_from_search: True
2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itsi_csv_import] [do_run] [104610] import_info: {"import_from_search": "1", "service": {"titleField": "Service Title", "serviceEnabled": "1", "criticality": "", "descriptionColumns": ["Service Description"], "serviceSecurityGroup": "default_itsi_security_group", "backfillEnabled":"0"}, "search_string": "| savedsearch \"SPLUNK:firewall_entities_search\"", "service_dependents": [], "selected_services": null, "interval": "0 */2 * * *","index_latest": "now", "updateType": "replace", "index_earliest": "-60m", "log_level": "DEBUG", "entity": {"identifyingFields": ["host"], "informationalFields": ["bunit", "owner", "os", "category"], "titleField": "device_hostname", "mergeField": "undefined", "service_column": [], "fieldMapping": {}, "descriptionColumns": ["description"]}, "selectedServices": [], "service_rel": [], "template": {}}
2019-08-01 14:00:17,272 DEBUG [itsi.csv_import] [itsi_csv_import] [import_via_search] [104610] Done running search. Modular input will now try to import your entities/services.
2019-08-01 14:00:17,294 INFO [itsi.csv_import] [itoa_bulk_import_specification] [_get_fields_to_import] [104610] Fields to Import: SpecFields(entity_fields=['os', 'device_hostname', 'description', 'category', 'bunit', 'owner', 'host'], service_fields=['', 'Service Title', 'Service Description'], entity_relationship_fields=[])
2019-08-01 14:00:17,300 INFO [itsi.csv_import] [itoa_bulk_import] [_bulk_import] [104610] CSV data load initializing mark start=1564668017.3

[R_B]

Hello @amartin6,

I saw that there hasn't been any responses to your question, however I am experiencing the same exact problem as you. I was wondering if you have any more insight into this problem since you posted this?

I don't have any answers yet, but I do have some more insight that may or may not help get to the bottom of this problem...

I too was doing a recurring import with REPLACE and DEBUG. The import works just fine when I do it as a normal import, but I get the same exact logs as you do when I set it up as a recurring import. I tried doing UPSERT and APPEND as the "update_type", but I get the same results.

In my case, and perhaps yours, I don't think the problem was with the splunk-system-user not having permission to read the index, as logs from $SPLUNK_HOME/var/log/splunk/itsi_csv_import-IMPORT-STANZA-NAME.log shows "Processing batch of size ###", "###" being exactly the size I get when doing the import normally through the GUI. Also, all of the logs with that source also shows that the fields from the recurring search are getting read. So it seems like the search is returning the expected results and Splunk is processing the search. I even got the logs like you that state "Modular input will now try to import your entities/services.".

The only kind of error that I found in my logs perhaps related to this was at the same time as these logs, I searched for index=_internal host=host-that-recurring-search-ran-on source=$SPLUNK_HOME/var/log/splunk/splunkd.log "itsi" "import" log_level=ERROR ... I see errors with "$SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py InsecureRequestWarning" and I see errors with "python $SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py $SPLUNK_HOME/etc/apps/SA-ITOA/lib/SA_ITOA_app_common_solnlib/packages/request/urllib3/connectionpool.py:port# InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings"

I'm not sure if this is the cause of the problem, but it does show that itsi_csv_import.py threw errors at exactly the same time the recurring import ran. So, there cold be some correlation there.

Anyway, I hope this gives a bit more insight into the problem. I'm going to keep troubleshooting this, however if you or anyone else has anymore insight please let me know. Thanks!