Hello @amartin6,
I saw that there hasn't been any responses to your question, however I am experiencing the same exact problem as you. I was wondering if you have any more insight into this problem since you posted this?
I don't have any answers yet, but I do have some more insight that may or may not help get to the bottom of this problem...
I too was doing a recurring import with REPLACE and DEBUG. The import works just fine when I do it as a normal import, but I get the same exact logs as you do when I set it up as a recurring import. I tried doing UPSERT and APPEND as the "update_type", but I get the same results.
In my case, and perhaps yours, I don't think the problem was with the splunk-system-user not having permission to read the index, as logs from $SPLUNK_HOME/var/log/splunk/itsi_csv_import-IMPORT-STANZA-NAME.log shows "Processing batch of size ###", "###" being exactly the size I get when doing the import normally through the GUI. Also, all of the logs with that source also shows that the fields from the recurring search are getting read. So it seems like the search is returning the expected results and Splunk is processing the search. I even got the logs like you that state "Modular input will now try to import your entities/services.".
The only kind of error that I found in my logs perhaps related to this was at the same time as these logs, I searched for index=_internal host=host-that-recurring-search-ran-on source=$SPLUNK_HOME/var/log/splunk/splunkd.log "itsi" "import" log_level=ERROR ... I see errors with "$SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py InsecureRequestWarning" and I see errors with "python $SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py $SPLUNK_HOME/etc/apps/SA-ITOA/lib/SA_ITOA_app_common_solnlib/packages/request/urllib3/connectionpool.py:port# InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings"
I'm not sure if this is the cause of the problem, but it does show that itsi_csv_import.py threw errors at exactly the same time the recurring import ran. So, there cold be some correlation there.
Anyway, I hope this gives a bit more insight into the problem. I'm going to keep troubleshooting this, however if you or anyone else has anymore insight please let me know. Thanks!
... View more