Hello @amartin6, I saw that there hasn't been any responses to your question, however I am experiencing the same exact problem as you. I was wondering if you have any more insight into this problem since you posted this? I don't have any answers yet, but I do have some more insight that may or may not help get to the bottom of this problem... I too was doing a recurring import with REPLACE and DEBUG. The import works just fine when I do it as a normal import, but I get the same exact logs as you do when I set it up as a recurring import. I tried doing UPSERT and APPEND as the "update_type", but I get the same results. In my case, and perhaps yours, I don't think the problem was with the splunk-system-user not having permission to read the index, as logs from $SPLUNK_HOME/var/log/splunk/itsi_csv_import-IMPORT-STANZA-NAME.log shows "Processing batch of size ###", "###" being exactly the size I get when doing the import normally through the GUI. Also, all of the logs with that source also shows that the fields from the recurring search are getting read. So it seems like the search is returning the expected results and Splunk is processing the search. I even got the logs like you that state "Modular input will now try to import your entities/services.". The only kind of error that I found in my logs perhaps related to this was at the same time as these logs, I searched for index=_internal host=host-that-recurring-search-ran-on source=$SPLUNK_HOME/var/log/splunk/splunkd.log "itsi" "import" log_level=ERROR ... I see errors with "$SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py InsecureRequestWarning" and I see errors with "python $SPLUNK_HOME/etc/apps/SA-ITOA/bin/itsi_csv_import.py $SPLUNK_HOME/etc/apps/SA-ITOA/lib/SA_ITOA_app_common_solnlib/packages/request/urllib3/connectionpool.py:port# InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings" I'm not sure if this is the cause of the problem, but it does show that itsi_csv_import.py threw errors at exactly the same time the recurring import ran. So, there cold be some correlation there. Anyway, I hope this gives a bit more insight into the problem. I'm going to keep troubleshooting this, however if you or anyone else has anymore insight please let me know. Thanks! ... View more

I didn't think about doing it that way, that's perfect, thank you! Ok I see. I was most curious about how Splunk was getting the information, but as you explain its in the .data files and optionally the .rawSize files. Thanks for your help! ... View more

Thank you all for replying with suggestions, my team really appreciates the input! We discussed all of the options. We are very much leaning towards @woodcock's suggestion to use a different port for each sourcetype, and for the devices that can only send on port 514, using a naming convention so that we can specificially apply the sourcetype to those devices. We felt that this would be the most simple, straightforward, and viable option in our environment. Once we finalize a decision and configure it in our environment, I'll report back here with how it goes, so that it can help anyone else that reads this or runs into a similar situation. Thanks! ... View more

To everyone that replied to this: sorry for this very delayed response and thank you all for helping out! I got this working a few weeks ago. If I am remembering correctly, I do believe it was a problem with the account permissions. We reviewed the account that we were using for Security Center, and using the Splunk doc we made sure everything was set up as needed. Then after that it started working without any problems. Thanks again! ... View more

I see, surprised nobody has tried it yet. Unless they have and just haven't posted anything about it. Would be interesting to do though. Oh that's good to know, thank you for the info! ... View more

Thanks for the quick reply! I see though. Seems like it's possible, but not easy. Have you seen anywhere if this has been done by somebody with Splunk yet? ... View more

Ok i see, that makes sense. Thank you for your feedback! ... View more

Ah sorry, I meant to ask how could I forward the data to the third party server in addition to the indexer by using the universal forwarder. In other words, how could I configure the universal forwarder to forward the data to the other server? Or really, what would be the best way to do it in this scenario? ... View more

Thank you very much for the information and feedback. I'm working through troubleshooting this some more using the info you provided. The version of ES I'm running is the latest version, 4.7.2. I think the first bullet you suggested is correct, there has to be something blocking the splunk server from reaching out to the threat intelligences, my next step is to just figure out what exactly that is. I will update this post with an answer when I figure it out, or some more questions if I get stuck again. Thanks! ... View more

Thank you! ... View more

Hello adonio, Yes, the first question I realized I asked about the brocade app when I realized I am using the brocade add-on, not the brocade app. I was going to delete that question but have no idea how. For both cases, I read through the documentation first, which is how I ended up here with my question. The documentation for these add-ons do not cover my situation, as far as I understand. I understand that the add-on works if you forward data directly from the device (juniper in this case) to the splunk indexers, and have the add-on installed on the search heads and indexers. However, my situation is different. As I explained in my question, my situation involves a syslog collection server. All of the juniper syslogs are being forwarded to the syslog server, written to a directory that contains rolling log files, then the universal forwarder that is installed on the syslog server is taking the rolling log files and forwarding them to the indexers. I have several types of devices all going to the syslog server, not just juniper. The Universal Forwarder forwards all of the devices data to the indexers using the same port, since all of the logs are being written to the same parent directory. The add-ons seem to be designed specifically to have the devices forward their data directly to the splunk indexers on a specific port. But, I don't have that option. So, I am left wanting to know if the add-on can work with my situation where the devices are forwarding their syslog data to a syslog server, and not directly to the splunk indexers. ... View more

That's a good point, wouldn't hurt to use the support! That makes sense that the sendRcvTimeout might not be the real root of the problem. Those two symptoms sounds pretty accurate, I'm going to dig some more into this. Thank you very much for the all the insight and help! ... View more

SloshBurch, thank you for pointing this out. I have not opened a case before as I'm still rather new with Splunk, and did not think about doing so. jhupka, that is a good point with losing 50% capacity with only 2 indexers, that is something that I will be considering. I did some more troubleshooting and think I fixed the problem, as I have not gotten any errors or other problems since implementing the fix. That makes sense that when the index queue is full, the other queues get backed up, I just don't understand how the problem was causing the index or other queues to be completely full and backed up. Perhaps you have some insight or better understanding on the situation? So what I discovered was when both of my search heads were active, I had the problem of all queues being full and all data being blocked from being forwarded to the indexers. However, if stopped the splunk service from running on just one of the search heads (essentially shutting off one search head and having just one search head running, did not matter which search head), the queues would empty and I had no problems. When both were running, in addition to having all the data being blocked, I had an error on the search heads stating that the configuration bundle was being timed out when it attempted to replicate to the search peers (the default time of 60 seconds, as set with sendRcvTimeout in distsearch.conf). So, I changed the sendRcvTimeout from the default 60 seconds to 300 seconds (5 minutes) on the search heads, restarted the splunk service on the search heads, and now everything has been working without errors since. I'm able to search on both search heads and ingest data without any errors. So two questions I don't understand is what configuration bundle is being replicated from the search heads to the search peers, and how is that causing all the queues starting with the index queue to fill up and not empty? ... View more

Hi SloshBurch, Yes, I've been told that it is best to have at least 3 search heads, and is something I am considering. Thank you for the advice! ... View more

Hi jhupka, thank you very much for the response. I do have the Splunk Monitoring Console configured on the license master. Looking through the Indexing menu, I saw that in the data pipeline for my two indexers the fill ratios are 100% for all 4 Queues (Parsing Queue, Aggregator Queue, Typing Queue, and Index Queue). None of the indexes are 100% full, except one of the indexes is over 99% full. I don't appear to be having any CPU or memory issues. Here is a section of splunkd.log (I replaced sensitive information with a description of what it is) 05-30-2017 14:59:27.656 -0400 WARN TcpOutputProc - Cooked connection to ip=Indexer#1IP:Port# timed out 05-30-2017 14:59:30.058 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 10 seconds. 05-30-2017 14:59:40.069 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 20 seconds. 05-30-2017 14:59:50.081 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 30 seconds. 05-30-2017 15:00:00.093 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 40 seconds. 05-30-2017 15:00:05.140 -0400 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_ClusterMasterIP_Port#_ClusterMasterIP_ClusterMasterHostName_80755AFE-FDB3-4B2A-8CEE-8B2DB16B69D4 05-30-2017 15:00:14.009 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 10 seconds. 05-30-2017 15:00:24.021 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 20 seconds. 05-30-2017 15:00:34.032 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 30 seconds. 05-30-2017 15:00:38.637 -0400 WARN TcpOutputProc - Read operation timed out expecting ACK from Indexer#2:Port# in 300 seconds. 05-30-2017 15:00:38.637 -0400 WARN TcpOutputProc - Possible duplication of events with channel=source::/opt/splunk/var/log/splunk/splunkd.log|host::ClusterMasterHostName|splunkd|199, streamId=2032662337362434837, offset=21761155 subOffset=199 on host=Indexer#2:Port# 05-30-2017 15:00:38.637 -0400 WARN TcpOutputProc - Possible duplication of events with channel=source::audittrail|host::ClusterMasterHostName|audittrail|, streamId=0, offset=0 on host=Indexer#2IP:Port# 05-30-2017 15:00:44.043 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 40 seconds. 05-30-2017 15:00:54.054 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 50 seconds. 05-30-2017 15:00:58.559 -0400 WARN TcpOutputProc - Cooked connection to ip=Indexer#2:Port# timed out 05-30-2017 15:01:04.067 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 60 seconds. 05-30-2017 15:01:05.147 -0400 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_ClusterMasterIP_Port#_ClusterMasterIP_ClusterMasterHostName_80755AFE-FDB3-4B2A-8CEE-8B2DB16B69D4 05-30-2017 15:01:14.078 -0400 WARN TcpOutputProc - Forwarding to indexer group myGroupName blocked for 70 seconds. So, I'm assuming that the problem here is the 4 data pipeline queues are filling to max capacity, right? How would that be happening though, and how could I fix that and prevent it from happening again? ... View more

Thank you very much for this insight! That's a good point with the license master, that shouldn't really be needed. Would I be able to monitor the cluster with the monitoring console instead of the cluster master? Or are there things that I can only do from the cluster master web? ... View more