sourcetype in props.conf in case-senstive [MySourcetype] is different from [mysourcetype] From Splunk Documentation (pros.conf) By default, [source::<source>] and [<sourcetype>] stanzas match in a case-sensitive manner, while [host::<host>] stanzas match in a case-insensitive manner. This is a convenient default, given that DNS names are case-insensitive. ... View more

This is how I solved this as well. ... View more

Did you find any solution for this? I can see the same behaviour for some other fields but its in a kind of loop to get some Calculated fields disabled and some keys added. Its doing it for 10 to 15 seconds hundrets of times for the same field, sometimes longer, what results in a long search time. It is not doing it for every event. There is not difference if i get 100 events back or 10K. The iteration is done up to a limit i think, or for each index maybe??   ... View more

Just an update on my end on this.  An upgrade fixed the problem. I think it was related to a setting around sslCompression internally in Splunk that looks to have been the issue. The new version 8.2.2 has this setting set to false, it was true in the old version we ran (8.1.3).   In server.conf on both search heads (search head cluster) and indexeres (indexer cluster): [sslConfig] useClientSSLCompression = false   I saw that this fixed the same problems on another customer on 8.1.4 (I think).   useClientSSLCompression is default true in older versions, it is false on the new.   If you run older versions of splunk and search head cluster (I have not seen it on single search head and indexer cluster) - you could try the above to see if that works.   Regards André ... View more

We see the same on a 8.1.4 search head cluster, if you have any info you got from splunk support ? Or we will try and upgrade first. ... View more

No errors in the monitoring console. I see this only in distributed setups, so no errors when searching locally on a all-in-one box. The worst installation has 1500+ broken pipes on these (search: 😞 09-09-2021 13:32:18.574 +0200 WARN HttpListener - Socket error from "search-head-ip":42800 while accessing /services/streams/search: Broken pipe and this type ERROR: 09-09-2021 13:40:33.742 +0200 ERROR HttpListener - Exception while processing request from "search-head-ip":45930 for /services/search/jobs/remote_splunksh03."domainname"_subsearch_nested_8294df4a2b86339a_1631187558.3/search.log: Broken pipe Other installations have between 20 and 50 of those pr 24 hours. These warnings and errors are logged a lot when searches start having errors like this (from search.log): 09-09-2021 14:00:45.634 ERROR SearchResultTransaction - Got status 502 from https://"indexer-ip":8089/services/streams/search?sh_sid=1631188834.20962_7D6DF087-C582-4B67-A82D-BD1F18B5BEA5 09-09-2021 14:00:45.634 INFO XmlParser - Entity: line 1: parser: Document is empty 09-09-2021 14:00:45.635 ERROR SearchResultParser - HTTP error status message from https://"indexer-ip":8089/services/streams/search?sh_sid=1631188834.20962_7D6DF087-C582-4B67-A82D-BD1F18B5BEA5: Error connecting: Connect Timeout 09-09-2021 14:00:45.635 WARN SearchResultParserExecutor - Error connecting: Connect Timeout for collector=splunkidx01.domainname 09-09-2021 14:00:45.635 ERROR DispatchThread - sid:1631188834.20962_7D6DF087-C582-4B67-A82D-BD1F18B5BEA5 Unknown error for indexer: splunkidx01.domainname. Search Results might be incomplete! If this occurs frequently, check on the peer. The peers are always up, the ulimit, thread limit and socket limit are all OK when viewing in the splunkd log when splunk is starting. (It is systemd managed on an ubuntu 20.02LTS, so ulimit set there, and is 65535)   The servers (sh's and indexers) are 80 cores (40 hyperthreaded, dual socket), 386 GB ram, pure SSD. Seach head cluster with 3 nodes Indexer cluster with 4 nodes.   We see no other network errors. No problems with 9887 replication data in the indexer cluster, no problems ingesting data.   ... View more

I also have this problem, any further updates to this? ... View more

https://docs.splunk.com/Documentation/Splunk/8.0.1/ReleaseNotes/MeetSplunk#What.27s_New_in_8.0.1 ... View more

Thank you. We are running 1.3.33, sorry I missed that. The guide say that only one fifo_reader.py should be running, and that: "You will observe that killing the nmon process will automatically terminate the fifo_reader.py", which does not. I can see that it then states "If the processes do not stop, then your problem became mine and please open an issue" - I will do that 🙂 ... View more

Hi Just saw this for 7.2.5, fixed issues in release notes: 2019-03-07 SPL-167347, SPL-165968 Frequent searches with outputlookup may trigger highly increased KV Store storage usage or in some cases crash of the mongod process André ... View more

Ahh yes - I see it now. Thank you ... View more

Waiting for a mod to approve my answer but set the backfill time in the CIM Setup in Enterprise Security it is an issue where the backfill is extending beyond the earliest time of data. ... View more

As per the documentation , AMQP 1.0 is not supported. ... View more

Yes, so if the server after a restart, maybe only in context of an OS upgrade, has a wrong system clock set until it gets a hold of its NTP server, it could yield some events that are off. I may have been unclear in my question above. The "looks like it may be related to a restart of the system". The "system" is the OS the forwarder runs on. ... View more

Thank you Yes, I will try to run it in the trial period also. We are just in the scoping phase of a project. We have the enterprise license in place. But we are uncertain of how much we should guess for Exchange license. I think we maybe should take it as a minimum of 5 GB Exchange pr day, and maybe more. The financial part should be in place before we start investing in time and licenses for the solution. So the above is fine for guessing the license size initially. Thank you André ... View more

OK After a lot of debugging in vain ... I rebooted the entire server. And then it worked ! I have no clue why. If anyone can elaborate, please do so, maybe others will come in the same situation. ... View more

Oh, you meant the slaves, as in license slaves, and not the slaves to the master in the cluster. I have removed the SA-VMW-Licensecheck from the search head, and will try and see if that helps. ... View more

no problem 😉 converted to answer so feel free to mark this as answered - thx ... View more