From what you’ve described, the “Socket error … broken pipe” on /services/streams/search is typically not the actual root failure but a downstream symptom of Splunk search connections being cut unexpectedly between the Search Head and indexers. In most cases, this happens when the indexer side closes the streaming connection because the underlying search process has stalled, been terminated, or hit a temporary resource constraint, even if overall system metrics like IOPS or CPU look normal on averages. The fact that you are also seeing preforked search processes hanging and captain disconnect messages strongly suggests intermittent search pipeline saturation or scheduling delays rather than a simple networking or thread configuration issue. Increasing values like maxSockets or maxThreads in server.conf may reduce visible errors temporarily, but it often just pushes more load into an already constrained search layer instead of solving the underlying bottleneck. In environments like this, the real issue is usually found in short bursts of CPU pressure, dispatch directory latency, search concurrency limits in limits.conf, or file descriptor exhaustion at the OS level. This is similar to diagnosing unstable flow in real systems where the average looks fine but momentary pressure drops cause failures, much like how services such as Plumber Singapore would investigate intermittent pipe blockages rather than just increasing pump capacity. A good next step is to correlate timestamps between indexer search process logs and SH streaming errors to identify whether searches are being killed, delayed, or timing out under load. ... View more

yes, we want to know how integrate splunk into keycloak as an IDP Provider? ... View more

The below is from Ovie in the Splunk community Slack channel #enterprise-security: Not sure if it is your issue but it is supposed to be related to duplicates like that. So it might apply to some of you. This can happen because of phased_execution_mode https://docs.splunk.com/Documentation/ES/5.2.0/RN/KnownIssues 2019-04-08 SOLNESS-18603 Incident Review: eventCount does not match resultCount causing display issues (such as events being displayed twice) Workaround: Set phased_execution_mode to singlethreaded For: limits.conf [search] phased_execution_mode = singlethreaded https://docs.splunk.com/Documentation/ES/5.1.0/Install/DeploymentPlanning#Splunk_Enterprise_system_requirements Splunk Enterprise Security 5.1 is compatible with Splunk Enterprise 7.1.0 and 7.1.1 only by setting phased_execution_mode=singlethreaded in the [search] stanza of the $SPLUNK_HOME/etc/system/local/limits.conf file to avoid an issue that is fixed in Splunk Enterprise 7.1.2. However, if you apply this workaround for 7.1.0 and 7.1.1 and then upgrade Splunk Enterprise but remain on ES 5.1, then you need to set it back to phased_execution_mode=multithreaded. Splunk Enterprise Security 5.2.x is compatible with Splunk Enterprise 7.1.0 and 7.1.1 only by setting phased_execution_mode=singlethreaded in the [search] stanza of the $SPLUNK_HOME/etc/system/local/limits.conf file to avoid an issue that is fixed in Splunk Enterprise 7.1.2. Bottom line is this setting has caused some serious grief. ... View more

Thx guys! Works great! 🙂 ... View more

I found something about logs, i use Windows Server 2012 R2 in french and dashboards on Windows Infrastructure App don't read french logs so it didn't work. Is there a way to resolve this ? Thanks Geoffrey ... View more

Yes, I ditched the jQuery xhr method and used plain JavaScript xmlHttpRequest instead. This way the header is not added and you can interact with the API. Since there has to be some creds for the API which must be accessible from the JavaScript I think this is not a good and secure solution. Afterwards I thought it would be maybe better to create a custom command for this, because custom commands are receiving an API token on execution. This would solve the "authentication problem" probably more elegantly. Otherwise I think there are some ways to store creds more securely than just putting them into the JavaScript...just FYI if you want to build something similar. In my case I granted the used API user only as much permissions as needed and additionally I created a dashboard based on audit data to monitor if this is not abused by some attacker with access to splunk. ... View more