It's a known issue for HEC that for indexed extraction `maxEventSize`  is not honored hence max json payload is 512KB. Splunk 9.0 and above issue is fixed. ... View more

The reason for memory growth is auto tuning for max_inactive and lowater_inactive configurations in limits.conf. With auto tuning,  max_inactive = 96 ( if total system memory is < 8GB) max_inactive = 1024  ( if total system memory is >= 8GB and < 26GB ) max_inactive = 32768  ( if total system memory is >= 26GB) lowater_inactive = (max_inactive/3) max_inactive = <integer> * The Maximum number of inactive input channel configurations to keep in cache. * Each source/sourcetype/host combination requires an independent input channel, which contains all relevant settings for ingestion. * When set to 'auto', the Splunk platform will tune this setting based on the physical RAM present in the server at startup. * Increasing this number might help with low ingestion throughput when there are no blocked queues (i.e., no 'blocked=true' events for 'group=queue' in metrics.log), and splunkd is creating a very high number of new input channels (see the value of 'new_channels' in 'group=map, name=pipelineinputchannel', also in metrics.log), usually in the order of thousands. However, this action is only effective when those input channels could have been reused: for example, the source, sourcetype, and host fields are not generated randomly and tend to be reused within the lifetime of cached channel entries. * Default: auto lowater_inactive = <integer> * Size of the inactive input channel cache after which entries will be considered for recycling: having its memory reused for storing settings for a different input channel. * When set to 'auto', the Splunk platform will tune this setting value based on the value of 'max_inactive'. * Default: auto As a result Universal forwarder/Search Head is creating a minimum cache of inactive channels as per lowater_inactive configuration. However these high settings are useful for only Indexer and Heavy forwarder. For edge Universal forwarder and search head these high values don't matter. Workaround: Set `max_inactive` as low as possible. Example [input_channels] max_inactive=10 ... View more

Run following search   index=_internal source=*metrics.log* ratio > 0.8| timechart max(ratio) by thread   Search will show threads using higher cpu. If one of the thread is  SplunkConfigChangeWatcherThread, then disable new splunk config change tracker feature. In server.conf [config_change_tracker] disabled = true   ... View more

Fix is in 9.1(next major release)  for this type of scenario. Try following workaround to reduce outage. In server.conf [queue=indexQueue] maxSize=500MB In indexes.conf [default] throttleCheckPeriod=5 maxConcurrentOptimizes=1 maxRunningProcessGroups=32  processTrackerServiceInterval=0 ... View more

Explain timeout configs, when and how to use these configs.                                                                                             Instance Conf file  Stanza Config Default Max Recommended Purpose When to use Search Head distsearch.conf [distributedSearch] statusTimeout 10 sec 120 sec Connect/read/write timeout to get search peer's info ( from SH main splunkd to peer main splunkd).  One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)                       sendTimeout 30 sec 120 sec send/write timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for send/write failure. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)       receiveTimeout 600 sec 1200 sec read/receive timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for read failure.        connectionTimeout 10 sec 120 sec connect timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for connect failure. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)       authTokenConnectionTimeout 5 sec 120 sec connect timeout to get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.  This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)       authTokenSendTimeout 10 sec 120 sec send/write timeout get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.  This ca happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)       authTokenReceiveTimeout 10 sec 120 sec read/receive timeout get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.     [replicationSettings] connectionTimeout 60 sec 120 sec connect timeout for actual bundle replication from SH main splunkd to peer main splunkd. One thread handles one peer.  Impacts actual bundle replication. Normally this never fails.       sendRcvTimeout  60 sec 120 sec  read/write timeout for actual bundle replication from SH main splunkd to peer main splunkd. One thread handles one peer.  Impacts actual bundle replication. Normally this never fails.   limits.conf [search] remote_timeline_connection_timeout 5 sec 30 sec connect timeout for SH search process to  peer's main splunkd for timeliner. Multi-threaded. Normally this never fails.       remote_timeline_send_timeout 10 sec 30 sec send timeout for SH search process to  peer's main splunkd for timeliner. Multi-threaded. Normally this never fails.   server.conf [shclustering] cxn_timeout_raft 2 sec 30 sec SH to SH raft communication connect timeout. One thread handles one member.         send_timeout_raft 5 sec  30 sec SH to SH raft communication send timeout. One thread handles one member.         rcv_timeout_raft 5 sec 30 sec SH to SH raft communication read timeout. One thread handles one member.   Indexer distsearch.conf [replicationSettings] sendRcvTimeout 60  sec 120 sec Read/Write timeout by search peer for bundle replication ( from peer main splunkd to SH main splunkd) Impacts actual bundle replication. Indexer/ Search head server.conf [httpServer]  busyKeepAliveIdleTimeout  12 sec  120 sec SH Peer/Member HTTP server keep-alive connection idle timeout. When number of HTTP connections > 60, it disconnects any idle connection. Impacts bundle replication and search.       streamInWriteTimeout 5 sec 30 sec Read/Write timeout when receiving http request by http server Normally this never fails.       dedicatedIoThreads auto 5 Http Listener thread will let additional I/O threads do read/write from socket.  Very useful for knowledge replication/config replication/raft/search artifact replication  and searches. You will see in metrics.log that  mgmt_httpd  is consistently > 0.8 index=_internal source=*metrics.log* mgmt_httpd | timechart span=30s max(mgmt_httpd) source=*metrics.log* thread=*dedicated* | timechart span=30s max(ratio) by thread This can be added to web.conf if you see consistently  webui ratio > 0.8. UI response is sluggish. Metrics - group=dutycycle, name=management, thread=webui, ratio=       dedicatedIoThreadsSelectionPolicy round_robin weighted_random         [ sslConfig ] useClientSSLCompression true false SSL compression used by Http client for SH→SH / SH→IDX / SH→CM/ IDX->CM Change this settings  if increase in network bandwidth usage is not an issue. DC   deploymentclient.conf [deployment-client] connect_timeout 60 120 DC->DS If DCs are timing out while connecting("Connect timeout"). DC deploymentclient.conf [deployment-client] send_timeout 60 120 DC->DS If there is a send timeout while sending request to DS DC deploymentclient.conf [deployment-client] recv_timeout 60 120 DC->DS If DC read timed out while waiting for a response from DS.         ... View more

The root cause of the problem is  SPL-210081.   Fixed on httpout side. Requires upgrade to Forwarder. SPL-224974 is re-fixing SPL-210081. Fixed on the server side.  This option is better because  Don't have to upgrade tons of forwarders. Just upgrade HF(or whichever layer is receiving httpout)  It will also take care of  all s2s clients( splunk or 3rd party software) . SPL-229622 is new fix that is likely to make into 9.0.3 is another enhanced version of SPL-224974. But you should be fine with  SPL-210081(UF upgrade) or SPL-224974(HF upgrade). ... View more

This issue is applicable only on UF if sendCookedData=false.  You may want to check General forwarder memory growth . Do you see a trend, where channels are gradually increasing and one of the tcpout group is set to sendCookedData=false? Metrics.log entries pointing to  channels growth. INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12950 INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12955 INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12959 It's a known issue with universal forwarder where some sources (monitor/udp/tcp) may emit EOF marker late. This results in UF not able to free up channels. However the same config ( 3rd party forwarding) on HF is not an issue. One way to test if you are hitting the issue, restart 3rd party receiver. If UF memory drops immediately, apply following workaround.   Until the issue is fixed by new patch, use following workaround. Set following config for 3rd party tcpout group only. forceTimebasedAutoLB=true This setting will force close connections and allow consolidation of channels. Thus every `autoLBFrequency` interval reclaim memory. Note: For all 8.2.x and older releases, forceTimebasedAutoLB works only if the total number of distinct 3rd party valid target <ip address:port> combinations are > 1. If there is only one receiver, forceTimebasedAutoLB setting is no-op. Please don't add dummy/no-existent <ip address:port> combination. If your 3rd party receiver is on same box as UF, then you should be able to  make > 1 receivers by adding `127.0.0.1` in `server` list. [tcpout:thirdpartytcpout] server=127.0.0.1:<target port>, <ip address of UF host>:<target port> sendCookedData=false For 9.x UFs, connectionsPerTarget setting, if set to `auto` or > 1, then forceTimebasedAutoLB=true works for single receiver tcpout groups. connectionsPerTarget = [<integer>|auto] * The maximum number of allowed outbound connections for each target IP address as resolved by DNS on the machine. * A value of "auto" or < 1 means splunkd configures a value for connections for each target IP address. Depending on the number of IP addresses that DNS resolves, splunkd sets 'connectionsPerTarget' as follows: * If the number of resolved target IP addresses is greater than or equal to 10, 'connectionsPerTarget' gets a value of 1. * If the number of resolved target IP addresses is greater than 5 and less than 10, 'connectionsPerTarget' gets a value of 2. * If the number of resolved target IP addresses is greater than 3 or less than equal to 5, 'connectionsPerTarget' gets a value of 3. * If the number of resolved target IP addresses is less than or equal to 3, 'connectionsPerTarget' gets a value of 4. * Default: auto   ... View more

New behavior to kill DNS lookup traffic, by default searches use https://<ip>:<port> instead of https://<FQDN hostname>:<port> To revert back to previous behavior of <hostname:port>, there is a config for 9.0.x. distsearch.conf [distributedSearch] useIPAddrAsHost=false For future releases, if sslVerifyServerName=true , then automatically useIPAddrAsHost=false ... View more