The root cause of the problem is  SPL-210081.   Fixed on httpout side. Requires upgrade to Forwarder. SPL-224974 is re-fixing SPL-210081. Fixed on the server side.  This option is better because  Don't have to upgrade tons of forwarders. Just upgrade HF(or whichever layer is receiving httpout)  It will also take care of  all s2s clients( splunk or 3rd party software) . SPL-229622 is new fix that is likely to make into 9.0.3 is another enhanced version of SPL-224974. But you should be fine with  SPL-210081(UF upgrade) or SPL-224974(HF upgrade). ... View more

Do you see a trend, where channels are gradually increasing and one of the tcpout group is set to sendCookedData=false? Metrics.log entries pointing to  channels growth. INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12950 INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12955 INFO  Metrics - group=map, ingest_pipe=0, name=pipelineinputchannel, current_size=12959 It's a known issue with universal forwarder where some sources (monitor/udp/tcp) may emit EOF marker late. This results in UF not able to free up channels. However the same config ( 3rd party forwarding) on HF is not an issue. One way to test if you are hitting the issue, restart 3rd party receiver. If UF memory drops immediately, apply following workaround.   Until the issue is fixed by new patch, use following workaround. Set following config for 3rd party tcpout group only. forceTimebasedAutoLB=true This setting will force close connections and allow consolidation of channels. Thus every `autoLBFrequency` interval reclaim memory. Note: For all 8.2.x and older releases, forceTimebasedAutoLB works only if the total number of distinct 3rd party valid target <ip address:port> combinations are > 1. If there is only one receiver, forceTimebasedAutoLB setting is no-op. Please don't add dummy/no-existent <ip address:port> combination. If your 3rd party receiver is on same box as UF, then you should be able to  make > 1 receivers by adding `127.0.0.1` in `server` list. [tcpout:thirdpartytcpout] server=127.0.0.1:<target port>, <ip address of UF host>:<target port> sendCookedData=false For 9.x UFs, connectionsPerTarget setting, if set to `auto` or > 1, then forceTimebasedAutoLB=true works for single receiver tcpout groups. connectionsPerTarget = [<integer>|auto] * The maximum number of allowed outbound connections for each target IP address as resolved by DNS on the machine. * A value of "auto" or < 1 means splunkd configures a value for connections for each target IP address. Depending on the number of IP addresses that DNS resolves, splunkd sets 'connectionsPerTarget' as follows: * If the number of resolved target IP addresses is greater than or equal to 10, 'connectionsPerTarget' gets a value of 1. * If the number of resolved target IP addresses is greater than 5 and less than 10, 'connectionsPerTarget' gets a value of 2. * If the number of resolved target IP addresses is greater than 3 or less than equal to 5, 'connectionsPerTarget' gets a value of 3. * If the number of resolved target IP addresses is less than or equal to 3, 'connectionsPerTarget' gets a value of 4. * Default: auto   ... View more

New behavior to kill DNS lookup traffic, by default searches use https://<ip>:<port> instead of https://<FQDN hostname>:<port> To revert back to previous behavior of <hostname:port>, there is a config for 9.0.x. distsearch.conf [distributedSearch] useIPAddrAsHost=false For future releases, if sslVerifyServerName=true , then automatically useIPAddrAsHost=false ... View more

1. Any monitor/batch inputs stanza with multiline_event_extra_waittime=true on UF should have EVENT_BREAKER_ENABLE=true and EVENT_BREAKER = <regular expression> in props.conf. Check LINE_BREAKER on HF/indexing side for these sourcetypes and set EVENT_BREAKER. For all sourcetypes that has set multiline_event_extra_waittime=true in inputs.conf on UF you have to set EVENT_BREAKER_ENABLE=true EVENT_BREAKER=<whatever LINE_BREAKER for the sourcetype set on HF or IDX> 2. For all other monitor/batch input stanzas on UF, have just EVENT_BREAKER_ENABLE=true in props.conf. For all sourcetypes that has multiline_event_extra_waittime=false in inputs.conf on UF just EVENT_BREAKER_ENABLE=true 3. For all TAs set EVENT_BREAKER same as LINE_BREAKER(on HF/ indexer side of props.conf) 4. Always EVENT_BREAKER(on UF side of props.conf) should be set to LINE_BREAKER(on HF/ indexer side of props.conf) value.   ... View more