If you are using persistent queue, you are likely to see following errors in splunkd.log      ERROR TcpInputProc - Encountered Streaming S2S error 1. "Cannot register new_channel" 2. "Invalid payload_size" 3. "Too many bytes_used" 4. "Message rejected. Received unexpected message of size" 5. "not a valid combined field name/value type for data received"     Many other streaming S2S streaming errors.   If your deployment is impacted, you should upgrade your HF/IHF/IUF/IDX instance that is using persistent queue to following patches. 9.3.2/9.2.4/9.1.7 This patch also fixes all the PQ related crashes and other PQ issues.  With 9.3.2/9.2.4/9.1.7 there are no known PQ issues. ... View more

Splunk version 9.0.8/9.1.3/9.2.x and above has added capability to process key value pairs that will be added at index time on all events flowing through the input.  Now it's possible to "tag" all data coming into a particular HEC token. HEC will support all present and future inputs.conf.spec configs(_meta/TCP_ROUTING/SYSLOG_ROUTING/queue etc.). ... View more

Apply following workaround in default-mode.conf Additionally you can also push this change via DS push across thousands of universal forwarders. Add index_thruput in the list of disabled processors.  Add following line as is in default-mode.conf.   #Turn off a processor [pipeline:indexerPipe] disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor     NOTE:  PLEASE DON'T APPLY ON HF/SH/IDX/CM/DS. You want to use different app( not SplunkUniversalForwarder app) to push the change. ... View more

Different crashes during tcpout reload.   Received fatal signal 6 (Aborted) on PID . Cause: Signal sent by PID running under UID . Crashing thread: indexerPipe_1 Backtrace (PIC build): [0x000014BC540AFB8F] gsignal + 271 (libc.so.6 + 0x4EB8F) [0x000014BC54082EA5] abort + 295 (libc.so.6 + 0x21EA5) [0x000055BCEBEFC1A7] __assert_fail + 135 (splunkd + 0x51601A7) [0x000055BCEBEC4BD9] ? (splunkd + 0x5128BD9) [0x000055BCE9013E72] _ZN34AutoLoadBalancedConnectionStrategyD0Ev + 18 (splunkd + 0x2277E72) [0x000055BCE905DC99] _ZN14TcpOutputGroupD1Ev + 217 (splunkd + 0x22C1C99) [0x000055BCE905E002] _ZN14TcpOutputGroupD0Ev + 18 (splunkd + 0x22C2002) [0x000055BCE905FC6F] _ZN15TcpOutputGroups14checkSendStateEv + 623 (splunkd + 0x22C3C6F) [0x000055BCE9060F08] _ZN15TcpOutputGroups4sendER15CowPipelineData + 88 (splunkd + 0x22C4F08) [0x000055BCE90002FA] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 362 (splunkd + 0x22642FA) [0x000055BCE9829628] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 72 (splunkd + 0x2A8D628) [0x000055BCE8D29D25] _ZN8Pipeline4mainEv + 1157 (splunkd + 0x1F8DD25) [0x000055BCEBF715EE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51D55EE) [0x000055BCEBF716FB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51D56FB) [0x000014BC552AC1DA] ? (libpthread.so.0 + 0x81DA)       Another reload crash   Backtrace (PIC build): [0x00007F456828700B] gsignal + 203 (libc.so.6 + 0x2100B) [0x00007F4568266859] abort + 299 (libc.so.6 + 0x859) [0x0000560602B5B4B7] __assert_fail + 135 (splunkd + 0x5AAA4B7) [0x00005605FF66297A] _ZN15TcpOutputClientD1Ev + 3130 (splunkd + 0x25B197A) [0x00005605FF6629F2] _ZN15TcpOutputClientD0Ev + 18 (splunkd + 0x25B19F2) [0x0000560602AD7807] _ZN9EventLoop3runEv + 839 (splunkd + 0x5A26807) [0x00005605FF3555AD] _ZN11Distributed11EloopRunner4mainEv + 205 (splunkd + 0x22A45AD) [0x0000560602BD03FE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x5B1F3FE) [0x0000560602BD050B] _ZN6Thread8callMainEPv + 139 (splunkd + 0x5B1F50B) [0x00007F4568CAD609] ? (libpthread.so.0 + 0x2609) [0x00007F4568363353] clone + 67 (libc.so.6 + 0xFD353) Linux / myhost / 5.15.0-1055-aws / #60~20.04.1-Ubuntu SMP Thu assertion_failure="!_hasDataInTransit" assertion_function="virtual TcpOutputClient::~TcpOutputClient()"           Since Splunk 9.2, splunk outputs.conf is reloadable. Whenever DC pulls bundle from DS, depending on the changes, during reload, conf files are reloaded. One of the conf file is outputs.conf. Prior to 9.2 outputs.conf was not reloadable that means hitting following endpoint would do nothing. /data/outputs/tcp/server or  https://<host>:<port>/servicesNS/-/-/admin/tcpout-group/_reload Behavior is changed from 9.2 and now outputs.conf is reloadable. However reloading outputs.conf is very complex process as it involves shutdown tcpout groups safely. Still there are cases where splunk crashes. We are working on fixing reported crashes. Workaround As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications Set in local/apps.conf as a workaround. [triggers] reload.outputs = simple With setting above, splunk will take no action on tcpout(outputs.conf) reload( a behavior  that was before 9.2)   If outputs.conf is changed via DS, restart splunk. ... View more

If you use Splunk Enterprise or SplunkCloud, you can guard against loss of data when forwarding by enabling the indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data that the receiver does not acknowledge as "received". You enable indexer acknowledgment on the forwarder, in the outputs.conf file. See how acknowledgement works. See how acknowledgement failure is handled. However if the forwarder is restarted/stopped while waiting for indexer acknowledgment, in most of the cases unacknowledged data is not resend upon forwarder restart/start. That is because indexer acknowledgment is just an agreement  between the output processor and the target server.  There are 4 major input types: File inputs(monitor/batch mode) Modular inputs Network inputs(TCP/UDP) HTTP inputs(http receiver endpoints/http event collector) Only the file input in the monitor mode can resend data if the forwarder is restarted/stopped while waiting for indexer acknowledgment.   Acknowledgement is sent back to the forwarder after replication factor is met. That means for rep factor 3, source indexer waits on acknowledgement from 2 replication target indexers. Inputs on forwarders are not aware of the indexer acknowledgement process. Latency increases when the target server is not an indexer as the intermediate tier will wait for acknowledgement before acknowledging back to edge forwarder. For more information, see the outputs.conf spec file. ... View more

If you find issue happens after windows server is restarted. Restarting splunk universal forwarder fixes the issue. Then try one of the following workarounds. Use 'Delayed Start' for the Splunk Forwarder service. ( https://community.splunk.com/t5/Getting-Data-In/Why-quot-FormatMessage-error-quot-appears-in-indexed...). However it's hard to configure thousands of DCs. Or Configure  interval as cron schedule instead. interval = [<decimal>|<cron schedule>]   [WinEventLog] interval=* * * * *   By default wineventlog interval is 60 sec. That means as soon as splunk is restarted, wineventlog (or any modinput) is immediately started. Subsequently every 60( configured interval) splunk checks if modinput is still running. If not re-launch modinput. Instead of setting interval 60 sec, if we use cron schedule to run every minute, then splunk is not going to launch modinput immediately. So essentially the idea is to convert interval setting from decimal to cron schedule to introduce a delay. If above does not solve the issue, meaning the issue is not related to windows server  restart.  Try one of the following workarounds.   Workaround 1. in inputs.conf Stop UF Set batch_size to 1. Start UF [<impacted channel>] batch_size=1 Workaround 2. in inputs.conf Stop UF backup checkpoint file $SPLUNK_HOME\var\lib\splunk\modinputs\WinEventLog\<impacted channel> . Saves current record id  In inputs.conf for impacted channel add use_old_eventlog_api=true Start UF Stop  UF after 30 sec Replace current record with  record id found in step 2. So that ingestion starts from right place. When setting use_old_eventlog_api=false ( in case if it does help). Follow all above steps so that UF correctly starts from desired record id.  [<impacted channel>] use_old_eventlog_api=true ... View more

While there can be so many reasons for memory growth, one of the reason could be increased memory usage by idle search process pool(search-launcher).   index=_introspection component=PerProcess host=<any one SH or IDX host> | timechart span=5s sum(data.mem_used) as mem_usedMB by data.process_type useother=f usenull=f   Example   If memory usage by `search-launcher` is way higher than `search` Then idle search process pool(search-launcher) is wasting system memory. If you see above trend, we want to reduce idle search process pool. There are several options to reduce idle search process pool in limits.conf One option is to set enable_search_process_long_lifespan = false in server.conf( new option in 9.1 and above)   enable_search_process_long_lifespan = <boolean> * Controls whether the search process can have a long lifespan. * Configuring a long lifespan on a search process can optimize performance by reducing the number of new processes that are launched and old processes that are reaped, and is a more efficient use of system resources. * When set to "true": Splunk software does the following: * Suppresses increases in the configuration generation. See the 'conf_generation_include' setting for more information. * Avoids unnecessary replication of search configuration bundles. * Allows a certain number of idle search processes to live. * Sets the size of the pool of search processes. * Checks memory usage before a search process is reused. * When set to "false": The lifespan of a search process at the 50th percentile is approximately 30 seconds. * NOTE: Do not change this setting unless instructed to do so by Splunk Support. * Default: true   Why idle search process pool appears to be un-used(more idle searches compared to the actual number of searches running on peer)? Before a search request is dispatched to peers, SHCs/SHs also need to first find  the common knowledge bundle across peers. On peer, only an idle search process created with matching common knowledge bundle is eligible for re-use. That's why in most cases idle search process pool remains un-used as overall idle search process pool is a collection of idle search processes  having association with different knowledge bundles.  Now think of a scenario having multiple SHC clusters (example ES/ITSI/ad-hoc etc). Each SH cluster replicating it's own knowledge bundles. The idle search process pool is a collection of idle search processes  having association with different knowledge bundles from different search heads. ... View more

Can we apply following example on UF? Keep specific events and discard the rest https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest The answer is no. The example is for any non-UF instance. For UF you can modify the example Edit props.conf and add the following: [source::/var/log/messages] TRANSFORMS-set= setnull,setparsing Edit transforms.conf and add the following: [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = \[sshd\] DEST_KEY = _TCP_ROUTING FORMAT = <valid-tcpoutgroup(s)> Or Edit props.conf and add the following: [source::/var/log/messages] TRANSFORMS-set= setnull,setparsing Edit transforms.conf and add the following: [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = \[sshd\] DEST_KEY = queue FORMAT = parsingQueue   ... View more