Different crashes during tcpout reload. Received fatal signal 6 (Aborted) on PID .
Cause:
Signal sent by PID running under UID .
Crashing thread: indexerPipe_1
Backtrace (PIC build):
[0x00...
See more...
Different crashes during tcpout reload. Received fatal signal 6 (Aborted) on PID .
Cause:
Signal sent by PID running under UID .
Crashing thread: indexerPipe_1
Backtrace (PIC build):
[0x000014BC540AFB8F] gsignal + 271 (libc.so.6 + 0x4EB8F)
[0x000014BC54082EA5] abort + 295 (libc.so.6 + 0x21EA5)
[0x000055BCEBEFC1A7] __assert_fail + 135 (splunkd + 0x51601A7)
[0x000055BCEBEC4BD9] ? (splunkd + 0x5128BD9)
[0x000055BCE9013E72] _ZN34AutoLoadBalancedConnectionStrategyD0Ev + 18 (splunkd + 0x2277E72)
[0x000055BCE905DC99] _ZN14TcpOutputGroupD1Ev + 217 (splunkd + 0x22C1C99)
[0x000055BCE905E002] _ZN14TcpOutputGroupD0Ev + 18 (splunkd + 0x22C2002)
[0x000055BCE905FC6F] _ZN15TcpOutputGroups14checkSendStateEv + 623 (splunkd + 0x22C3C6F)
[0x000055BCE9060F08] _ZN15TcpOutputGroups4sendER15CowPipelineData + 88 (splunkd + 0x22C4F08)
[0x000055BCE90002FA] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 362 (splunkd + 0x22642FA)
[0x000055BCE9829628] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 72 (splunkd + 0x2A8D628)
[0x000055BCE8D29D25] _ZN8Pipeline4mainEv + 1157 (splunkd + 0x1F8DD25)
[0x000055BCEBF715EE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51D55EE)
[0x000055BCEBF716FB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51D56FB)
[0x000014BC552AC1DA] ? (libpthread.so.0 + 0x81DA) Another reload crash Backtrace (PIC build):
[0x00007F456828700B] gsignal + 203 (libc.so.6 + 0x2100B)
[0x00007F4568266859] abort + 299 (libc.so.6 + 0x859)
[0x0000560602B5B4B7] __assert_fail + 135 (splunkd + 0x5AAA4B7)
[0x00005605FF66297A] _ZN15TcpOutputClientD1Ev + 3130 (splunkd + 0x25B197A)
[0x00005605FF6629F2] _ZN15TcpOutputClientD0Ev + 18 (splunkd + 0x25B19F2)
[0x0000560602AD7807] _ZN9EventLoop3runEv + 839 (splunkd + 0x5A26807)
[0x00005605FF3555AD] _ZN11Distributed11EloopRunner4mainEv + 205 (splunkd + 0x22A45AD)
[0x0000560602BD03FE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x5B1F3FE)
[0x0000560602BD050B] _ZN6Thread8callMainEPv + 139 (splunkd + 0x5B1F50B)
[0x00007F4568CAD609] ? (libpthread.so.0 + 0x2609)
[0x00007F4568363353] clone + 67 (libc.so.6 + 0xFD353)
Linux / myhost / 5.15.0-1055-aws / #60~20.04.1-Ubuntu SMP Thu
assertion_failure="!_hasDataInTransit" assertion_function="virtual TcpOutputClient::~TcpOutputClient()" Starting Splunk 9.2, splunk outputs.conf is reloadable. Whenever DC pulls bundle from DS, depending on the changes, during reload, conf files are reloaded. One of the conf file is outputs.conf. Prior to 9.2 outputs.conf was not reloadable that means hitting following endpoint would do nothing. /data/outputs/tcp/server or https://<host>:<port>/servicesNS/-/-/admin/tcpout-group/_reload Behavior is changed from 9.2 and now outputs.conf is reloadable. However reloading outputs.conf is very complex process as it involves shutdown tcpout groups safely. Still there are cases where splunk crashes. We are working on fixing reported crashes. NOTE: (Splunkcloud and others), following workaround is NOT for a crash caused by /debug/refresh induced forced reload. There is no workaround available for a crash caused by /debug/refresh, except not to use /debug/refresh. Workaround As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications Set in local/apps.conf as a workaround. [triggers] reload.outputs = simple With setting above, splunk will take no action on tcpout(outputs.conf) reload( a behavior that was before 9.2) If outputs.conf is changed via DS, restart splunk.