Different crashes during tcpout reload.
Received fatal signal 6 (Aborted) on PID .
Cause:
Signal sent by PID running under UID .
Crashing thread: indexerPipe_1
Backtrace (PIC build):
[0x000014BC540AFB8F] gsignal + 271 (libc.so.6 + 0x4EB8F)
[0x000014BC54082EA5] abort + 295 (libc.so.6 + 0x21EA5)
[0x000055BCEBEFC1A7] __assert_fail + 135 (splunkd + 0x51601A7)
[0x000055BCEBEC4BD9] ? (splunkd + 0x5128BD9)
[0x000055BCE9013E72] _ZN34AutoLoadBalancedConnectionStrategyD0Ev + 18 (splunkd + 0x2277E72)
[0x000055BCE905DC99] _ZN14TcpOutputGroupD1Ev + 217 (splunkd + 0x22C1C99)
[0x000055BCE905E002] _ZN14TcpOutputGroupD0Ev + 18 (splunkd + 0x22C2002)
[0x000055BCE905FC6F] _ZN15TcpOutputGroups14checkSendStateEv + 623 (splunkd + 0x22C3C6F)
[0x000055BCE9060F08] _ZN15TcpOutputGroups4sendER15CowPipelineData + 88 (splunkd + 0x22C4F08)
[0x000055BCE90002FA] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 362 (splunkd + 0x22642FA)
[0x000055BCE9829628] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 72 (splunkd + 0x2A8D628)
[0x000055BCE8D29D25] _ZN8Pipeline4mainEv + 1157 (splunkd + 0x1F8DD25)
[0x000055BCEBF715EE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51D55EE)
[0x000055BCEBF716FB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51D56FB)
[0x000014BC552AC1DA] ? (libpthread.so.0 + 0x81DA)
Another reload crash
Backtrace (PIC build):
[0x00007F456828700B] gsignal + 203 (libc.so.6 + 0x2100B)
[0x00007F4568266859] abort + 299 (libc.so.6 + 0x859)
[0x0000560602B5B4B7] __assert_fail + 135 (splunkd + 0x5AAA4B7)
[0x00005605FF66297A] _ZN15TcpOutputClientD1Ev + 3130 (splunkd + 0x25B197A)
[0x00005605FF6629F2] _ZN15TcpOutputClientD0Ev + 18 (splunkd + 0x25B19F2)
[0x0000560602AD7807] _ZN9EventLoop3runEv + 839 (splunkd + 0x5A26807)
[0x00005605FF3555AD] _ZN11Distributed11EloopRunner4mainEv + 205 (splunkd + 0x22A45AD)
[0x0000560602BD03FE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x5B1F3FE)
[0x0000560602BD050B] _ZN6Thread8callMainEPv + 139 (splunkd + 0x5B1F50B)
[0x00007F4568CAD609] ? (libpthread.so.0 + 0x2609)
[0x00007F4568363353] clone + 67 (libc.so.6 + 0xFD353)
Linux / myhost / 5.15.0-1055-aws / #60~20.04.1-Ubuntu SMP Thu
assertion_failure="!_hasDataInTransit" assertion_function="virtual TcpOutputClient::~TcpOutputClient()"
Starting Splunk 9.2, splunk outputs.conf is reloadable. Whenever DC pulls bundle from DS, depending on the changes, during reload, conf files are reloaded. One of the conf file is outputs.conf.
Prior to 9.2 outputs.conf was not reloadable that means hitting following endpoint would do nothing.
/data/outputs/tcp/server or
https://<host>:<port>/servicesNS/-/-/admin/tcpout-group/_reload
Behavior is changed from 9.2 and now outputs.conf is reloadable. However reloading outputs.conf is very complex process as it involves shutdown tcpout groups safely. Still there are cases where splunk crashes. We are working on fixing reported crashes.
NOTE: (Splunkcloud and others), following workaround is NOT for a crash caused by /debug/refresh induced forced reload.
There is no workaround available for a crash caused by /debug/refresh, except not to use /debug/refresh.
Workaround
As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications
Set in local/apps.conf as a workaround.
If outputs.conf is changed via DS, restart splunk.
@hrawat wrote:
As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications
We've used /services/data/outputs/tcp/server/_reload to successfully reload updated `clientCert` certificates and `server` hosts for years when using Splunk Enterprise 8.x and 9.0.x instances.