Knowledge Management

Splunk crash during tcpout (outputs.conf) reload

hrawat
Splunk Employee
Splunk Employee

Different crashes during tcpout reload.

Received fatal signal 6 (Aborted) on PID .
 Cause:
   Signal sent by PID  running under UID .
 Crashing thread: indexerPipe_1

 Backtrace (PIC build):
  [0x000014BC540AFB8F] gsignal + 271 (libc.so.6 + 0x4EB8F)
  [0x000014BC54082EA5] abort + 295 (libc.so.6 + 0x21EA5)
  [0x000055BCEBEFC1A7] __assert_fail + 135 (splunkd + 0x51601A7)
  [0x000055BCEBEC4BD9] ? (splunkd + 0x5128BD9)
  [0x000055BCE9013E72] _ZN34AutoLoadBalancedConnectionStrategyD0Ev + 18 (splunkd + 0x2277E72)
  [0x000055BCE905DC99] _ZN14TcpOutputGroupD1Ev + 217 (splunkd + 0x22C1C99)
  [0x000055BCE905E002] _ZN14TcpOutputGroupD0Ev + 18 (splunkd + 0x22C2002)
  [0x000055BCE905FC6F] _ZN15TcpOutputGroups14checkSendStateEv + 623 (splunkd + 0x22C3C6F)
  [0x000055BCE9060F08] _ZN15TcpOutputGroups4sendER15CowPipelineData + 88 (splunkd + 0x22C4F08)
  [0x000055BCE90002FA] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 362 (splunkd + 0x22642FA)
  [0x000055BCE9829628] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 72 (splunkd + 0x2A8D628)
  [0x000055BCE8D29D25] _ZN8Pipeline4mainEv + 1157 (splunkd + 0x1F8DD25)
  [0x000055BCEBF715EE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51D55EE)
  [0x000055BCEBF716FB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51D56FB)
  [0x000014BC552AC1DA] ? (libpthread.so.0 + 0x81DA)

Another reload crash

 Backtrace (PIC build):
  [0x00007F456828700B] gsignal + 203 (libc.so.6 + 0x2100B)
  [0x00007F4568266859] abort + 299 (libc.so.6 + 0x859)
  [0x0000560602B5B4B7] __assert_fail + 135 (splunkd + 0x5AAA4B7)
  [0x00005605FF66297A] _ZN15TcpOutputClientD1Ev + 3130 (splunkd + 0x25B197A)
  [0x00005605FF6629F2] _ZN15TcpOutputClientD0Ev + 18 (splunkd + 0x25B19F2)
  [0x0000560602AD7807] _ZN9EventLoop3runEv + 839 (splunkd + 0x5A26807)
  [0x00005605FF3555AD] _ZN11Distributed11EloopRunner4mainEv + 205 (splunkd + 0x22A45AD)
  [0x0000560602BD03FE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x5B1F3FE)
  [0x0000560602BD050B] _ZN6Thread8callMainEPv + 139 (splunkd + 0x5B1F50B)
  [0x00007F4568CAD609] ? (libpthread.so.0 + 0x2609)
  [0x00007F4568363353] clone + 67 (libc.so.6 + 0xFD353)
 Linux / myhost / 5.15.0-1055-aws / #60~20.04.1-Ubuntu SMP Thu 
assertion_failure="!_hasDataInTransit" assertion_function="virtual TcpOutputClient::~TcpOutputClient()" 

 

Starting Splunk 9.2, splunk outputs.conf is reloadable. Whenever DC pulls bundle from DS, depending on the changes, during reload, conf files are reloaded. One of the conf file is outputs.conf.
Prior to 9.2 outputs.conf was not reloadable that means hitting following endpoint would do nothing.

/data/outputs/tcp/server or 

https://<host>:<port>/servicesNS/-/-/admin/tcpout-group/_reload

Behavior is changed from 9.2 and now outputs.conf is reloadable. However reloading outputs.conf is very complex process as it involves shutdown tcpout groups safely. Still there are cases where splunk crashes. We are working on fixing reported crashes.

NOTE: (Splunkcloud and others), following workaround is NOT for a crash caused by  /debug/refresh induced forced reload. 
There is no workaround available for a crash caused by /debug/refresh, except not to use /debug/refresh.


Workaround

As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications

Set in local/apps.conf as a workaround.

[triggers]
reload.outputs = simple

With setting above, splunk will take no action on tcpout(outputs.conf) reload( a behavior  that was before 9.2)

 

If outputs.conf is changed via DS, restart splunk.

Labels (1)

jstratton
Explorer

@hrawat wrote:


As mentioned before 9.2 outputs.conf was never reloadable ( no-op for _reload), thus no crashes/complications

We've used /services/data/outputs/tcp/server/_reload to successfully reload updated `clientCert` certificates and `server` hosts for years when using Splunk Enterprise 8.x and 9.0.x instances.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @hrawat ,

open soon a case to Splunk Support.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...