Splunk Enterprise

Why Splunk indexers crash with third party S2S client and useACK=true?

hrawat_splunk
Splunk Employee
Splunk Employee

Crash log

Crashing thread: FwdDataReceiverThread
 Registers:
    RIP:  [0x00007F412B89E70F] gsignal + 271 (libc.so.6 + 0x3770F)
    RDI:  [0x0000000000000002]
    RSI:  [0x00007F41097FE060]
    RBP:  [0x00007F412B9EEC28]
    RSP:  [0x00007F41097FE060]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x00007F412B89E70F]
    RDX:  [0x0000000000000000]
    R8:  [0x0000000000000000]
    R9:  [0x00007F41097FE060]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000246]
    R12:  [0x000055B181DD32C8]
    R13:  [0x000055B181D2B95A]
    R14:  [0x0000000000000C9A]
    R15:  [0x000000000000080B]
    EFL:  [0x0000000000000246]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x002B000000000033]
    OLDMASK:  [0x0000000000000000]

 OS: Linux
 Arch: x86-64

Backtrace (PIC build):
  [0x00007F412B89E70F] gsignal + 271 (libc.so.6 + 0x3770F)
  [0x00007F412B888B25] abort + 295 (libc.so.6 + 0x21B25)
  [0x00007F412B8889F9] ? (libc.so.6 + 0x219F9)
  [0x00007F412B896CC6] ? (libc.so.6 + 0x2FCC6)
  [0x000055B17FCC89D7] CookedTcpChannel::kickOutput() + 791 (splunkd + 0x19B09D7)
  [0x000055B17FCCC608] CookedTcpChannel::sendACK_unlocked(bool) + 168 (splunkd + 0x19B4608)
  [0x000055B17FCD6E2D] CookedTcpChannel::addUncommitedEventId(unsigned long) + 109 (splunkd + 0x19BEE2D)
  [0x000055B17FCD6F2E] CookedTcpChannel::s2sDataAvailable(CowPipelineData&, S2SPerEventInfo const&, unsigned long) + 190 (splunkd + 0x19BEF2E)
  [0x000055B17FCD7020] FwdDataChannel::s2sDataAvailable(CowPipelineData&, S2SPerEventInfo const&, unsigned long) + 96 (splunkd + 0x19BF020)
  [0x000055B18072E3CD] S2SReceiver::gotOlds2sEvent(CowPipelineData&, S2SPerEventInfo const&) + 381 (splunkd + 0x24163CD)
  [0x000055B1805196AE] StreamingS2SParser::parse(char const*, char const*) + 11710 (splunkd + 0x22016AE)
  [0x000055B17FCC8B24] CookedTcpChannel::consume(TcpAsyncDataBuffer&) + 244 (splunkd + 0x19B0B24)
  [0x000055B17FCCB08D] CookedTcpChannel::dataAvailable(TcpAsyncDataBuffer&) + 45 (splunkd + 0x19B308D)
  [0x000055B1809D7973] TcpChannel::when_events(PollableDescriptor) + 531 (splunkd + 0x26BF973)
  [0x000055B18092355C] PolledFd::do_event() + 124 (splunkd + 0x260B55C)
  [0x000055B1809244D0] EventLoop::run() + 624 (splunkd + 0x260C4D0)
  [0x000055B1809D269C] Base_TcpChannelLoop::_do_run() + 28 (splunkd + 0x26BA69C)
  [0x000055B1809D279E] SubordinateTcpChannelLoop::run() + 222 (splunkd + 0x26BA79E)
  [0x000055B1809DF4D7] Thread::callMain(void*) + 135 (splunkd + 0x26C74D7)
  [0x00007F412BC312DE] ? (libpthread.so.0 + 0x82DE)
  [0x00007F412B962E83] clone + 67 (libc.so.6 + 0xFBE83)
Labels (1)
Tags (1)
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

It's possible third party S2S client has enabled/proxied acknowledge (useACK=true), however most of the 3rd party clients are unable to handle acknowledge received from indexers/receivers. After sometime indexer/receiver aborts(assertion failure) after detecting that S2S client is unable to process ACKs.

Workaround:
Turn off useACK on third party S2S client side.
Turn off useACK on UF if it routes via 3rd party S2S client.

Note: use Splunk INGEST ACTIONS instead of 3rd party S2S client. 

 

View solution in original post

Tags (1)

hrawat_splunk
Splunk Employee
Splunk Employee

It's possible third party S2S client has enabled/proxied acknowledge (useACK=true), however most of the 3rd party clients are unable to handle acknowledge received from indexers/receivers. After sometime indexer/receiver aborts(assertion failure) after detecting that S2S client is unable to process ACKs.

Workaround:
Turn off useACK on third party S2S client side.
Turn off useACK on UF if it routes via 3rd party S2S client.

Note: use Splunk INGEST ACTIONS instead of 3rd party S2S client. 

 

Tags (1)
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...