Why Splunk indexers crash with third party S2S client and useACK=true?

[hrawat_splunk]

Crash log

Crashing thread: FwdDataReceiverThread
 Registers:
    RIP:  [0x00007F412B89E70F] gsignal + 271 (libc.so.6 + 0x3770F)
    RDI:  [0x0000000000000002]
    RSI:  [0x00007F41097FE060]
    RBP:  [0x00007F412B9EEC28]
    RSP:  [0x00007F41097FE060]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x00007F412B89E70F]
    RDX:  [0x0000000000000000]
    R8:  [0x0000000000000000]
    R9:  [0x00007F41097FE060]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000246]
    R12:  [0x000055B181DD32C8]
    R13:  [0x000055B181D2B95A]
    R14:  [0x0000000000000C9A]
    R15:  [0x000000000000080B]
    EFL:  [0x0000000000000246]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x002B000000000033]
    OLDMASK:  [0x0000000000000000]

 OS: Linux
 Arch: x86-64

Backtrace (PIC build):
  [0x00007F412B89E70F] gsignal + 271 (libc.so.6 + 0x3770F)
  [0x00007F412B888B25] abort + 295 (libc.so.6 + 0x21B25)
  [0x00007F412B8889F9] ? (libc.so.6 + 0x219F9)
  [0x00007F412B896CC6] ? (libc.so.6 + 0x2FCC6)
  [0x000055B17FCC89D7] CookedTcpChannel::kickOutput() + 791 (splunkd + 0x19B09D7)
  [0x000055B17FCCC608] CookedTcpChannel::sendACK_unlocked(bool) + 168 (splunkd + 0x19B4608)
  [0x000055B17FCD6E2D] CookedTcpChannel::addUncommitedEventId(unsigned long) + 109 (splunkd + 0x19BEE2D)
  [0x000055B17FCD6F2E] CookedTcpChannel::s2sDataAvailable(CowPipelineData&, S2SPerEventInfo const&, unsigned long) + 190 (splunkd + 0x19BEF2E)
  [0x000055B17FCD7020] FwdDataChannel::s2sDataAvailable(CowPipelineData&, S2SPerEventInfo const&, unsigned long) + 96 (splunkd + 0x19BF020)
  [0x000055B18072E3CD] S2SReceiver::gotOlds2sEvent(CowPipelineData&, S2SPerEventInfo const&) + 381 (splunkd + 0x24163CD)
  [0x000055B1805196AE] StreamingS2SParser::parse(char const*, char const*) + 11710 (splunkd + 0x22016AE)
  [0x000055B17FCC8B24] CookedTcpChannel::consume(TcpAsyncDataBuffer&) + 244 (splunkd + 0x19B0B24)
  [0x000055B17FCCB08D] CookedTcpChannel::dataAvailable(TcpAsyncDataBuffer&) + 45 (splunkd + 0x19B308D)
  [0x000055B1809D7973] TcpChannel::when_events(PollableDescriptor) + 531 (splunkd + 0x26BF973)
  [0x000055B18092355C] PolledFd::do_event() + 124 (splunkd + 0x260B55C)
  [0x000055B1809244D0] EventLoop::run() + 624 (splunkd + 0x260C4D0)
  [0x000055B1809D269C] Base_TcpChannelLoop::_do_run() + 28 (splunkd + 0x26BA69C)
  [0x000055B1809D279E] SubordinateTcpChannelLoop::run() + 222 (splunkd + 0x26BA79E)
  [0x000055B1809DF4D7] Thread::callMain(void*) + 135 (splunkd + 0x26C74D7)
  [0x00007F412BC312DE] ? (libpthread.so.0 + 0x82DE)
  [0x00007F412B962E83] clone + 67 (libc.so.6 + 0xFBE83)

[hrawat_splunk · Accepted solution]

It's possible third party S2S client has enabled/proxied acknowledge (useACK=true), however most of the 3rd party clients are unable to handle acknowledge received from indexers/receivers. After sometime indexer/receiver aborts(assertion failure) after detecting that S2S client is unable to process ACKs.

Workaround:
Turn off useACK on third party S2S client side.
Turn off useACK on UF if it routes via 3rd party S2S client.

Note: use Splunk INGEST ACTIONS instead of 3rd party S2S client.