Getting Data In

Why does the Splunk Universal Forwarder 6.3.0 on Linux x86_64 server keep crashing?

kishen2016
Explorer

Splunk Universal Forwarder agent keeps crashing - Agent version 6.3.0 ...Server is Linux x86_64

crashlog updated:

[splunk@ftdcslsapp638 splunk]$ cat crash-2016-10-12-11:52:08.log
[build aa7d4b1ccb80] 2016-10-12 11:52:08
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 5484 running under UID 5043.
 Crashing thread: tailreader0
 Registers:
    RIP:  [0x0000003B25A325E5] gsignal + 53 (/lib64/libc.so.6)
    RDI:  [0x000000000000156C]
    RSI:  [0x000000000000159E]
    RBP:  [0x000000000187DC28]
    RSP:  [0x00007F72847FE868]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F726E3FF000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0xFEFEFEFEFEFEFEFF]
    R9:  [0x00007F7290F49F60]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000206]
    R12:  [0x0000000001813019]
    R13:  [0x000000000187EFE0]
    R14:  [0x00007F7288BD79B8]
    R15:  [0x00007F72847FEDE8]
    EFL:  [0x0000000000000206]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

OS: Linux
Arch: x86-64

 Backtrace:
  [0x0000003B25A325E5] gsignal + 53 (/lib64/libc.so.6)
  [0x0000003B25A33DC5] abort + 373 (/lib64/libc.so.6)
  [0x0000003B25A2B70E] ? (/lib64/libc.so.6)
  [0x0000003B25A2B7D0] __assert_perror_fail + 0 (/lib64/libc.so.6)
  [0x0000000000A503EA] ? (splunkd)
  [0x0000000000A4E6C3] _ZNK11TailWatcher12setupConfigsER15WatchedTailFile + 1507 (splunkd)
  [0x0000000000A4E7D2] _ZNK11TailWatcher19initializeFileStateER15WatchedTailFileRK8Pathname + 66 (splunkd)
  [0x0000000000A679F5] _ZN10TailReader10handleFileEP15WatchedTailFileP11TailWatcher + 69 (splunkd)
  [0x0000000000A6A2DA] _ZN12ReaderThread4mainEv + 378 (splunkd)
  [0x000000000109F0EE] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x0000003B25E07AA1] ? (/lib64/libpthread.so.0)
  [0x0000003B25AE8AAD] clone + 109 (/lib64/libc.so.6)
 Linux / ftdcslsapp638.ftiz.cummins.com / 2.6.32-642.1.1.el6.x86_64 / #1 SMP Fri May 6 14:54:05 EDT 2016 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2016-10-11 16:01:49.002 -0500 splunkd started (build aa7d4b1ccb80)
    splunkd: /home/build/build-src/ember/src/pipeline/input/Tailing.h:178: bool StatWrap::isDir() const: Assertion `_valid' failed.
    2016-10-12 10:18:26.775 -0500 splunkd started (build aa7d4b1ccb80)
    2016-10-12 10:20:58.906 -0500 Interrupt signal received
    2016-10-12 10:21:16.753 -0500 splunkd started (build aa7d4b1ccb80)
    2016-10-12 10:25:49.955 -0500 Interrupt signal received
    2016-10-12 10:26:03.211 -0500 splunkd started (build aa7d4b1ccb80)
    2016-10-12 10:37:14.366 -0500 Interrupt signal received
    2016-10-12 10:37:31.114 -0500 splunkd started (build aa7d4b1ccb80)
    splunkd: /home/build/build-src/ember/src/pipeline/input/Tailing.h:178: bool StatWrap::isDir() const: Assertion `_valid' failed.

 /etc/redhat-release: Red Hat Enterprise Linux Server release 6.8 (Santiago)
 glibc version: 2.12
 glibc release: stable
Last errno: 2
Threads running: 35
Runtime: 4477.407956s
argv: [splunkd -p 8089 restart]
Thread: "tailreader0", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f72888ab610:
00000000  00 f7 7f 84 72 7f 00 00                           |....r...|
00000008
ReaderThread: mode=0, queueSize=89704, shutdown=N, reconfigure=N, mode=0
Reading File-WatchedTailFile-WatchedFileState: path="/opt/splunkforwarder/var/run/nmon/var/csv_repository/ftdcslsapp638_12_OCT_2016_101833_CPUnn_51942_20161012103734.nmon.csv", flags=0x24213
First 144 bytes of PathnameStat @0x7f727ab95608:
00000000  00 00 00 00 00 00 00 00  80 55 b9 7a 72 7f 00 00  |.........U.zr...|
00000010  00 00 00 00 00 00 00 00  a0 ba 1f 85 72 7f 00 00  |............r...|
00000020  00 b3 8a 88 72 7f 00 00  40 a2 f1 80 72 7f 00 00  |....r...@...r...|
00000030  00 00 00 00 00 00 00 00  30 a0 97 01 00 00 00 00  |........0.......|
00000040  00 57 b9 7a 72 7f 00 00  28 b8 1f 85 72 7f 00 00  |.W.zr...(...r...|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  77 5b fe 57 00 00 00 00  0f aa 09 00 00 00 00 00  |w[.W............|
00000070  00 00 00 00 00 00 00 00  50 bd 96 01 00 00 00 00  |........P.......|
00000080  00 00 00 00 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000090
FilesystemChangeWatcher: _timeoutActive=N, _throttled=N, _waitingForNotifyCount=89705
  EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y
  Timeout: _when = 1476287351.633359, _initialMsec = 0
file-in: _initialized=Y, _lastCharWasNewline=N, _lastReadHadNulls=N, _wasCrcConflict=N, _warned=N
         _nullsWarned=N, _wasTooNew=N, _exists=N, _noDebug=N
         _hadExplicitSource=N, _crossedInitCrcLenBoundary=N, _classifiedAtLeastOnce=N, _fileReplaced=N, _readPathAfterRealEOF=N
         _onlyNotifiedOnce=Y, _isArchive=N, _isCached=111213, _unowned=N, _deleteOnEOF=N
         _overrideDeleteOnEOF=N, _doNotDeleteChildren=N, _readFromEnd=N, _readIrregardless=N
         _fileCheckMethod=0, _crcSalt=<null>, _origPath=<null>
         _bytesRead=0, _storingBytesRead=0, _initCrc=0x0, _seekCrc=0x0
         _filenameCrc=0xc260a7a429dae89a, _fallbackCrc=0x0, _lastEOFTime=<zero>, _modTime=<zero>
         _eofSeconds=3, _ignoreThresh=<zero>, _initCrcBytes=256, _initCrcForBatch=0x0
         _pendingMetadata=<null>
         _prevFd=-1, _pdModels=[0 PDs]
         _rescheduleDelay=1000, _rescheduleTarget=<zero>, _name=/opt/splunkforwarder/var/run/nmon/var/csv_repository/ftdcslsapp638_12_OCT_2016_101833_CPUnn_51942_20161012103734.nmon.csv, _statusName=
         _st=[dev=633625, ino=0, mode=0, size=0, mtime=0, owner=0, group=1]
         _toStringPrefix=state=0x0x7f727ab95580, _backoff=0
         _stdataInputHeaderProcessing=[]

         _detectTrailingNulls=N, _detectReadingFromOffSet=N, _readAndSkipHeader=N, _uniqueId=0
  _rawPath=

x86 CPUID registers:

         0: 0000000A 756E6547 6C65746E 49656E69
         1: 000006F1 06010800 80002201 0FABFBFF
         2: 76036301 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000000 00000000 00000000 00000000
         6: 00000077 00000002 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300401 0000007F 00000000 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 20100800
  80000002: 20202020 746E4920 52286C65 65582029
  80000003: 52286E6F 50432029 35452055 3536342D
  80000004: 76204C37 20402032 30342E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...

katanguriabhi
Explorer

Check for Warnings in Splunkd logs, there might be a problem with ulimits on your Linux server. If there is a warning for ulimit values, change them to recommend values or to unlimite.

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...