I upgraded to 6.1 and now Splunk is crashing while reading my disk_objects.log file. Why?

[DerekB]

It's crashing on the maintailingthread while reading the disk_objects.log. This is on Windows and the crash looks like this:

Crashing thread: MainTailingThread for WatchedTailFile-WatchedFileState: path="C:\Program Files\Splunk\var\log\introspection\disk_objects.log

[build 206881] 2014-05-09 04:01:09
 Access violation, cannot read at address [0x000079646F626F00]
 Exception address: [0x0000000140614B21]
 Crashing thread: MainTailingThread
    MxCsr:  [0x0000000000001FA0]
    SegDs:  [0x000000000000002B]
    SegEs:  [0x000000000000002B]
    SegFs:  [0x0000000000000053]
    SegGs:  [0x000000000000002B]
    SegSs:  [0x000000000000002B]
    SegCs:  [0x0000000000000033]
    EFlags:  [0x0000000000010206]
    Rsp:  [0x000000001192D1D0]
    Rip:  [0x0000000140614B21] ?
    Dr0:  [0x0000000000000000]
    Dr1:  [0x0000000000000000]
    Dr2:  [0x0000000000000000]
    Dr3:  [0x0000000000000000]
    Dr6:  [0x0000000000000000]
    Dr7:  [0x0000000000000000]
    Rax:  [0x000079646F626F00]
    Rcx:  [0x0000000022C11268]
    Rdx:  [0x000000001192E368]
    Rbx:  [0x000000001192E2C0]
    Rbp:  [0x0000000000000000]
    Rsi:  [0x000000001192E368]
    Rdi:  [0x0000000000000000]
    R8:  [0x000000001192E2C0]
    R9:  [0x0000000000000000]
    R10:  [0x000000004E584490]
    R11:  [0x000000004E584990]
    R12:  [0x000000001192E420]
    R13:  [0x0000000000000500]
    R14:  [0x0000000022C11268]
    R15:  [0x0000000000000000]
    DebugControl:  [0x0000000000000000]
    LastBranchToRip:  [0x0000000000000000]
    LastBranchFromRip:  [0x0000000000000000]
    LastExceptionToRip:  [0x0000000000000000]
    LastExceptionFromRip:  [0x0000000000000000]

 OS: Windows
 Arch: x86-64

[hexx · Accepted solution]

This issue has been identified as a bug affecting Splunk Enterprise 6.1 and 6.1.1 on Windows platforms only - the reference is SPL-83975.

It is triggered by the use of JSON-based indexed field extractions during the acquisition of two native files:

• %SPLUNK_HOME%\var\log\introspection\resource_usage.log
• %SPLUNK_HOME%\var\log\introspection\disk_objects.log

We hope to fix this issue in an upcoming maintenance release very soon, but in the meantime you can work-around it by adding the following configuration to %SPLUNK_HOME%\etc\system\local\props.conf:

[splunk_disk_objects]
INDEXED_EXTRACTIONS = 

[splunk_resource_usage]
INDEXED_EXTRACTIONS = 

This will suppress the INDEXED_EXTRACTIONS = json directive that is applied by default to these sourcetypes in %SPLUNK_HOME%\etc\system\default\props.conf.

[Ellen]

SPL-83975 was reported under Windows but also seen in some instances on Linux. This bug has been fixed as of 6.1.4 as referenced here