Why is my configuration causing a search for field=value to return only single line events, not multiline?

[avilandau]

I have a log, broken to fields, where the free text field is the last field and can be multiline. After defining the fields in transforms.conf (i.e. rather than in props.conf) and making sure that every field, including the multiline text, is wrapped with the delimiter, it now works perfectly i.e. If I search by sourcetype I see all logs entries, each one with its fields. However, if I add to the search field=value (where field is one of the fields in the event) I get back only the single line events and none of the multiline ones.

Any idea what causes that?

Props.conf

[gpp_error_log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+####)
LINE_BREAKER_LOOKBEHIND=32000
REPORT-fields = gpp_error_log_fields

transforms.conf

[gpp_error_log_fields]
DELIMS = "~"
FIELDS = timestamp,profile,severity,code,message

The log

####2015-01-27 11:16:23.370~POC4~9~29245~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:16:46.516~POC4~4~22557~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:17:11.723~POC4~4~33772~Simulated GPP error from POC4 of severity 4 from code 33772~
####2015-01-27 11:17:46.877~POC4~7~26846~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:17:57.979~POC4~5~36591~Simulated GPP error from POC4 of severity 5 from code 36591~
####2015-01-27 11:18:20.050~POC4~6~29483~Simulated GPP error from POC4 of severity 6 from code 29483~

[bmacias84]

@avilandau,

I would recommend using the data previewer in Splunk to make sure you data is being broken correctly. You really should be using SHOULD_LINEMERGE=true=, which combines several lines of data into a single multiline event. The MAX_TIMESTAMP_LOOKAHEAD will also speed up indexing by telling Splunk to only lookup to 32 char instead of scanning the event. The other stuff can be ignored. Make sure these settings are applied to your indexer. The settings below have been tested with your sample data.

[gpp_error_log]
 SHOULD_LINEMERGE=true
 MAX_TIMESTAMP_LOOKAHEAD=32
 REPORT-fields = gpp_error_log_fields

Cheers,