Getting Data In

Why isn't my LINE_BREAKER configuration breaking my multiline event as expected?

avilandau
Path Finder

I have a simple multiline log (which I can control and change if needed). As recommended I could use LINE_BREAKER so I have #### at the beginning of each line (whether it is multi or single line event as shown below.

My props.conf is:

[gpp_error_log]
FIELD_DELIMITER=,
FIELD_QUOTE="
FIELD_NAMES=timestamp,profile,severity,code,message
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)####
LINE_BREAKER_LOOKBEHIND=32000

I've tried several combination of the LINE_BREAKER playing with the capture group, but nothing works. The single line are broken correctly but the multi-line are not. I assume the #### is discarded which is what I want.

Here is the log:

####2015-01-26 22:37:57.963,POC1,7,30837,Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121
####2015-01-26 22:38:05.075,POC1,6,20657,Simulated GPP error from POC1 of severity 6 from code 20657
####2015-01-26 22:38:11.150,POC1,2,35194,Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121
####2015-01-26 22:38:29.275,POC1,8,39442,Simulated GPP error from POC1 of severity 8 from code 39442
####2015-01-26 22:38:40.331,POC1,4,14836,Simulated GPP error from POC1 of severity 4 from code 14836
####2015-01-26 22:38:54.443,POC1,9,39012,Simulated GPP error from POC1 of severity 9 from code 39012
0 Karma

emiller42
Motivator

Another option is to use an EXTRACT instead of DELIMS+FIELDS.

Here is how I would approach the props.conf entry:

[gpp_error_log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)(?=####)
MAX_TIMESTAMP_LOOKAHEAD = 23
TIME_PREFIX=^####
TIME_FORMAT=%F %T.%3N
EXTRACT-gpp_fields=^####(?<timestamp>[^,]+),(?<profile>[^,]+),(?<severity>[^,]+),(?<code>[^,]+),(?<message>.+)$

Produces the following:

alt text

0 Karma

_d_
Splunk Employee
Splunk Employee

Chances are INDEXED_EXTRACTIONS are messing up multi-line events. Try this instead and then use regular DELIMS based extractions:

props.conf

[gpp_error_log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+####)
REPORT-fields = error_fields

transforms.conf

[error_fields]
DELIMS = ","
FIELDS = timestamp,profile,severity,code,message

Note that the LINE_BREAKER above will toss out the # signs

0 Karma

_d_
Splunk Employee
Splunk Employee

That basically means that the field that you searching with is not being extracted correctly. Given that you have a combination of single and multiline events you can try extracting the line with fields first and then apply DELIMS on it. This way you're guaranteed that the extractor will work on clean lines:

props.conf

[gpp_error_log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+####)
REPORT-fields = error_fields
EXTRACT-first = (?<first_line>.*?)$|\n

transforms.conf

[error_fields]
SOURCE_KEY = first_line
DELIMS = ","
FIELDS = timestamp,profile,severity,code,message
0 Karma

avilandau
Path Finder

It didn't work. I still get only the single line when I added field=value. But the interesting thing is (also before I made this last change) that when I search just by sourcetype and get all the lines, if I add the field to the list of displayed field I do see the right value of field. This means it is a recognized field so why the search doesn't work on it?

0 Karma

avilandau
Path Finder

To illustrate my point: I do a search with only sourcetype so I get all lines (those that are mutli-line show only the first line in the grid) and all of the lines show the right fields. I click for drilldown (default to search for now) and if I click on the first line of a mutli line the search comes back with no result found and if I click on a single line drilldown search finds it.

0 Karma

_d_
Splunk Employee
Splunk Employee

Not sure what search you're using or what other configs you have in place, but with the configs above this search emits the correct multiline event on my end:
Search:
index=my_index source=my source code=35194

Result:
2015-01-26 22:38:11.150,POC1,2,35194,Exception Report: null java.lang.NullPointerException at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48) at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411) at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88) at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130) at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199) at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188) at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613) .... more stack lines will be here...... File:sun.reflect.NativeMethodAccessorImpl, Line:48, User ID:, IP Address:192.168.168.121

0 Karma

avilandau
Path Finder

When I'm doing
index=my_index sourcetype=gpp_error_log
I get the expected result of all lines where the profile, code, and severity are displayed correctly (in table format) for all lines

If I do
index=my_index sourcetype=gpp_error_log profile=POC4 (or even POC4*)
I get only the single lines

If I do index=my_index sourcetype=gpp_error_log code=nnnn (where nnnn is from a multi line)
I get no result found

0 Karma

avilandau
Path Finder

Hold on.... I just tried profile=POC4 and I got all lines!!! I'm not sure how to explain it.

0 Karma

avilandau
Path Finder

I meant to write {asterisk}POC4 and it worked. The field message in a table shows the first line but that is actually better because in the table grid it is good to see all entries and for those that are multi-line the first line only thus the grid looks unified. I'll do another drilldown to get the _raw of a single multiline entry.

0 Karma

avilandau
Path Finder

Thanks. The change to transforms makes it work but still one little problem. The field message is missing the last line for some reason. Any idea what needs to be changed for that? Maybe I should end it with a coma or maybe I should have a more distinct DELIMS character.

0 Karma

avilandau
Path Finder

Adding a delimiter at the end of line did the trick and now I see the full multi-line but I do have a problem with a search. If I use any of the extracted field in the search only the single line are returned. I'll probably open a new question for that because I think the title above is not a good description of the problem as it stands now.

0 Karma

avilandau
Path Finder

I assume what you are saying is that the fact I have field extraction and multi-line log event is the issue. is that right?

I'll try later both tests later: (1) without my fields extractions (2) with the fields and the transform you advised.

0 Karma

ppablo
Retired

Hi @avilandau

I just edited your post's format so the backslash in your regex shows properly.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...