topic Re: How to parse events before indexing? in Splunk Search

How to parse events before indexing?

Thulasinathan_M — Mon, 07 Aug 2023 19:30:20 GMT

Hi Splunk Experts,
I want to break all lines as a single Line event [\r\n]. But if there are logs with stacktrace I want to consider them as multi-line event.
I've tested below regex and it works as expected, but I'm not sure on, what are the properties I should apply them for a sourcetype. This is for an application which logs millions of event in a minute. Please assist me with an optimized solution.

(.*[\n]((.*\)\])?(\s+at.*\)\n))+)

Sample logs:

[(2023-08-03 10:00:03)] INFO: Request completed successfully. [(2023-08-03 10:00:03)] ERROR: Request got failed. [(2023-08-03 10:00:02)] Exception in thread "main" java.lang.NullPointerException at com.example.MyClass.method1(MyClass.java:12) at com.example.MyClass.method2(MyClass.java:34) at com.example.AnotherClass.someMethod(AnotherClass.java:56) at com.example.Main.main(Main.java:23) [(2023-08-03 10:00:03)] INFO: Request Submitted successfully. [(2023-08-03 10:00:03)] INFO: Request completed successfully. [(2023-08-03 10:00:03)] WARN: Request failed unsuccessfully. [(2023-08-03 10:00:02)] java.io.FileNotFoundException: file.txt (No such file or directory) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.open0(Native Method) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.open(FileInputStream.java:219) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) [(2023-08-03 10:00:02)] at com.example.FileDemo.readFromFile(FileDemo.java:55) [(2023-08-03 10:00:02)] at com.example.Main.main(Main.java:12) [(2023-08-03 10:00:03)] INFO: Request completed successfully. [(2023-08-03 10:00:04)] DEBUG: Processing request: /api/v1/data?id=67890 [(2023-08-03 10:00:03)] WARN: Request failed unsuccessfully. java.lang.IllegalArgumentException: Invalid input: negative value not allowed [(2023-08-03 10:00:02)] at com.example.MathUtils.squareRoot(MathUtils.java:42) [(2023-08-03 10:00:02)] at com.example.Main.main(Main.java:33) [(2023-08-03 10:00:02)] ERROR: Failed to fetch data from the database.

Expected First Multi-Line Event:

[(2023-08-03 10:00:02)] Exception in thread "main" java.lang.NullPointerException at com.example.MyClass.method1(MyClass.java:12) at com.example.MyClass.method2(MyClass.java:34) at com.example.AnotherClass.someMethod(AnotherClass.java:56) at com.example.Main.main(Main.java:23)

Expected Second Multi-Line Event:

[(2023-08-03 10:00:02)] java.io.FileNotFoundException: file.txt (No such file or directory) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.open0(Native Method) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.open(FileInputStream.java:219) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) [(2023-08-03 10:00:02)] at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) [(2023-08-03 10:00:02)] at com.example.FileDemo.readFromFile(FileDemo.java:55) [(2023-08-03 10:00:02)] at com.example.Main.main(Main.java:12)

Expected Third Multi-Line Event:

[(2023-08-03 10:00:03)] WARN: Request failed unsuccessfully. java.lang.IllegalArgumentException: Invalid input: negative value not allowed [(2023-08-03 10:00:02)] at com.example.MathUtils.squareRoot(MathUtils.java:42) [(2023-08-03 10:00:02)] at com.example.Main.main(Main.java:33)

Re: How to parse events before indexing?

ITWhisperer — Mon, 07 Aug 2023 21:57:11 GMT

Essentially, you want your events to break at a newline followed by a timestamp?

What do you currently have configured?

Re: How to parse events before indexing?

Thulasinathan_M — Tue, 08 Aug 2023 02:18:08 GMT

Hi @ITWhisperer,

Currently the configuration is below

CHARSET=UTF-8 SHOULD_LINEMERGE=true disabled=false LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true

Re: How to parse events before indexing?

inventsekar — Tue, 08 Aug 2023 03:50:24 GMT

you have a single file, but you want to do multiple line breaking in that single file... is that right?

Re: How to parse events before indexing?

Thulasinathan_M — Tue, 08 Aug 2023 04:03:25 GMT

Hi @inventsekar
Yes that's correct!! I want to break each & every lines as a single line event, but if there are any logs met above condition mentioned in the post, then I want to wrap those logs as a multi-line event and this should be applicable to all files under my sourcetype.

Re: How to parse events before indexing?

inventsekar — Tue, 08 Aug 2023 04:29:47 GMT

Pls check this https://community.splunk.com/t5/Getting-Data-In/Line-break-with-multiple-Linebreaker/m-p/400335

the values for BREAK_ONLY_BEFORE, MUST_NOT_BREAK_AFTER, MUST_BREAK_AFTER should be updated for your requirement properly..

rough one..

BREAK_ONLY_BEFORE=^Exception | ^java.io.FileNotFoundException | ^WARN MUST_NOT_BREAK_AFTER=something here MUST_BREAK_AFTER=something here

Re: How to parse events before indexing?

isoutamo — Tue, 08 Aug 2023 06:50:41 GMT

Based on your example you could try something like this. If/when needed you should add those keywords (into LINE_BREAKER) which are starting a new event here

[<your sourcetype>] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+)\[\(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d\)\]\s([\w\.]+:|Exception) NO_BINARY_CHECK=true TIME_FORMAT=%F %T TIME_PREFIX=^\[\( MAX_TIMESTAMP_LOOKAHEAD=20

Usually it's better if you could avoid those BREAK_ONLY_BEFORE, UST_NOT_BREAK_AFTER and MUST_BREAK_AFTER. Those are working, but those use more resources than using only LINE_BREAKER definition.

r. Ismo

Re: How to parse events before indexing?

Thulasinathan_M — Tue, 08 Aug 2023 06:55:48 GMT

Thanks for the pointers, I've come-up with below config, but still it's not working and it looks bit messy. What I've done is:
Line_Breaker: Broken down events based on the Timestamp/ Previous line of first matching line with 'at'/ Last matching line with 'at'
BREAK_ONLY_BEFORE: Timestamp/ Previous line of first matching line with 'at'
MUST_NOT_BREAK_AFTER: Previous line of first matching line with 'at'
MUST_BREAK_AFTER: Last matching line with 'at'
Any suggestion on what I've done wrong please.

SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)(?:(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\])|(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?!at).*)\n(?:\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?=at).*)|(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?=at).*)\n(?:\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?!at).*) NO_BINARY_CHECK=true MUST_NOT_BREAK_AFTER=^(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?!at).*)\n(?:\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?=at).*) BREAK_ONLY_BEFORE=^(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\])|^(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?!at).*)\n(?:\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?=at).*) MUST_BREAK_AFTER=^(\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?=at).*)\n(?:\[\(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\)\]\s+(?!at).*)

Re: How to parse events before indexing?

isoutamo — Tue, 08 Aug 2023 06:58:20 GMT

Have you try my example? Based on your example events, it works on my test env.

Re: How to parse events before indexing?

Thulasinathan_M — Tue, 08 Aug 2023 07:07:20 GMT

Thanks @isoutamo. It's Working as expected with sample inputs. But I can't rely on below pattern because it could be anything. Only thing I could make sure is 'at' followed by path to the file(Stacktraces).

([\w\.]+:|Exception)

Re: How to parse events before indexing?

isoutamo — Tue, 08 Aug 2023 07:38:23 GMT

Ok. I just look that your log events contains <Log level>: or <java class>: or that Exception key words which are defining a start of new individual event.

You could try to get correct regex to match this on Splunk Slack #regex channel https://splunk-usergroups.slack.com/archives/C3WFE5V5G