Splunk Search

REGEX LINE BREAKER - multiline

oylkm
Explorer

I've got F5-LTM logs being sent as syslog in CEF format, I would like to break up the logs per-line with regex and I'm not making headway. Eventually I want to look like this for every event line instead of a multi-line event. 

 

2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Response|http_method=GET|site=CommonSWF_8080|x.x.x.x|x.x.x.x|dest_port=8080|uri_path=/|url=GET / HTTP/1.1|status=200|duration=12|http_user_agent=cosmos-v1|bytes_out=181
2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Request|http_method=GET|site=/CommonCommon_8080|x.x.x.x|x.x.x.x|dest_port=8080|uri_path=Common/submittedFormList.html|url=GET Common/submittedFormList.html HTTP/1.1|http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36
2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Request|http_method=GET|site=/CommonCommon_7003|x.x.x.x|x.x.x.x|dest_port=7003|uri_path=/|url=GET / HTTP/1.1|http_user_agent=cosmos-v1
2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Response|http_method=GET|site=/CommonCommon_7003|x.x.x.x|x.x.x.x|dest_port=7003|uri_path=/|url=GET / HTTP/1.1|status=403|duration=1|http_user_agent=cosmos-v1|bytes_out=1166
2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Request|http_method=POST|site=/CommonCommon_7020|x.x.x.x|x.x.x.x|dest_port=7020|uri_path=nerf/connection-application/connection-offer.jsf|url=POST nerf/connection-application/connection-offer.jsf HTTP/1.1|http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36
2021-10-25T09:13:50+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Response|http_method=POST|site=/CommonCommon_7020|x.x.x.x|x.x.x.x|dest_port=7020|uri_path=nerf/connection-application/connection-offer.jsf|url=POST nerf/connection-application/connection-offer.jsf HTTP/1.1|status=200|duration=136|http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36|bytes_out=26270
2021-10-25T09:13:50+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM|http_phase=Response|http_method=GET|site=/CommonCommon_8080|x.x.x.x|x.x.x.x|dest_port=8080|uri_path=Common/submittedFormList.html|url=GET Common/submittedFormList.html HTTP/1.1|status=200|duration=700|http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36|bytes_out=330

Labels (1)
0 Karma

tscroggins
Influencer

If the events do not contain line breaks, the default LINE_BREAKER value of ([\r\n]+) should work. SHOULD_LINEMERGE must be set to false.

Try a props.conf stanza on your heavy forwarder or indexer with the following settings:

MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%:z

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It would help to know what you've tried so far.

Have you tried this?

LINE_BREAKER = ()\d{4}-\d\d-\d\dT
---
If this reply helps you, Karma would be appreciated.
0 Karma

oylkm
Explorer

This is what I have so far - ()\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\D\d\d\D\d\d\s\d+.\d+.\d+.\d+\s\w*\D\d\Wvendor_product=F5-LTM. I want to re-arrange this log into its own sourcetype. The logs are coming in via UF on a Syslog server and I understand breaking up the sourcetype from its main sourcetype requires applying the settings on the indexers or HF. I have no idea how to go about this. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That regex looks like it's a lot longer than is necessary, but does it work?

Assigning this sourcetype to logs is a two-step process.  The first step is to reference the sourcetype in an inputs.conf file on the UF.

 

[monitor:///path/to/file]
sourcetype = mynewsourcetype

 

The second step is to add sourcetype settings to a props.conf file on the indexers or heavy forwarders (whichever sees the data first)).

 

[mynewsourcetype]
LINE_BREAKER = ()\d{4}-\d\d-\d\d
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%:z

 

Be sure to restart the instances after modifying the config files.

EDIT: Fixed "]" typo in LINE_BREAKER setting.

---
If this reply helps you, Karma would be appreciated.
0 Karma

oylkm
Explorer

I've also tried this format: ()\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d.\d\d.\d\d\s\d+.\d+.\d+.\d+\s\bCEF\b but this didn't work either. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

"It didn't work" doesn't say enough.  What results do you get and how to those results not meet expectations?  How are you testing the changes?  One easy way is with the Add Data wizard on a SH.

Remember to restart the parsing instance after changing props.conf.  Also, changes to LINE_BREAKER apply only to new data - not data already indexed.

---
If this reply helps you, Karma would be appreciated.
0 Karma

oylkm
Explorer

Hi Rich, when I say it doesn't work I've attempted 'Add Data' in Splunk to see if my regex will work based on the new sourcetype I want to define. Presently all the logs are going into f5:bigip:syslog and this is what I want to achieve : 

2021-10-25T09:13:49+11:00 10.100.5.4 CEF:0|vendor_product=F5-LTM -  write a regex and extract based on this regex into a new sourcetype(f5:bigip:ltm). The regex I've tried didn't seem to work when testing using "Add Data", hope that makes more sense now. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I found and fixed a typo in the LINE_BREAKER setting in my earlier answer.  Please try it.

---
If this reply helps you, Karma would be appreciated.
0 Karma

oylkm
Explorer

No it didn't work. I would like the LTM part of the log to be picked up as well so that I can put it into the new sourcetype. 

0 Karma

oylkm
Explorer

I have tried a few regex definitions to try and split it into lines based on the date and time but that is not working. I basically want to split precisely on the time and date 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...