Getting Data In

how to split log file on non-standard date and special characters

ssaenger
Communicator

Hi All,

I have a log file that has a non standard date/time and special characters and i am trying to split the lines on the forwarder, however i cannot get Splunk to read anything less than 257 lines at a time!

My Log lines are as follows;

[Wed May 31 12:34:22.431862 2017] [:debug] [pid 10890] ils_util.c(615): SSS_DBG:[10890,55584809]:-  
+------------------- Start Of Request --------------------
| Id = 55584809,  SessionId = 1300,  Sid = 28716112
+---------------------------------------------------------
[Wed May 31 12:34:22.431867 2017] [:debug] [pid 10890] ils_util.c(618): SSS_DBG:[10890,55584809]:-  Start Of Request - Id = 55584809,  SessionId = 1300,  Sid = 28716112

my props.conf file is below

[int_error_log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^[\d{3}\s\d{3}\s\d{2}\s\d{2}:\d{2}:\d{2}.\d{6}\s\d{4}]\s
MAX_TIMESTAMP_LOOKAHEAD = 35
TIME_PREFIX = ^

i have tried to use LINE_BREAKER=([+||]+) in order to deal with the special characters, however this resulted in even more lines per break.

thanks

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

First you need to setup the event parsing (line breaking, timestamp identification etc) on your indexers/heavy forwarders. If the instance monitoring this is Universal forwarders, then move the line breaking configurations to your Indexers/heavy forwarders (a restart of Splunk would e needed).

Second, give this config a try

[int_error_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\[\w+\s+\w+\s+\d+\s+\d+\:\d+\:\d+\.\d+)
TIME_PREFIX = ^\[\w+\s+
TIME_FORMAT = %b %d %H:%M:%S.%N %Y
MAX_TIMESTAMP_LOOKAHEAD = 27

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

First you need to setup the event parsing (line breaking, timestamp identification etc) on your indexers/heavy forwarders. If the instance monitoring this is Universal forwarders, then move the line breaking configurations to your Indexers/heavy forwarders (a restart of Splunk would e needed).

Second, give this config a try

[int_error_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\[\w+\s+\w+\s+\d+\s+\d+\:\d+\:\d+\.\d+)
TIME_PREFIX = ^\[\w+\s+
TIME_FORMAT = %b %d %H:%M:%S.%N %Y
MAX_TIMESTAMP_LOOKAHEAD = 27
0 Karma

ssaenger
Communicator

Thanks somesoni2,
Worked perfectly 🙂 - apologies for delay in my reply have been away 🙂

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...