topic Re: how to split log file on non-standard date and special characters in Getting Data In

how to split log file on non-standard date and special characters

ssaenger — Tue, 29 Sep 2020 14:17:27 GMT

Hi All,

I have a log file that has a non standard date/time and special characters and i am trying to split the lines on the forwarder, however i cannot get Splunk to read anything less than 257 lines at a time!

My Log lines are as follows;

[Wed May 31 12:34:22.431862 2017] [:debug] [pid 10890] ils_util.c(615): SSS_DBG:[10890,55584809]:-  
+------------------- Start Of Request --------------------
| Id = 55584809,  SessionId = 1300,  Sid = 28716112
+---------------------------------------------------------
[Wed May 31 12:34:22.431867 2017] [:debug] [pid 10890] ils_util.c(618): SSS_DBG:[10890,55584809]:-  Start Of Request - Id = 55584809,  SessionId = 1300,  Sid = 28716112

my props.conf file is below

[int_error_log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^[\d{3}\s\d{3}\s\d{2}\s\d{2}:\d{2}:\d{2}.\d{6}\s\d{4}]\s
MAX_TIMESTAMP_LOOKAHEAD = 35
TIME_PREFIX = ^

i have tried to use LINE_BREAKER=([+||]+) in order to deal with the special characters, however this resulted in even more lines per break.

thanks

Re: how to split log file on non-standard date and special characters

somesoni2 — Wed, 31 May 2017 16:42:41 GMT

First you need to setup the event parsing (line breaking, timestamp identification etc) on your indexers/heavy forwarders. If the instance monitoring this is Universal forwarders, then move the line breaking configurations to your Indexers/heavy forwarders (a restart of Splunk would e needed).

Second, give this config a try

[int_error_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\[\w+\s+\w+\s+\d+\s+\d+\:\d+\:\d+\.\d+)
TIME_PREFIX = ^\[\w+\s+
TIME_FORMAT = %b %d %H:%M:%S.%N %Y
MAX_TIMESTAMP_LOOKAHEAD = 27

Re: how to split log file on non-standard date and special characters

ssaenger — Wed, 14 Jun 2017 15:14:35 GMT

Thanks somesoni2,
Worked perfectly 🙂 - apologies for delay in my reply have been away 🙂