i am trying to build a props.conf for the following log entry.
The log is based on an sql run and so is a mixture of an sql output as well as text.
My props.conf is;
SHOULDLINEMERGE = false
LINEBREAKER = ([\r\n]+)
MAXTIMESTAMPLOOKAHEAD = 17
TIMEPREFIX = ^
TIME_FORMAT = %y/%m/%d %H:%M:%S
TRUNCATE = 0
03/03/20 07:11:01 Events without Source entries NUM SERVICE_KEY NAME KEY EVENT_DATE_TIME EVENT_DATE_TIME START_DATE_TIME NAME
45 401 Greats 0429785 58911.3298611 03-Mar-20 07:55:00 58911.3298611 High 45 401 Greats 0429786 58911.4131944 03-Mar-20 09:55:00 58911.4131944 Men 45 401 Greats 0429787 58911.4791667 03-Mar-20 11:30:00 58911.4791667 Blind 45 401 Greats 0429788 58911.5729167 03-Mar-20 13:45:00 58911.5729167 Desert 45 401 Greats 0429789 58911.6388889 03-Mar-20 15:20:00 58911.6388889 Jaw 45 401 Greats 0429790 58911.7291667 03-Mar-20 17:30:00 58911.7291667 War 45 401 Greats 0429791 58911.8125 03-Mar-20 19:30:00 58911.8125 Men 45 401 Greats 0429792 58911.875 03-Mar-20 21:00:00 58911.875 Blind 45 401 Greats 0429793 58911.96875 03-Mar-20 23:15:00 58911.96875 First 45 401 Greats 0429794 58912.0416667 04-Mar-20 01:00:00 58912.0416667 Blood 45 401 Greats 0429795 58912.1145833 04-Mar-20 02:45:00 58912.1145833 3 45 401 Greats 0429796 58912.1909722 04-Mar-20 04:35:00 58912.1909722 Desert 12 rows selected. 03/03/20 07:11:01 Gaps in Push Schedule Last Event for Service Key: 409 03-Mar-2020 09:35:00 58911.3993056 Duration: 01:45:00 'land: Tap' Warning: Last Scheduled Asset Finishes 10 Minutes+ before Service Switch Off (or now + 48 hours)... Last Event for Service Key: 409 03-Mar-2020 09:33:00 58911.3979167 Duration: 01:45:00 'land: Tap' Warning: Last Scheduled Asset Finishes 10 Minutes+ before Service Switch Off (or now + 48 hours)...
Its currently splitting on each line and associating the date in the log line with an entry.
_time is no.1 and no.16?
LINE_BREAKER = ([\r\n]+) ,
this setting can't extract _time.
I am not sure what you want to separate and extract logs.
I want to have ever line in my log as a new line in Splunk. This is what a thought
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 17 TIME_PREFIX = ^ TIME_FORMAT = %y/%m/%d %H:%M:%S
I thought this would extract the time and also ignore ALL other time references in the log line.
The time that relates to log line entry is
My problem is, as you have stated. Splunk is entering the log line against the time extracted from the log line. eh
This is what i want to avoid.
An event should have time.
the result of `LINEBREAKER = ([\r\n]+)
, 45 401 Greats 0429785 58911.3298611 03-Mar-20 07:55:00 58911.3298611 High`
is single event.
There must be _time, _time is extracted.
LINE_BREAKER = (?=\d\d\/\d\d\/\d\d)
How about this?
So, you want to ingest all those lines as individual events into splunk, but you want to use the overall timestamp from the header of that structured file?
Not sure that is possible really. You might get better results by ingesting this whole thing as a single event, with the timestamp from the header, and do further manipulation of the event as part of your searches. The
multikv command will be very useful for that, see https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Multikv
Either that or change the way you produce this log to produce a log that contains something more compatible with Splunk (ie. line by line events, including a timestamp, rather than such structured log files). Or do some preprocessing to transform these files into something that is easy to ingest into splunk.