Props.conf when not every line has a date time

[ssaenger]

Hi,

i am trying to build a props.conf for the following log entry.
The log is based on an sql run and so is a mixture of an sql output as well as text.

My props.conf is;
[check_script]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %y/%m/%d %H:%M:%S
TRUNCATE = 0

       03/03/20 07:11:01
       Events without Source entries

NUM         SERVICE_KEY NAME                       KEY             EVENT_DATE_TIME EVENT_DATE_TIME  START_DATE_TIME NAME

---

       45           401        Greats         0429785         58911.3298611 03-Mar-20 07:55:00   58911.3298611 High
       45           401        Greats           0429786         58911.4131944 03-Mar-20 09:55:00   58911.4131944 Men
       45           401        Greats        0429787         58911.4791667 03-Mar-20 11:30:00   58911.4791667   Blind
       45           401        Greats        0429788         58911.5729167 03-Mar-20 13:45:00   58911.5729167  Desert 
       45           401        Greats        0429789         58911.6388889 03-Mar-20 15:20:00   58911.6388889   Jaw
       45           401        Greats        0429790         58911.7291667 03-Mar-20 17:30:00   58911.7291667   War 
       45           401        Greats        0429791            58911.8125 03-Mar-20 19:30:00      58911.8125          Men
       45           401        Greats        0429792             58911.875 03-Mar-20 21:00:00       58911.875            Blind
       45           401        Greats        0429793           58911.96875 03-Mar-20 23:15:00     58911.96875       First
       45           401        Greats        0429794         58912.0416667 04-Mar-20 01:00:00   58912.0416667  Blood 
       45           401        Greats        0429795         58912.1145833 04-Mar-20 02:45:00   58912.1145833   3
       45           401        Greats        0429796         58912.1909722 04-Mar-20 04:35:00   58912.1909722  Desert 

     12 rows selected.

      03/03/20 07:11:01
      Gaps in Push Schedule

       Last Event for Service Key: 409 03-Mar-2020 09:35:00 58911.3993056 Duration: 01:45:00  'land: Tap'
       Warning: Last Scheduled Asset Finishes 10 Minutes+ before Service Switch Off (or now + 48 hours)...
       Last Event for Service Key: 409 03-Mar-2020 09:33:00 58911.3979167 Duration: 01:45:00  'land:  Tap'
       Warning: Last Scheduled Asset Finishes 10 Minutes+ before Service Switch Off (or now + 48 hours)...

Its currently splitting on each line and associating the date in the log line with an entry.

[FrankVl]

So, you want to ingest all those lines as individual events into splunk, but you want to use the overall timestamp from the header of that structured file?

Not sure that is possible really. You might get better results by ingesting this whole thing as a single event, with the timestamp from the header, and do further manipulation of the event as part of your searches. The multikv command will be very useful for that, see https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Multikv

Either that or change the way you produce this log to produce a log that contains something more compatible with Splunk (ie. line by line events, including a timestamp, rather than such structured log files). Or do some preprocessing to transform these files into something that is easy to ingest into splunk.

[to4kawa]

_time is no.1 and no.16?

but LINE_BREAKER = ([\r\n]+) ,
this setting can't extract _time.
I am not sure what you want to separate and extract logs.

[ssaenger]

Hi to4kawa.

I want to have ever line in my log as a new line in Splunk. This is what a thought
LINE_BREAKER = ([\r\n]+)
did.
My Line

MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %y/%m/%d %H:%M:%S

I thought this would extract the time and also ignore ALL other time references in the log line.
The time that relates to log line entry is 03/03/20 07:11:01

My problem is, as you have stated. Splunk is entering the log line against the time extracted from the log line. eh 04-Mar-20 02:45:00.
This is what i want to avoid.

[to4kawa]

An event should have _time.
the result of LINE_BREAKER = ([\r\n]+),
45 401 Greats 0429785 58911.3298611 03-Mar-20 07:55:00 58911.3298611 High
is single event.
There must be _time, _time is extracted.

LINE_BREAKER = (?=\d\d\/\d\d\/\d\d)
How about this?