topic Re: How to do event linebreaking and timestamp recognition in header lines without timestamp information! in Getting Data In

How to do event linebreaking and timestamp recognition in header lines without timestamp information!

smdasim — Thu, 01 Feb 2018 06:26:50 GMT

Hi Folks,

I am adding data from a log file with filename: server_zmslx1xt1119.log

For the timestamp, first 7 lines does not have timestamp and fail to parse and default to filemod time instead of (20180123154844 or 01/23/18 3:48:44:000).
I want splunk to pick up the timestamp for header fields from the line that have the timestamp (20180123154844)

Please advice

--------------------------------log file ------------------------
McAfee ePO 5.3.1.296
Server name: ZMSLX1XT1119 (zmslx1xt1119.xx.xxxx.xxxx.com.in)
Platform: Server 6.2
Processors: 4
Architecture: 64-bit
Physical memory: 16383 MB
20180123154844 I #02828 NAIMSERV PSO load: id=7298 ts=6480670
20180123154844 I #02828 NAIMSERV PSO load: id=7299 ts=6480679
20180123154844 I #02828 NAIMSERV PSO load: id=7300 ts=6480712
20180123154844 I #02828 NAIMSERV PSO load: id=7301 ts=6480716
20180123154844 I #02828 NAIMSERV PSO load: id=7302 ts=6480740
20180123154844 I #02828 NAIMSERV PSO load: id=7303 ts=6480749
20180123154844 I #02828 NAIMSERV PSO load: id=7304 ts=141092230
--------------------------------log file ------------------------

Regards
smdasim

Re: How to do event linebreaking and timestamp recognition in header lines without timestamp information!

micahkemp — Sat, 03 Feb 2018 04:32:25 GMT

If each of the lines in your example are expected to represent individual events, there is no way for Splunk to use a timestamp seen in a later event to base its own timestamp off of. If, however, you want to group the non-timestamped lines with the first timestamped line, this config will do that:

props.conf:

[<sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ts=[0-9]+([\n\r]+)
MAX_TIMESTAMP_LOOKAHEAD = 250 
TIME_FORMAT = %Y%m%d%H%M%S

This will group the non-timestamp lines with the first line that has a timestamp.

You could also consider dropping the non-timestamped lines entirely, if they're not really needed.

Re: How to do event linebreaking and timestamp recognition in header lines without timestamp information!

smdasim — Sat, 03 Feb 2018 12:12:17 GMT

Hi Micahkemp,

Can you please let me know more about the last line ,how it can be implemented .
"you could have Splunk use the first instance of a timestamp it sees (based on a regex) be used for that event".As an option i can put first few lines into a single event with timestamp -20180123154844
Thank you for your reply.
Regards
sdasim

Re: How to do event linebreaking and timestamp recognition in header lines without timestamp information!

micahkemp — Sat, 03 Feb 2018 20:49:17 GMT

Update my answer based on your comment.

Re: How to do event linebreaking and timestamp recognition in header lines without timestamp information!

micahkemp — Thu, 08 Feb 2018 02:07:02 GMT

Considering your latest answer post, did this solution not work for you, or did you just decide you don't want to have the six header lines associated with the first event?