topic Formatting line breaks and timestamps with props.conf in Getting Data In

Formatting line breaks and timestamps with props.conf

Lazarix — Mon, 20 Jan 2014 17:09:57 GMT

Hi guys,

I'm trying to deal with a specific requirement and I've tried modifying the props.conf to my liking, but I can't get it to do what I'd like it to do.

I have a multi-line event that I'd like parsed as a single event as follows:

Mon Jan 20 16:55:01 GMT 2014
Jan 20 16:54 SCHEDULER.log
Jan 20 16:54 DOWNLOADFROMNZ.log
Jan 20 16:52 UPLOADTONZ.log
Jan 20 16:54 NZJOBLOGGER.log
Jan 20 16:29 ETLPROCESS.log

From the above, the rules would be as follows:

Timestamp would be the first line, always in that format
I would then have the multi-line event below parsed as a single event so that I could table it and show that table
The table columns would be through field extraction as:

Last_Active Filename

My problems are:

I can't get it to parse all of the above as a single event

I can't seem to exclude the timestamp in the first line. The line itself doesn't need to be indexed, only referenced to grab the event timestamp

Does anyone have any ideas how best I can achieve this?

Thanks in advance

Re: Formatting line breaks and timestamps with props.conf

kristian_kolb — Mon, 20 Jan 2014 20:14:20 GMT

Something like this ought to work. The "Last_Active" values will be ordinary strings, so don't expect any date-like behaviour, unless you convert it further, e.g. with convert, or strftime/strptime.

props.conf

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n+])\w{3}\s\w{3}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
REPORT-my_extractions

transforms.conf

[my_extractions]
REGEX = (?m)(?<Last_Active>\w+\s\d+\s\d+:\d+)\s+(?<Filename>\S+)[\r\n]*
MV_ADD = true

Re: Formatting line breaks and timestamps with props.conf

Lazarix — Tue, 21 Jan 2014 10:05:59 GMT

Yep, This did exactly what I wanted it to do, apart from it including the first line (the date/time stamp) in the event. It did however make everything one event.

The problem I found after doing this, is that I couldn't table it because the field extractions stopped working. I guess I'll have to go back to a single line for each entry.