topic Is it possible to assign different timestamps based on log line contents within the same sourcetype? in Getting Data In

Is it possible to assign different timestamps based on log line contents within the same sourcetype?

daishih — Tue, 29 Sep 2020 12:51:27 GMT

I am sending "pan:traffic" logs from our Palo Alto 3050 firewall to Splunk. I want the "_time" fields to be the same value as the "start_time" field when the log line contains "start" and use the time stamp that follows "PA-3050 1," when the log line contains "end"

Is it possible to do different time stamps for events in the same source-type "pan:traffic" ? I've bolded the time stamps I want to use for each in the samples below:

Sample start log line:
Feb 14 09:07:25 PA-3050 1,2017/02/14 09:07:25,001701007055,TRAFFIC,start,0,2017/02/14 09:07:25,10.0.0.1,77.000.000.88,0.0.0.0,0.0.0.0,User-to-Ext-Allow-DLP,domain\user,,web-browsing,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 09:07:25,55537,1,58861,80,0,0,0x0,tcp,allow,740,678,62,4,2017/02/14 09:07:24,0,any,0,442739216,0x0,10.0.0.0-10.255.255.255,US,0,3,1,n/a,0,0,0,0,,PA-3050,from-policy

Sample end log line:
Feb 14 10:21:25 PA-3050 1,2017/02/14 10:21:24,001701007055,TRAFFIC,end,0,2017/02/14 10:21:24,10.1.1.2,8.8.8.8,0.0.0.0,0.0.0.0,IntDC-to-ExtDNS,,,dns,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 10:21:24,256848,1,50770,53,0,0,0x19,udp,allow,842,85,757,2,2017/02/14 10:20:52,30,any,0,443053418,0x0,10.0.0.0-10.255.255.255,US,0,1,1,aged-out,0,0,0,0,,PA-3050,from-policy

Re: Is it possible to assign different timestamps based on log line contents within the same sourcetype?

gcusello — Tue, 14 Feb 2017 16:41:38 GMT

Hi daishih,
no: timestamp is unique for a sourcetype.

Maybe you could a try but you have more license consuption:

index all logs with the first sourcetype (so the first timestamp),
extract the logs that you want with the second sourcetype (so the second timestamp) writing them in a file
reindex the second one using the second sourcetype
use a filer in your searches to exclude the extracted events from the first search

I understand that is a pork-around, but I don't see anything else.

Bye.
Giuseppe

Re: Is it possible to assign different timestamps based on log line contents within the same sourcetype?

jkat54 — Tue, 14 Feb 2017 16:48:12 GMT

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

Re: Is it possible to assign different timestamps based on log line contents within the same sourcetype?

daishih — Tue, 14 Feb 2017 16:59:06 GMT

This works perfectly, thank you so much! It had never occurred to me to do a transform like that.

Re: Is it possible to assign different timestamps based on log line contents within the same sourcetype?

jkat54 — Tue, 14 Feb 2017 17:01:45 GMT

Im actually surprised it works because the transforms.conf documentation says _time is time in epoch... but hey... if it works, it works!