Is it possible to assign different timestamps based on log line contents within the same sourcetype?

[daishih]

I am sending "pan:traffic" logs from our Palo Alto 3050 firewall to Splunk. I want the "_time" fields to be the same value as the "start_time" field when the log line contains "start" and use the time stamp that follows "PA-3050 1," when the log line contains "end"

Is it possible to do different time stamps for events in the same source-type "pan:traffic" ? I've bolded the time stamps I want to use for each in the samples below:

Sample start log line:
Feb 14 09:07:25 PA-3050 1,2017/02/14 09:07:25,001701007055,TRAFFIC,start,0,2017/02/14 09:07:25,10.0.0.1,77.000.000.88,0.0.0.0,0.0.0.0,User-to-Ext-Allow-DLP,domain\user,,web-browsing,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 09:07:25,55537,1,58861,80,0,0,0x0,tcp,allow,740,678,62,4,2017/02/14 09:07:24,0,any,0,442739216,0x0,10.0.0.0-10.255.255.255,US,0,3,1,n/a,0,0,0,0,,PA-3050,from-policy

Sample end log line:
Feb 14 10:21:25 PA-3050 1,2017/02/14 10:21:24,001701007055,TRAFFIC,end,0,2017/02/14 10:21:24,10.1.1.2,8.8.8.8,0.0.0.0,0.0.0.0,IntDC-to-ExtDNS,,,dns,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 10:21:24,256848,1,50770,53,0,0,0x19,udp,allow,842,85,757,2,2017/02/14 10:20:52,30,any,0,443053418,0x0,10.0.0.0-10.255.255.255,US,0,1,1,aged-out,0,0,0,0,,PA-3050,from-policy

[jkat54 · Accepted solution]

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

[daishih]

This works perfectly, thank you so much! It had never occurred to me to do a transform like that.

[jkat54]

Im actually surprised it works because the transforms.conf documentation says _time is time in epoch... but hey... if it works, it works!

[gcusello]

Hi daishih,
no: timestamp is unique for a sourcetype.

Maybe you could a try but you have more license consuption:

• index all logs with the first sourcetype (so the first timestamp),
• extract the logs that you want with the second sourcetype (so the second timestamp) writing them in a file
• reindex the second one using the second sourcetype
• use a filer in your searches to exclude the extracted events from the first search

I understand that is a pork-around, but I don't see anything else.

Bye.
Giuseppe