Getting Data In

Is it possible to assign different timestamps based on log line contents within the same sourcetype?

daishih
Path Finder

I am sending "pan:traffic" logs from our Palo Alto 3050 firewall to Splunk. I want the "_time" fields to be the same value as the "start_time" field when the log line contains "start" and use the time stamp that follows "PA-3050 1," when the log line contains "end"

Is it possible to do different time stamps for events in the same source-type "pan:traffic" ? I've bolded the time stamps I want to use for each in the samples below:

Sample start log line:
Feb 14 09:07:25 PA-3050 1,2017/02/14 09:07:25,001701007055,TRAFFIC,start,0,2017/02/14 09:07:25,10.0.0.1,77.000.000.88,0.0.0.0,0.0.0.0,User-to-Ext-Allow-DLP,domain\user,,web-browsing,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 09:07:25,55537,1,58861,80,0,0,0x0,tcp,allow,740,678,62,4,2017/02/14 09:07:24,0,any,0,442739216,0x0,10.0.0.0-10.255.255.255,US,0,3,1,n/a,0,0,0,0,,PA-3050,from-policy

Sample end log line:
Feb 14 10:21:25 PA-3050 1,2017/02/14 10:21:24,001701007055,TRAFFIC,end,0,2017/02/14 10:21:24,10.1.1.2,8.8.8.8,0.0.0.0,0.0.0.0,IntDC-to-ExtDNS,,,dns,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 10:21:24,256848,1,50770,53,0,0,0x19,udp,allow,842,85,757,2,2017/02/14 10:20:52,30,any,0,443053418,0x0,10.0.0.0-10.255.255.255,US,0,1,1,aged-out,0,0,0,0,,PA-3050,from-policy

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

daishih
Path Finder

This works perfectly, thank you so much! It had never occurred to me to do a transform like that.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Im actually surprised it works because the transforms.conf documentation says _time is time in epoch... but hey... if it works, it works!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi daishih,
no: timestamp is unique for a sourcetype.

Maybe you could a try but you have more license consuption:

  • index all logs with the first sourcetype (so the first timestamp),
  • extract the logs that you want with the second sourcetype (so the second timestamp) writing them in a file
  • reindex the second one using the second sourcetype
  • use a filer in your searches to exclude the extracted events from the first search

I understand that is a pork-around, but I don't see anything else.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...