Getting Data In

Is it possible to assign different timestamps based on log line contents within the same sourcetype?

daishih
Path Finder

I am sending "pan:traffic" logs from our Palo Alto 3050 firewall to Splunk. I want the "_time" fields to be the same value as the "start_time" field when the log line contains "start" and use the time stamp that follows "PA-3050 1," when the log line contains "end"

Is it possible to do different time stamps for events in the same source-type "pan:traffic" ? I've bolded the time stamps I want to use for each in the samples below:

Sample start log line:
Feb 14 09:07:25 PA-3050 1,2017/02/14 09:07:25,001701007055,TRAFFIC,start,0,2017/02/14 09:07:25,10.0.0.1,77.000.000.88,0.0.0.0,0.0.0.0,User-to-Ext-Allow-DLP,domain\user,,web-browsing,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 09:07:25,55537,1,58861,80,0,0,0x0,tcp,allow,740,678,62,4,2017/02/14 09:07:24,0,any,0,442739216,0x0,10.0.0.0-10.255.255.255,US,0,3,1,n/a,0,0,0,0,,PA-3050,from-policy

Sample end log line:
Feb 14 10:21:25 PA-3050 1,2017/02/14 10:21:24,001701007055,TRAFFIC,end,0,2017/02/14 10:21:24,10.1.1.2,8.8.8.8,0.0.0.0,0.0.0.0,IntDC-to-ExtDNS,,,dns,vsys1,trust,untrust,ethernet1/2,ethernet1/1,LF-LOG-ALLOW,2017/02/14 10:21:24,256848,1,50770,53,0,0,0x19,udp,allow,842,85,757,2,2017/02/14 10:20:52,30,any,0,443053418,0x0,10.0.0.0-10.255.255.255,US,0,1,1,aged-out,0,0,0,0,,PA-3050,from-policy

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Give this a shot, should happen at index time (put on forwarders & indexers)

Props.conf:

[pan:traffic]
TRANSFORMS-dateStartTimeTransform=dateStartTimeTransform
TRANSFORMS-dateEndTimeTransform=dateEndTimeTransform

Transforms.conf:

[dateStartTimeTransform]
SOURCE_KEY = _raw
REGEX = (?<=start).*(\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})
DEST_KEY = _time

[dateEndTimeTransform]
SOURCE_KEY = _raw
REGEX = (\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2})(?=.*end)
DEST_KEY = _time

daishih
Path Finder

This works perfectly, thank you so much! It had never occurred to me to do a transform like that.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Im actually surprised it works because the transforms.conf documentation says _time is time in epoch... but hey... if it works, it works!

0 Karma

gcusello
Legend

Hi daishih,
no: timestamp is unique for a sourcetype.

Maybe you could a try but you have more license consuption:

  • index all logs with the first sourcetype (so the first timestamp),
  • extract the logs that you want with the second sourcetype (so the second timestamp) writing them in a file
  • reindex the second one using the second sourcetype
  • use a filer in your searches to exclude the extracted events from the first search

I understand that is a pork-around, but I don't see anything else.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...