Getting Data In

how to set timestamp format for each event in a log file?

gduggan1
Path Finder

Here is the scenario. We have a log file that comes in that we do some modification on the sourcetype to set it based on the lines in the event. We are doing this using props and transforms. I am looking to set the time format for each of the rows individually. I would like to set the main sourcetype (first event of the log file) to "DATETIME_CONFIG=CURRENT" and the other sourcetypes (all other events in the log file) to "TIME_FORMAT = %s". Is this possible using the transforms assignment or will they have an affect on eachother?

props.conf

# I want this to be DATETIME_CONFIG=CURRENT
[sourcetypeexample:keyA:keyB]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 50
TZ = UTC
TIME_FORMAT = %s
TRUNCATE = 60000
PREAMBLE_REGEX = ^EndOfFile.*
TRANSFORMS-sourcetype = sourcetypetransform

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch1]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch2]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch3]
SHOULD_LINEMERGE = false
KV_MODE = none

transforms.conf

[sourcetypetransform]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,([a-zA-Z-_0-9]+),([a-zA-Z-_0-9]+)
FORMAT = sourcetype::sourcetypeexample:keyA:keyB:$1:$2

Is the time format set on the initial sourcetype assingment and not changeable after the transform metadata change?

0 Karma
1 Solution

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

View solution in original post

woodcock
Esteemed Legend

You should be able modify the time at the very end of the processing with INGEST_EVAL like this:

[appropriate stanza header here]
INGEST_EVAL _time = _indextime
0 Karma

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...