Getting Data In

how to set timestamp format for each event in a log file?

gduggan1
Path Finder

Here is the scenario. We have a log file that comes in that we do some modification on the sourcetype to set it based on the lines in the event. We are doing this using props and transforms. I am looking to set the time format for each of the rows individually. I would like to set the main sourcetype (first event of the log file) to "DATETIME_CONFIG=CURRENT" and the other sourcetypes (all other events in the log file) to "TIME_FORMAT = %s". Is this possible using the transforms assignment or will they have an affect on eachother?

props.conf

# I want this to be DATETIME_CONFIG=CURRENT
[sourcetypeexample:keyA:keyB]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 50
TZ = UTC
TIME_FORMAT = %s
TRUNCATE = 60000
PREAMBLE_REGEX = ^EndOfFile.*
TRANSFORMS-sourcetype = sourcetypetransform

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch1]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch2]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch3]
SHOULD_LINEMERGE = false
KV_MODE = none

transforms.conf

[sourcetypetransform]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,([a-zA-Z-_0-9]+),([a-zA-Z-_0-9]+)
FORMAT = sourcetype::sourcetypeexample:keyA:keyB:$1:$2

Is the time format set on the initial sourcetype assingment and not changeable after the transform metadata change?

0 Karma
1 Solution

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

View solution in original post

woodcock
Esteemed Legend

You should be able modify the time at the very end of the processing with INGEST_EVAL like this:

[appropriate stanza header here]
INGEST_EVAL _time = _indextime
0 Karma

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...