Getting Data In

how to set timestamp format for each event in a log file?

gduggan1
Path Finder

Here is the scenario. We have a log file that comes in that we do some modification on the sourcetype to set it based on the lines in the event. We are doing this using props and transforms. I am looking to set the time format for each of the rows individually. I would like to set the main sourcetype (first event of the log file) to "DATETIME_CONFIG=CURRENT" and the other sourcetypes (all other events in the log file) to "TIME_FORMAT = %s". Is this possible using the transforms assignment or will they have an affect on eachother?

props.conf

# I want this to be DATETIME_CONFIG=CURRENT
[sourcetypeexample:keyA:keyB]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 50
TZ = UTC
TIME_FORMAT = %s
TRUNCATE = 60000
PREAMBLE_REGEX = ^EndOfFile.*
TRANSFORMS-sourcetype = sourcetypetransform

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch1]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch2]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch3]
SHOULD_LINEMERGE = false
KV_MODE = none

transforms.conf

[sourcetypetransform]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,([a-zA-Z-_0-9]+),([a-zA-Z-_0-9]+)
FORMAT = sourcetype::sourcetypeexample:keyA:keyB:$1:$2

Is the time format set on the initial sourcetype assingment and not changeable after the transform metadata change?

0 Karma
1 Solution

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

View solution in original post

woodcock
Esteemed Legend

You should be able modify the time at the very end of the processing with INGEST_EVAL like this:

[appropriate stanza header here]
INGEST_EVAL _time = _indextime
0 Karma

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...