Getting Data In

how to set timestamp format for each event in a log file?

gduggan1
Path Finder

Here is the scenario. We have a log file that comes in that we do some modification on the sourcetype to set it based on the lines in the event. We are doing this using props and transforms. I am looking to set the time format for each of the rows individually. I would like to set the main sourcetype (first event of the log file) to "DATETIME_CONFIG=CURRENT" and the other sourcetypes (all other events in the log file) to "TIME_FORMAT = %s". Is this possible using the transforms assignment or will they have an affect on eachother?

props.conf

# I want this to be DATETIME_CONFIG=CURRENT
[sourcetypeexample:keyA:keyB]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 50
TZ = UTC
TIME_FORMAT = %s
TRUNCATE = 60000
PREAMBLE_REGEX = ^EndOfFile.*
TRANSFORMS-sourcetype = sourcetypetransform

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch1]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch2]
SHOULD_LINEMERGE = false
KV_MODE = none

# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch3]
SHOULD_LINEMERGE = false
KV_MODE = none

transforms.conf

[sourcetypetransform]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,([a-zA-Z-_0-9]+),([a-zA-Z-_0-9]+)
FORMAT = sourcetype::sourcetypeexample:keyA:keyB:$1:$2

Is the time format set on the initial sourcetype assingment and not changeable after the transform metadata change?

0 Karma
1 Solution

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

View solution in original post

woodcock
Esteemed Legend

You should be able modify the time at the very end of the processing with INGEST_EVAL like this:

[appropriate stanza header here]
INGEST_EVAL _time = _indextime
0 Karma

darrenfuller
Contributor

Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...