Here is the scenario. We have a log file that comes in that we do some modification on the sourcetype to set it based on the lines in the event. We are doing this using props and transforms. I am looking to set the time format for each of the rows individually. I would like to set the main sourcetype (first event of the log file) to "DATETIME_CONFIG=CURRENT" and the other sourcetypes (all other events in the log file) to "TIME_FORMAT = %s". Is this possible using the transforms assignment or will they have an affect on eachother?
props.conf
# I want this to be DATETIME_CONFIG=CURRENT
[sourcetypeexample:keyA:keyB]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 50
TZ = UTC
TIME_FORMAT = %s
TRUNCATE = 60000
PREAMBLE_REGEX = ^EndOfFile.*
TRANSFORMS-sourcetype = sourcetypetransform
# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch1]
SHOULD_LINEMERGE = false
KV_MODE = none
# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch2]
SHOULD_LINEMERGE = false
KV_MODE = none
# I want this to be TIME_FORMAT = %s
[sourcetypeexample:keyA:keyB:card:cardSch3]
SHOULD_LINEMERGE = false
KV_MODE = none
transforms.conf
[sourcetypetransform]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,([a-zA-Z-_0-9]+),([a-zA-Z-_0-9]+)
FORMAT = sourcetype::sourcetypeexample:keyA:keyB:$1:$2
Is the time format set on the initial sourcetype assingment and not changeable after the transform metadata change?
Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.
You should be able modify the time at the very end of the processing with INGEST_EVAL
like this:
[appropriate stanza header here]
INGEST_EVAL _time = _indextime
Alas, you are kind of sunk, because timestamp recognition happens before TRANSFORMS's. By the time your transforms goes through, the timestamp has already been identified and the system has moved on to other things.