How do I configure timestamp recognition on a single input for multiple files?

[tkwaller1]

Hello

I am collecting data via AWS add on and what I have found is that my timestamp recognition isn't working properly.

I have a single AWS input using the [aws:s3:csv] sourcetype. this then uses transforms to update the sourcetype based on the file name the data comes from.

Config snips:
props.conf

[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000
TRANSFORMS-awss3 =sourcetypechange:awss3-object_rolemap_audit,sourcetypechange:awss3-authz-audit-logs

[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
#TZ=GMT
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

transforms.conf

[sourcetypechange:awss3-object_rolemap_audit]
SOURCE_KEY = MetaData:Source
REGEX = .*?object_rolemap_audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:object_rolemap_audit

[sourcetypechange:awss3-authz-audit-logs]
SOURCE_KEY = MetaData:Source
REGEX = .*?authz-audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:authz_audit

It seems that the data comes in at indextime from what I can see, even though I set recognition for each sourcetype. I believe that timestamping is happening at the initial pass into Splunk before it gets the transforms applied. 

 How can i set timestamping via the initial sourcetype if there are multiple formats for the sourcetype depending on the file? Since its not honoring the timestamp recognition setting post-transforms.

Thanks for the help.

[richgalloway]

Timestamp extraction is done before transforms are processed.

Consider setting props based on source rather than sourcetype.

[source::object_rolemap_audit.csv]
sourcetype = awss3:object_rolemap_audit

[source::authz-audit.csv]
sourcetype = awss3:authz_audit

[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000

[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1