Hello
I am collecting data via AWS add on and what I have found is that my timestamp recognition isn't working properly.
I have a single AWS input using the [aws:s3:csv] sourcetype. this then uses transforms to update the sourcetype based on the file name the data comes from.
Config snips:
props.conf
[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000
TRANSFORMS-awss3 =sourcetypechange:awss3-object_rolemap_audit,sourcetypechange:awss3-authz-audit-logs
[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
#TZ=GMT
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
transforms.conf
[sourcetypechange:awss3-object_rolemap_audit]
SOURCE_KEY = MetaData:Source
REGEX = .*?object_rolemap_audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:object_rolemap_audit
[sourcetypechange:awss3-authz-audit-logs]
SOURCE_KEY = MetaData:Source
REGEX = .*?authz-audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:authz_audit
It seems that the data comes in at indextime from what I can see, even though I set recognition for each sourcetype. I believe that timestamping is happening at the initial pass into Splunk before it gets the transforms applied.
How can i set timestamping via the initial sourcetype if there are multiple formats for the sourcetype depending on the file? Since its not honoring the timestamp recognition setting post-transforms.
Thanks for the help.
Timestamp extraction is done before transforms are processed.
Consider setting props based on source rather than sourcetype.
[source::object_rolemap_audit.csv]
sourcetype = awss3:object_rolemap_audit
[source::authz-audit.csv]
sourcetype = awss3:authz_audit
[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000
[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1