Splunk Enterprise

How do I configure timestamp recognition on a single input for multiple files?

tkwaller1
Path Finder

Hello

I am collecting data via AWS add on and what I have found is that my timestamp recognition isn't working properly.

I have a single AWS input using the [aws:s3:csv] sourcetype. this then uses transforms to update the sourcetype based on the file name the data comes from.

Config snips:
props.conf

 

[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000
TRANSFORMS-awss3 =sourcetypechange:awss3-object_rolemap_audit,sourcetypechange:awss3-authz-audit-logs

[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
#TZ=GMT
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

 

transforms.conf

 

[sourcetypechange:awss3-object_rolemap_audit]
SOURCE_KEY = MetaData:Source
REGEX = .*?object_rolemap_audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:object_rolemap_audit

[sourcetypechange:awss3-authz-audit-logs]
SOURCE_KEY = MetaData:Source
REGEX = .*?authz-audit.csv
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::awss3:authz_audit

 

 

It seems that the data comes in at indextime from what I can see, even though I set recognition for each sourcetype. I believe that timestamping is happening at the initial pass into Splunk before it gets the transforms applied. 

 How can i set timestamping via the initial sourcetype if there are multiple formats for the sourcetype depending on the file? Since its not honoring the timestamp recognition setting post-transforms.


Thanks for the help.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Timestamp extraction is done before transforms are processed.

Consider setting props based on source rather than sourcetype.

[source::object_rolemap_audit.csv]
sourcetype = awss3:object_rolemap_audit

[source::authz-audit.csv]
sourcetype = awss3:authz_audit

[aws:s3:csv]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
TRUNCATE = 20000

[awss3:object_rolemap_audit]
TIME_FORMAT=%d %b %Y %H:%M:%S
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1

[awss3:authz_audit]
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...