Hello Splunk Community, My team is currently processing logs from a single source that can contain events with different timestamp formats. We are debating the best configuration approach and would like input from the community. Option 1: Using props.conf with transforms (Current setup) We are currently using a TRANSFORMS-split rule in our props.conf file to differentiate the source types based on some criteria, and then applying a single TIME_FORMAT within each resulting source type stanza. This involves creating several dedicated source types for essentially the same data stream. Configuration Snippet Example: [xxx:tomcat9:catalina1] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) TIME_FORMAT=%d-%b-%Y %H:%M:%S TIME_PREFIX=^ MAX_TIMESTAMP_LOOKAHEAD=20 TRANSFORMS-split = catalina1, catalina2 [tomcat9stdout1] REGEX = \d+\-[a-zA-Z]+-\d+\s\d+\:\d+\:\d+ DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::xxx:tomcat9:stdout1 [tomcat9stdout2] REGEX = [0-9]{4}\-\d+\-\d+\s\d+\:\d+\:\d+ ... (etc for other formats) Option 2: Using datetime.xml The alternative approach suggested is to use a single source type and configure a datetime.xml file. This XML file would contain multiple regular expressions, allowing Splunk to iterate through them and automatically identify the correct timestamp format for each individual event. Question Which approach is considered the industry best practice for handling this specific scenario? Is the datetime.xml method generally more robust and maintainable than splitting source types via transforms? Thanks for your guidance! ... View more

We configured rsyslog to route data from a certain host to the file system of the server, and what we see is that lots of data reaches the file system as well as /var/log/messages. So, if rsyslog is configured to route to the file system, why do we still see data in /var/log/messages? ... View more