I am not sure if I can do this and it may be a problem with the host as we are looking into that as well. I have a host sending logs in UTC and EST. Can I specify in a props.conf events from source1 and host1 it is in UTC and source2 and host1 is EST? I am looking to pass 2 parameters since passing 1 parameter could impact other events at this time.
I understand this could be, and probably is, more of a consistency issue on the server side and we are addressing that issue. I am looking to find out if this is even possible and if so what would the stanza look like.
The only way this might work is to convert this host's Forwarder to a Heavy Forwarder so that all the indexing work for this host is done ON THIS HOST instead of on the Indexers. Then give this host have a custom
props.conf file that is different from the one you are using now that is only sent to this Heavy Forwarder that has this in it:
[source::/path/to/source1/*] TZ = UTC [source::/path/to/source2/*] TZ = US/Eastern
If by chance, all of the events from this host have an explicit TZ value inside the timestamps strings (even if it is wrong), there is another way to do it, but I am assuming that this is not the case.
You are right, this is not possible - unless you change to a heavy forwarder instead of a universal forwarder.
The problem is that the timezone must be set where the data is parsed and a universal forwarder does not parse timestamps. Once your data from source2 arrives at the indexers, it is parsed along with the source2 data from other hosts...
However, if you switch to a heavy forwarder, all of the props.conf and transforms.conf settings would need to be placed on the heavy forwarder. Because the heavy forwarder is only dealing with host1 data, you can set the following in props.conf
Assuming that host1 is UTC, you only need to set stanzas in props.conf for the sources that are different.
You are also right that this problem really should be resolved by fixing the logs. But if you can't do that, this will work. Overall, a heavy forwarder will be less efficient. This may also up the complexity of your environment. So are there are downsides. But the problem can be solved.
A final note: you know what would solve this problem forever? If the developers AWAYS INCLUDED A TIMEZONE in the timestamps. Splunk would see the timezone and just take care of it properly, without any need for hairpulling, or even simple configuration. THAT's what you should be pushing for! Here's a larger wish list: Splunk Logging best practices