The best way to solve this would be to configure all your network equipment (or at least those that do not log time zone info) to run / log in a common time zone (e.g. UTC).
Alternatively, you could configure Splunk with per-host TZ setting in props.conf, but with many devices that might become hard to manage.
If you really want to go down the route of adding an additional timestamp based on when syslog-ng processed the event, you can do that by defining your own template for how the events get written to disk. For example:
template t_extra_timestamp { template("${R_ISODATE} ${RAWMSG}\n"); };
destination d_juniper { file( "/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log" template(t_extra_timestamp) ); };
Note the R_ variant of the ISODATE macro, this uses the syslog-ng receipt time, rather than the timestamp from the message.
There are other timestamp macros available as well, if you'd prefer a different timestamp format.
The downside of this approach of course is that you can no longer rely on the indexed _time field in splunk for any searches that rely on when the event exactly happened (e.g. for Flow Tracing as you mentioned). Which likely means you also can no longer use accelerated data models for that kind of work, as those will be based on _time.
And if you want to correlate events from different devices that are not in the same timezone, using search time extracted timestamps from the event will also not work (unless you come up with some tricky evals to adjust for the time zone offset as observed against the syslog-ng receipt time).
... View more