Why are we seeing an additional date field created in indexed data?

[yanivdutt]

Hi,

We are seeing data different in splunk than actual logs.

Switch - di1-c-qcy 10.xxx.xxx.xx below is data from n/w device

2015 Apr 13 12:58:38.864 di1-c-qcy-new %ARP-4-INVAL_IP:  arp [9746]  Received packet with invalid source IP address (127.0.0.2) from 7269.e253.0ac8 on Vlan373

below is data from syslog server

Apr 13 12:58:39 10.xxx.xxx.xx 2015 Apr 13 12:58:39.622 PDT: %ARP-4-INVAL_IP:  arp [9746]  Received packet with invalid source IP address (127.0.0.2) from 7269.e253.0ac8 on Vlan373

below is what we are seeing on Splunk

Apr 13 12:58:39 10.153.142.16 2015 Apr 13 12:58:39.622 PDT: %ARP-4-INVAL_IP:  arp [9746]  Received packet with invalid source IP address (127.0.0.2) from 7269.e253.0ac8 on Vlan373" | table _time, host, _raw

[mikaelbje]

If you are receiving the logs with a UDP input on your Splunk server you can add

no_appending_timestamp = true

To that input if you want to get rid of the extra timestamp. I usually don't do that since I trust the clock on the Splunk server more than the clock on the endpoint.

[sideview]

What you're seeing in Splunk is identical to what's in syslog, except of course that syslog is configured to add it's own timestamp followed by the host. This makes sense I think. Presented with two timestamps to choose from, and without any specific config to tell it otherwise, Splunk then picks the first one, which is why the timestamp of the event in Splunk is the one rounded up by syslog, rather than the full precision timestamp with milliseconds from the device.

As to why the event from the device's logs has a timestamp that is almost a second earlier than what you're thinking is the same event in syslog, Is it possible that this simply isn't the same event? That there were two otherwise identical events on the same device within 800ms of eachother?