Why is the timestamp showing up in the future on some sourcetypes?

[Hemnaath]

Hi Team, Currently we are having issue for certain sourcetype the indexed events are with the future time stamp. The problem is with Symantec logs source that are forwarded from third party device called tap server to the heavy forwarder and then it gets indexed to the indexer instances.

When we run the below query, we could see that data are being ingested in future time instead of actual Geographic time.

earliest=@d latest=@d+1d sourcetype=symantec:tap*

Props.conf details :

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
TIME_PREFIX=time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0

[symantec:tap:incidentevents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
EVAL-file_name = lower('file.name')
TIME_PREFIX=log_time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0

Note : we had removed " DATETIME_CONFIG = CURRENT" from above props.conf details, so will that be a problem. Kindly guide me how to correct the future time stamp issue.

[richgalloway]

Given TIME_PREFIX=log_time, Splunk is using 2018-01-02T17:13:01.490Z as the event time, but is probably ignoring the time zone specifier, which would explain the 5-hour difference.

Try:

TIME_PREFIX = log_time:\s+
TIME_FORMAT = %Y-%m-%dT%H:%H:%S.%3N%Z

TZ shouldn't be needed since the time zone is given in the timestamp.

[Hemnaath]

Hi richgalloway, thanks for your effort on this, actually I had tried to test this by ingesting the sample data in to the indexer instance via GUI

Settings-->Add Data --> Upload -->select source-->setsourcetype--> "symantec:tap:incidentevents" 
 timestamp --> Advanced 
Timezone --> "Auto"
Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" 
Time stamp Preffix -> log_time:\s+
Lookhead --> 27 

By following the above steps, we could see that the index time is changed to the current time zone but at the same time I am unable to match the log_time filed in the events with the index time.

Example :
_time: 1/4/18 5:26:52.000 AM and log_time: 2018-01-02T17:13:01.490Z field in the events are not matching .

If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z .

Is there a possibility that we can match _time value and log_time in the event field value with the current time zone.

Kindly guide me on this please.

[richgalloway]

If your last statement - "If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z " - is correct then you have found the solution. The two timestamps you give are equivalent. If you need to make them into the exact same format try strftime(_time,"%Y-%m-%dT%H:%M:%S.%3N%Z").

[Hemnaath]

Hey but In that case we are unable to match the _time value and log_time in the event field value with the current time.

Our requirement is that we want _time should have current time and the same time should match the log_time value in the event field. Is there a possibility to do this.

Kindly guide me on this.

thanks in advance.

[richgalloway]

Why do you want the log_time to match the current time? The event happened when it happened and that is not necessary "now". _time is always the event time, unless you replace it with something like eval _time=now() (which you probably don't need to do).

[richgalloway]

It's considered a Best Practice to specify the TIME_FORMAT attribute for every sourcetype in your props.conf file. You may also want to consider using the TZ attribute if the event time stamp does not include a time zone.

[Hemnaath]

Hi richgalloway, thanks for your effort on this, we are getting the below events details after executing the query. Currently system time is 12:37 PM EST whereas _time shows as 5:13:01.490 PM, so how to correct the props.conf to fix this issue.

1/2/18
5:13:01.490 PM  
{   [-] 
     actual_action:  Left alone 
     actual_action_idx:  4  
     agent_version:  14.0.3752.1000 
     atp_host:   10.x.x.x   
     data_source_url_domain:    
     device_ip:  10.x.x.x   
     device_name:    node01 
     device_time:    2018-01-02T17:08:28.000Z   
     device_uid:     d41db0f9-4f98-484d-92b3-ca03e6a33d46   
     domain_name:    test.com   
     external_ip:   
     file:  {   [+] 
    }   
     host_name:  node01 
     incident:   2daf1330-ef75-11e7-fb05-0000000000e1   
     internal_ip:    10.x.x.x   
     local_host_mac:     00-05-9a-3c-7a-00  
     log_name:   epmp_incident-2018-01-02/event 
     log_time:   2018-01-02T17:13:01.490Z   
     no_of_viruses:  1  
     sep_installed:  true   
     source:     Real Time Scan 
     threat:    {   [+] 
    }   
     type_id:    4123   
     user_name:  user1  
     uuid:   2ffcd120-efe0-11e7-f1d4-000000004810   
     virus_def:  2018-01-01 rev. 003    
     virus_name:     XXX.XXXXX  
}

Kindly guide me what TIME_FORMAT should be used based on the above events and shall I include TZ = EDT along with other props.conf stanza details.

[naidusadanala]

Hopefully this document helps

https://wiki.splunk.com/Community:HandlingEventsTimeStampedInTheFuture

[naidusadanala]

Are all the servers are in same timezone ?

[Hemnaath]

Hi naidusadanaia, thanks for your effort all are in the same location. still we could see data being ingested in future time stamp. Kindly guide me to fix the issue.