Getting Data In

Why is the timestamp showing up in the future on some sourcetypes?

Hemnaath
Motivator

Hi Team, Currently we are having issue for certain sourcetype the indexed events are with the future time stamp. The problem is with Symantec logs source that are forwarded from third party device called tap server to the heavy forwarder and then it gets indexed to the indexer instances.

When we run the below query, we could see that data are being ingested in future time instead of actual Geographic time.

earliest=@d latest=@d+1d sourcetype=symantec:tap*

Props.conf details :

[symantec:tap:incidents]
SHOULD_LINEMERGE = false
TIME_PREFIX=time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0

[symantec:tap:incidentevents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
EVAL-file_name = lower('file.name')
TIME_PREFIX=log_time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0

Note : we had removed " DATETIME_CONFIG = CURRENT" from above props.conf details, so will that be a problem. Kindly guide me how to correct the future time stamp issue.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Given TIME_PREFIX=log_time, Splunk is using 2018-01-02T17:13:01.490Z as the event time, but is probably ignoring the time zone specifier, which would explain the 5-hour difference.

Try:

TIME_PREFIX = log_time:\s+
TIME_FORMAT = %Y-%m-%dT%H:%H:%S.%3N%Z

TZ shouldn't be needed since the time zone is given in the timestamp.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

Hi richgalloway, thanks for your effort on this, actually I had tried to test this by ingesting the sample data in to the indexer instance via GUI

Settings-->Add Data --> Upload -->select source-->setsourcetype--> "symantec:tap:incidentevents" 
 timestamp --> Advanced 
Timezone --> "Auto"
Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" 
Time stamp Preffix -> log_time:\s+
Lookhead --> 27 

By following the above steps, we could see that the index time is changed to the current time zone but at the same time I am unable to match the log_time filed in the events with the index time.

Example :
_time: 1/4/18 5:26:52.000 AM and log_time: 2018-01-02T17:13:01.490Z field in the events are not matching .

If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z .

Is there a possibility that we can match _time value and log_time in the event field value with the current time zone.

Kindly guide me on this please.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If your last statement - "If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z " - is correct then you have found the solution. The two timestamps you give are equivalent. If you need to make them into the exact same format try strftime(_time,"%Y-%m-%dT%H:%M:%S.%3N%Z").

---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

Hey but In that case we are unable to match the _time value and log_time in the event field value with the current time.

Our requirement is that we want _time should have current time and the same time should match the log_time value in the event field. Is there a possibility to do this.

Kindly guide me on this.

thanks in advance.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why do you want the log_time to match the current time? The event happened when it happened and that is not necessary "now". _time is always the event time, unless you replace it with something like eval _time=now() (which you probably don't need to do).

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's considered a Best Practice to specify the TIME_FORMAT attribute for every sourcetype in your props.conf file. You may also want to consider using the TZ attribute if the event time stamp does not include a time zone.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

Hi richgalloway, thanks for your effort on this, we are getting the below events details after executing the query. Currently system time is 12:37 PM EST whereas _time shows as 5:13:01.490 PM, so how to correct the props.conf to fix this issue.

1/2/18
5:13:01.490 PM  
{   [-] 
     actual_action:  Left alone 
     actual_action_idx:  4  
     agent_version:  14.0.3752.1000 
     atp_host:   10.x.x.x   
     data_source_url_domain:    
     device_ip:  10.x.x.x   
     device_name:    node01 
     device_time:    2018-01-02T17:08:28.000Z   
     device_uid:     d41db0f9-4f98-484d-92b3-ca03e6a33d46   
     domain_name:    test.com   
     external_ip:   
     file:  {   [+] 
    }   
     host_name:  node01 
     incident:   2daf1330-ef75-11e7-fb05-0000000000e1   
     internal_ip:    10.x.x.x   
     local_host_mac:     00-05-9a-3c-7a-00  
     log_name:   epmp_incident-2018-01-02/event 
     log_time:   2018-01-02T17:13:01.490Z   
     no_of_viruses:  1  
     sep_installed:  true   
     source:     Real Time Scan 
     threat:    {   [+] 
    }   
     type_id:    4123   
     user_name:  user1  
     uuid:   2ffcd120-efe0-11e7-f1d4-000000004810   
     virus_def:  2018-01-01 rev. 003    
     virus_name:     XXX.XXXXX  
}

Kindly guide me what TIME_FORMAT should be used based on the above events and shall I include TZ = EDT along with other props.conf stanza details.

0 Karma

naidusadanala
Communicator
0 Karma

naidusadanala
Communicator

Are all the servers are in same timezone ?

0 Karma

Hemnaath
Motivator

Hi naidusadanaia, thanks for your effort all are in the same location. still we could see data being ingested in future time stamp. Kindly guide me to fix the issue.

0 Karma
Get Updates on the Splunk Community!

Edge Processor | New Resiliency Improvements & Support for Additional Data Sources

We are excited to announce several exciting updates for Edge Processor aimed at hardening overall product ...

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...