Hi Team, Currently we are having issue for certain sourcetype the indexed events are with the future time stamp. The problem is with Symantec logs source that are forwarded from third party device called tap server to the heavy forwarder and then it gets indexed to the indexer instances.
When we run the below query, we could see that data are being ingested in future time instead of actual Geographic time.
earliest=@d latest=@d+1d sourcetype=symantec:tap*
Props.conf details :
[symantec:tap:incidents]
SHOULD_LINEMERGE = false
TIME_PREFIX=time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0
[symantec:tap:incidentevents]
SHOULD_LINEMERGE = false
FIELDALIAS-event_host = tap_host as event_host
EVAL-file_name = lower('file.name')
TIME_PREFIX=log_time
MAX_TIMESTAMP_LOOKAHEAD=27
KV_MODE = json
TRUNCATE = 0
Note : we had removed " DATETIME_CONFIG = CURRENT" from above props.conf details, so will that be a problem. Kindly guide me how to correct the future time stamp issue.
Given TIME_PREFIX=log_time
, Splunk is using 2018-01-02T17:13:01.490Z as the event time, but is probably ignoring the time zone specifier, which would explain the 5-hour difference.
Try:
TIME_PREFIX = log_time:\s+
TIME_FORMAT = %Y-%m-%dT%H:%H:%S.%3N%Z
TZ
shouldn't be needed since the time zone is given in the timestamp.
Hi richgalloway, thanks for your effort on this, actually I had tried to test this by ingesting the sample data in to the indexer instance via GUI
Settings-->Add Data --> Upload -->select source-->setsourcetype--> "symantec:tap:incidentevents"
timestamp --> Advanced
Timezone --> "Auto"
Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z"
Time stamp Preffix -> log_time:\s+
Lookhead --> 27
By following the above steps, we could see that the index time is changed to the current time zone but at the same time I am unable to match the log_time filed in the events with the index time.
Example :
_time: 1/4/18 5:26:52.000 AM and log_time: 2018-01-02T17:13:01.490Z field in the events are not matching .
If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z .
Is there a possibility that we can match _time value and log_time in the event field value with the current time zone.
Kindly guide me on this please.
If your last statement - "If we remove the Timestampformat -->"%Y-%m-%dT%H:%H:%S.%3N%Z" and make changes Time stamp prefix = log_time then I could see that _time: 1/2/18 5:13:01.490 PM and log_time: 2018-01-02T17:13:01.490Z " - is correct then you have found the solution. The two timestamps you give are equivalent. If you need to make them into the exact same format try strftime(_time,"%Y-%m-%dT%H:%M:%S.%3N%Z")
.
Hey but In that case we are unable to match the _time value and log_time in the event field value with the current time.
Our requirement is that we want _time should have current time and the same time should match the log_time value in the event field. Is there a possibility to do this.
Kindly guide me on this.
thanks in advance.
Why do you want the log_time to match the current time? The event happened when it happened and that is not necessary "now". _time is always the event time, unless you replace it with something like eval _time=now()
(which you probably don't need to do).
It's considered a Best Practice to specify the TIME_FORMAT
attribute for every sourcetype in your props.conf file. You may also want to consider using the TZ
attribute if the event time stamp does not include a time zone.
Hi richgalloway, thanks for your effort on this, we are getting the below events details after executing the query. Currently system time is 12:37 PM EST whereas _time shows as 5:13:01.490 PM, so how to correct the props.conf to fix this issue.
1/2/18
5:13:01.490 PM
{ [-]
actual_action: Left alone
actual_action_idx: 4
agent_version: 14.0.3752.1000
atp_host: 10.x.x.x
data_source_url_domain:
device_ip: 10.x.x.x
device_name: node01
device_time: 2018-01-02T17:08:28.000Z
device_uid: d41db0f9-4f98-484d-92b3-ca03e6a33d46
domain_name: test.com
external_ip:
file: { [+]
}
host_name: node01
incident: 2daf1330-ef75-11e7-fb05-0000000000e1
internal_ip: 10.x.x.x
local_host_mac: 00-05-9a-3c-7a-00
log_name: epmp_incident-2018-01-02/event
log_time: 2018-01-02T17:13:01.490Z
no_of_viruses: 1
sep_installed: true
source: Real Time Scan
threat: { [+]
}
type_id: 4123
user_name: user1
uuid: 2ffcd120-efe0-11e7-f1d4-000000004810
virus_def: 2018-01-01 rev. 003
virus_name: XXX.XXXXX
}
Kindly guide me what TIME_FORMAT should be used based on the above events and shall I include TZ = EDT along with other props.conf stanza details.
Hopefully this document helps
https://wiki.splunk.com/Community:HandlingEventsTimeStampedInTheFuture
Are all the servers are in same timezone ?
Hi naidusadanaia, thanks for your effort all are in the same location. still we could see data being ingested in future time stamp. Kindly guide me to fix the issue.