Getting Data In

Why does MAX_TIMESTAMP_LOOKAHEAD=1 cause the time the event was indexed to be assigned as the event timestamp?

coleman07
Path Finder

When I load data as described below, the indexed timestamp does not match the timestamp in the event. I finally figured out it had to do with the the MAX_TIMESTAMP_LOOKAHEAD=1 being set in the props.conf file. Basically, the indexed timestamp is the time when the data was indexed. (This file was written by Splunk). When I commented out the MAX_... line, Splunk starts to correctly record
the proper index time. Please explain why that line messes up the data indexing.

From the Splunk_TA_Mcafee add-on, the props.conf file has the following entry:

[source::....mcafee_epo]
sourcetype = mcafee:epo
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+
MAX_TIMESTAMP_LOOKAHEAD=1
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TZ=UTC

# McAfee EPO DB Connect #
[mcafee:epo]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+
MAX_TIMESTAMP_LOOKAHEAD=1
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TZ=UTC

Here is part of the data in the sample:

2010-09-01 22:34:38 AutoID="87760225" signature="none" threat_type="none" signat
ure_id="1038" category="ops.task.end" severity_id="2" event_description="Scan fo
und infected files." received_timestamp="2010-09-01 22:32:24" file_name="None" d
etection_method="(g\xE9r\xE9e) acme OnDemand Scan (VSE 8.7i) (N)" action="none" 
threat_handled="1" logon_user="Syst\xE8me" user="t_charlyc" dest_nt_domain="acme
tech" dest_dns="PARLCHARLYC01" dest_nt_host="PARLCHARLYC01" fqdn="PARLCHARLYC01.
acmetech.net" dest_ip="10.72.5.39" dest_netmask="" dest_mac="0018de7ccd03" os="W
indows 7" sp="" os_version="6.1" os_build="7600" timezone="Paris, Madrid" src_dn
s="None" src_ip="10.72.3.164" src_mac="None" process="None" url="None" logon_use
r="None" is_laptop="1" product="VirusScan Enterprise" product_version="8.7" engi
ne_version="5400.1158" dat_version="5400.1158" vse_dat_version="6091.0000" vse_e
ngine64_version="" vse_engine_version="5400.1158" vse_hotfix="3" vse_product_ver
sion="8.7.0.570.Wrk" vse_sp="" antispyware_version="8.7.0.129
2010-09-01 22:34:28 AutoID="87760224" signature="none" threat_type="none" signat
ure_id="1038" category="ops.task.end" severity_id="2" event_description="Scan fo
und infected files." received_timestamp="2010-09-01 22:30:43" file_name="None" d
etection_method="(managed) acme OnDemand Scan (VSE 8.7i) (N)" action="none" thre
at_handled="1" logon_user="SYSTEM" user="peteh" dest_nt_domain="acmetech" dest_d
ns="LONWPETEH02" dest_nt_host="LONWPETEH02" fqdn="LONWPETEH02.acmetech.net" dest
_ip="10.30.160.39" dest_netmask="" dest_mac="b8ac6fa02447" os="Windows 7" sp="" 
os_version="6.1" os_build="7600" timezone="GMT Standard Time" src_dns="None" src
_ip="10.30.160.39" src_mac="None" process="None" url="None" logon_user="None" is
_laptop="0" product="VirusScan Enterprise" product_version="8.7" engine_version=
"5400.1158" dat_version="5400.1158" vse_dat_version="6091.0000" vse_engine64_ver
sion="5400.1158" vse_engine_version="5400.1158" vse_hotfix="3" vse_product_versi
on="8.7.0.570.Wrk" vse_sp="" antispyware_version="8.7.0.129
Tags (2)
0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Hi @coleman07,
Just remove the line with MAX_TIMESTAMP_LOOKAHEAD and hopefully that will work.
Do not forget to apply debug/refresh after changing props.conf file.!!!!

0 Karma

harsmarvania57
Ultra Champion

Please set MAX_TIMESTAMP_LOOKAHEAD=19 as per your time format given in logs.

jeffland
SplunkTrust
SplunkTrust

It's because that line tells splunk to look for a timestamp at a maximum of 1 characters behind the timestamp prefix (if no prefix is set, this is from the beginning of the event). Instead of removing the line (or commenting it out), you should increase the value to one that fits your data - in your case, you might want to use 20. The default is 150. See here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Configuretimestamprecognition
PS: When the timestamp can't be found (for example because you're only looking 1 character into your event), splunk uses another time to set the _time field for the event (e.g. the time the file was modified).

Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...