Timestamp Issue with Splunk Add-on for McAfee

[rsolutions]

I have followed the implementation as per the documentation and can see the EPO query builds the timestamp properly at the beginning of the log, but for some reason the indexer is not picking up the appropriate timestamp of the event. Instead, the time that Splunk indexes the log entry is being used for the timestamp.

Timestamp at the beginning of the raw event:  2014-07-16 17:47:33
Timestamp Splunk is using:  7/16/14 11:55:00.000 AM

props.conf in the TA for the sourcetype:
[mcafee:epo]
  SHOULD_LINEMERGE=false
  LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+
  MAX_TIMESTAMP_LOOKAHEAD=1
  TIME_FORMAT=%Y-%m-%d %H:%M:%S
  TZ=UTC

I must be missing something... any ideas?

[somesoni2 · Accepted solution]

This setting worked for me (with sample data imported to index). It was showing correct time extraction without TZ = UTC in preview but was defaulting to system time when got indexed. After adding TX = UTC (spaces before and after 😃 it showed correct time in both preview and indexed data.

[mcafee]
BREAK_ONLY_BEFORE = ([\r\n]*)\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+
MAX_TIMESTAMP_LOOKAHEAD = 20
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:%S
pulldown_type = 1
TZ = UTC

[rsolutions]

This worked, though I will forward this along to Splunk to see if they can integrate it into the TA. I commented out the LINE_BREAKER and used the above BREAK_ONLY_BEFORE line above in a props.conf saved in the local directory of the Splunk Add-on for McAfee app.

[rsolutions]

The Splunk Add-on for McAfee is installed on both the Search Head and the Indexer. The DB Connect is only installed on the Search Head. Here are a few sample logs:

2014-07-16 17:54:30 AutoID=389732 signature="Common Standard Protection:Prevent termination of McAfee processes" threat_type="access protection" signature_id=1092 category=hip.file severity_id=5 event_description="Access Protection rule violation detected and blocked" detected_timestamp=1405554869.000 file_name="C:\\Program Files (x86)\\McAfee\\VirusScan Enterprise\\x64\\scan64.exe" detection_method=OAS vendor_action="deny terminate" threat_handled=true logon_user="NT AUTHORITY\\SYSTEM" user=freddie3 dest_nt_domain=SLICK dest_dns=HW005029 dest_nt_host=HW005029 fqdn=HW005029.slick.ca dest_ip=10.233.11.61 dest_netmask= dest_mac=fc4dd4d210ab os="Windows 7" sp="Service Pack 1" os_version=6.1 os_build=7601 timezone="Canada Central Standard Time" src_dns=_ src_ip=10.233.52.49 src_mac= process="C:\\WINDOWS\\CCM\\CcmExec.exe" url= logon_user_1= is_laptop=0 product="VirusScan Enterprise" product_version=8.8 engine_version= dat_version= vse_dat_version=7500.0000 vse_engine64_version=5600.1067 vse_engine_version=5600.1067 vse_hotfix=2 vse_product_version=8.8.0.975.Wrk vse_sp=

2014-07-16 17:52:03 AutoID=389731 signature="Common Standard Protection:Prevent termination of McAfee processes" threat_type="access protection" signature_id=1092 category=hip.file severity_id=5 event_description="Access Protection rule violation detected and blocked" detected_timestamp=1405554721.000 file_name="C:\\Program Files (x86)\\McAfee\\VirusScan Enterprise\\x64\\scan64.exe" detection_method=OAS vendor_action="deny terminate" threat_handled=true logon_user="NT AUTHORITY\\SYSTEM" user=N/A dest_nt_domain=SLICK dest_dns=HW005193 dest_nt_host=HW005193 fqdn=HW005193.slick.ca dest_ip=10.233.11.100 dest_netmask= dest_mac=fc4dd4d3c3ea os="Windows 7" sp="Service Pack 1" os_version=6.1 os_build=7601 timezone="Canada Central Standard Time" src_dns=_ src_ip=10.233.5.108 src_mac= process="C:\\WINDOWS\\CCM\\CcmExec.exe" url= logon_user_1= is_laptop=0 product="VirusScan Enterprise" product_version=8.8 engine_version= dat_version= vse_dat_version=7500.0000 vse_engine64_version=5600.1067 vse_engine_version=5600.1067 vse_hotfix=2 vse_product_version=8.8.0.975.Wrk vse_sp=

2014-07-16 17:49:37 AutoID=389730 signature="Common Standard Protection:Prevent termination of McAfee processes" threat_type="access protection" signature_id=1092 category=hip.file severity_id=5 event_description="Access Protection rule violation detected and blocked" detected_timestamp=1405554576.000 file_name="C:\\Program Files (x86)\\McAfee\\VirusScan Enterprise\\x64\\scan64.exe" detection_method=OAS vendor_action="deny terminate" threat_handled=true logon_user="NT AUTHORITY\\SYSTEM" user=N/A dest_nt_domain=SLICK dest_dns=NSLT04 dest_nt_host=NSLT04 fqdn=NSLT04.slick.ca dest_ip=10.233.15.58 dest_netmask= dest_mac=005056ae0a74 os="Windows 7" sp="Service Pack 1" os_version=6.1 os_build=7601 timezone="Canada Central Standard Time" src_dns=_ src_ip=10.233.15.58 src_mac= process="C:\\WINDOWS\\CCM\\CcmExec.exe" url= logon_user_1= is_laptop=0 product="VirusScan Enterprise" product_version=8.8 engine_version= dat_version= vse_dat_version=7500.0000 vse_engine64_version=5600.1067 vse_engine_version=5600.1067 vse_hotfix=2 vse_product_version=8.8.0.975.Wrk vse_sp=

[chanfoli]

Hello,

Your problem might be this:
MAX_TIMESTAMP_LOOKAHEAD=1

MAX_TIMESTAMP_LOOKAHEAD constrains the timestamp extraction range so it will never find a timestamp if you are constraining it to the first character.

From http://docs.splunk.com/Documentation/Splunk/6.1.2/admin/Propsconf

> MAX_TIMESTAMP_LOOKAHEAD = <integer>
> * Specifies how far (in characters) into an event Splunk should look for a
> timestamp.
> * This constraint to timestamp extraction is applied from the point
> of the TIME_PREFIX-set location.
> * For example, if TIME_PREFIX positions a location 11 characters
> into the event, and   
> MAX_TIMESTAMP_LOOKAHEAD is set to 10,
> timestamp extraction will be
> constrained to characters    11
> through 20.
> * If set to 0, or -1, the length constraint for timestamp recognition
> is   effectively disabled.  This can
> have negative performance implications
> which   scale with the length of input
> lines (or with event size when
> LINE_BREAKER   is redefined for event
> splitting

> ).
> * Defaults to 150 (characters).

[chanfoli]

Interesting, data preview seems to have no problem extracting the timestamp from the sample you provided without any props configuration.

[rsolutions]

I tried MAX_TIMESTAMP_LOOKAHEAD=20 and it doesn't change the behavior.

[chanfoli]

I would think with the above, you would not need much more than:
MAX_TIMESTAMP_LOOKAHEAD=20

and maybe a:
TZ=[event timezone]

-Sean

[somesoni2]

Can you post some sample logs? Also, ensure that this props.conf changes are there on Indexer or heavy forwarder, NOT on universal forwarder.

[rsolutions]

Ignore the errors in the LINE_BREAKER... it is just the way the posting turned out, but it appears correct