Splunk Search

Job inspector: How to identify search time extraction is kicking in

splunk_zen
Builder

We may be having performance issues as newly saved search time extractions are not working even after being successfully tested via the Field Extractor

Sample example:

"faQUF","2.3.7","False","2","4","9","1","N-281","PF","19800","India Standard Time","3.8.0.5","2016-11-03T07:19:17.000Z","2016-11-03T10:49:35.000Z","3.8.0.8","/x/api/v2/hosts/fUF","","None","Windows 7 Enterprise","Service Pack 1","64-bit","7x-5x-fx-0x-xx-xx","dcfb"

the following props.conf on were set on the SH

[fireye:hx:asset_inventory]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
EXTRACT-agentId,agentVersion,excluded_from_containment,stats_acqs,stats_alerting_conditions,stats_alerts,stats_exploit_alerts,hostname,domain,gmt_offset_seconds,timezone,src_ip,last_audit_timestamp,last_poll_timestamp,last_poll_ip,url,last_alert_id,last_alert_timstamp,os_product_name,os_patch_level,os_bitness,src_mac,md5 = \"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\"
EXTRACT-agentId = ^"(?P<agentId>[^"]*)

NOTES:

** Search was run on Verbose
** the extraction was tested first as belonging to its owner, and then shared globally
** Both the single EXTRACT-agentId as well as the composed fields one were tested separately, just kept the single one to exemplify even such a simple extraction is not working

Using the job inspector I'm seeing a very quick key value extraction (the 6 invocations may be the 6 default interesting fields Splunk extracts)

Duration (seconds)          Component       Invocations
0.01          command.search.kv     6 

I can only see the expected fields when I use the very same regex as a | rex command

sourcetype = fireye:hx:asset_inventory
| rex field=_raw "\"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\""

and as expected we get the rex command kicking in

Duration (seconds)          Component       Invocations
1.40        command.rex     5,501 

Anything which can point me to why this is broken?

0 Karma

vasanthmss
Motivator

Hope you noticed or missed, while restarting splunk should through some exception on the props.conf malformed or something because of Extract class name? Since your first extract failed after the confs will be ignored.

I would suggest keep the Extract class name as small and meaningful. class name is a unique literal string. So simple change the below,

from

EXTRACT-agentId,agentVersion,excluded_from_containment,stats_acqs,stats_alerting_conditions,stats_alerts,stats_exploit_alerts,hostname,domain,gmt_offset_seconds,timezone,src_ip,last_audit_timestamp,last_poll_timestamp,last_poll_ip,url,last_alert_id,last_alert_timstamp,os_product_name,os_patch_level,os_bitness,src_mac,md5=

To

EXTRACT-sourcefields

Hope this will helps you!!!!

V

splunk_zen
Builder

Thanks vasanthmss
After I restarted with the shorted name it did work
Though not totally convinced it was that as when I rolled back to the previous long EXTRACT- class name it kept working
May have been related to a colleague having suspended the knowledge bundle sync as it had reached a huge size

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...