topic Re: Automatic Lookup and indexed_kv_limit limit reached ? in Getting Data In

Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit — Wed, 23 Sep 2020 11:01:42 GMT

Hello,

I'm looking for details on indexed_kv_limit parameter following an upgrade from 7.x to 8.x.

After an upgrade, I saw a warning message from my indexers saying :

The search you ran returned a number of fields that exceeded the current indexed field extraction limit. To ensure that all fields are extracted for search, set limits.conf: [kv] / indexed_kv_limit to a number that is higher than the number of fields contained in the files that you index.

When I inspect the job, I noticed many lookup are loaded by AutoLookupDriver not related to my sourcetype. These lookup are configured by various TA (palo alto, cisco et c....).

After, the lookup loading, I noticed more than 400 fields appears in Final required field list.

Are Fields required related to lookup loaded ? (from my understanding : yes)

However, I don't understand why the search doesn't limit the lookup to the sourcetype of the logs. Is-it possible to limit these loading ?

Regards.

Re: Automatic Lookup and indexed_kv_limit limit reached ?

thambisetty — Wed, 23 Sep 2020 11:32:03 GMT

Automatic lookups are configured based on sourcetype. They are applied only to data which is matching with their configured sourcetype.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Limitsconf

[kv]

avg_extractor_time = <integer>
* Maximum amount of CPU time, in milliseconds, that the average (over search
  results) execution time of a key-value pair extractor will be allowed to take
  before warning. Once the average becomes larger than this amount of time a
  warning will be issued
* Default: 500 (.5 seconds)

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have data with a large
  number of columns and want to ensure that searches display all fields extracted
  from an automatic key-value field (auto kv) configuration.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time and search time.
* Default: 100

indexed_kv_limit = <integer>
* The maximum number of fields that can be extracted at index time from a data source.
* Fields that can be extracted at index time include default fields, custom fields,
  and structured data header fields.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype', 'linecount',
  'splunk_server', and 'splunk_server_group' do not count against this limit and are
  always returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time.
* Default: 200

Re: Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit — Wed, 23 Sep 2020 11:49:27 GMT

I'm aware of limits.conf file, however, i would like to try to resolve my lookup issue instead of rising the limit.

When, I inspect the search.log file, I noticed all lookup are loaded :

09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-action 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-dest_interface_name 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-facility_categories 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-icmp_code 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-messages 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-severity 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-src_interface_name 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_analytics_traps 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_aperture 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config_traps 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_hipmatch 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_system_traps 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_threat_traps 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_traps4 09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_traffic_action

Re: Automatic Lookup and indexed_kv_limit limit reached ?

thambisetty — Wed, 23 Sep 2020 11:58:37 GMT

I have automatic lookups but I am not seeing AutoLookupDriver. But I am seeing like below:

09-23-2020 11:57:17.664 INFO  CsvDataProvider - Reading schema for lookup table='identity_lookup_default_fields', file size=21, modtime=1572594789

Re: Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit — Wed, 23 Sep 2020 12:34:19 GMT

After looking into my other indexes, i noticed the same lookup are always selected but the number of command.search.kv doesn't seem linked (and so the limit indexed_kv_limit is not triggered).

I will continue my debug :).

Re: Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit — Wed, 23 Sep 2020 13:40:21 GMT

I checked release notes and saw a known issue in 8.0.3 :

2020-04-24

SPL-186424, SPL-185211

indexed_kv_limit related warning messages

I'm in 8.0.6, so normally, i'm not affected by this issue, but for the moment, didn't see anything else which could explain my issue.

Re: Automatic Lookup and indexed_kv_limit limit reached ?

_varied — Tue, 26 Jan 2021 00:32:12 GMT

@Rodelanuit did you manage to resolve the lookup issue?