Getting Data In

Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit
Explorer

Hello,

I'm looking for details on indexed_kv_limit parameter following an upgrade from 7.x to 8.x.

After an upgrade, I saw a warning message from my indexers saying : 

The search you ran returned a number of fields that exceeded the current indexed field extraction limit. To ensure that all fields are extracted for search, set limits.conf: [kv] / indexed_kv_limit to a number that is higher than the number of fields contained in the files that you index.

When I inspect the job, I noticed many lookup are loaded by AutoLookupDriver not related to my sourcetype. These lookup are configured by various TA (palo alto, cisco et c....).

After, the lookup loading, I noticed more than 400 fields appears in Final required field list.

Are Fields required related to lookup loaded ? (from my understanding : yes)

However, I don't understand why the search doesn't limit the lookup to the sourcetype of the logs. Is-it possible to limit these loading ?

Regards.

 

 

 

 

 

 

 

 

 

 

 

Labels (1)
Tags (2)
0 Karma

_varied
Loves-to-Learn

@Rodelanuit did you manage to resolve the lookup issue?

0 Karma

Rodelanuit
Explorer

I checked release notes and saw a known issue in 8.0.3 :

2020-04-24SPL-186424, SPL-185211indexed_kv_limit related warning messages

 

I'm in 8.0.6, so normally, i'm not affected by this issue, but for the moment, didn't see anything else which could explain my issue.

0 Karma

Rodelanuit
Explorer

After looking into my other indexes, i noticed the same lookup are always selected but the number of  command.search.kv doesn't seem linked (and so the limit indexed_kv_limit is not triggered).

I will continue my debug :).

 

0 Karma

thambisetty
Super Champion

I have automatic lookups but I am not seeing AutoLookupDriver. But I am seeing like below:

09-23-2020 11:57:17.664 INFO  CsvDataProvider - Reading schema for lookup table='identity_lookup_default_fields', file size=21, modtime=1572594789
————————————
If this helps, give a like below.
0 Karma

Rodelanuit
Explorer

I'm aware of limits.conf file, however, i would like to try to resolve my lookup issue instead of rising the limit.

 

When, I inspect the search.log file, I noticed all lookup are loaded :

09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-action
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-dest_interface_name
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-facility_categories
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-icmp_code
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-messages
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-severity
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-src_interface_name
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_analytics_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_aperture
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_hipmatch
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_system_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_threat_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_traps4
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_traffic_action
0 Karma

thambisetty
Super Champion

Automatic lookups are configured based on sourcetype. They are applied only to data which is matching with their configured sourcetype.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Limitsconf

[kv]

avg_extractor_time = <integer>
* Maximum amount of CPU time, in milliseconds, that the average (over search
  results) execution time of a key-value pair extractor will be allowed to take
  before warning. Once the average becomes larger than this amount of time a
  warning will be issued
* Default: 500 (.5 seconds)

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have data with a large
  number of columns and want to ensure that searches display all fields extracted
  from an automatic key-value field (auto kv) configuration.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time and search time.
* Default: 100

indexed_kv_limit = <integer>
* The maximum number of fields that can be extracted at index time from a data source.
* Fields that can be extracted at index time include default fields, custom fields,
  and structured data header fields.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype', 'linecount',
  'splunk_server', and 'splunk_server_group' do not count against this limit and are
  always returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time.
* Default: 200

 

————————————
If this helps, give a like below.
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!