Getting Data In

Automatic Lookup and indexed_kv_limit limit reached ?

Rodelanuit
Explorer

Hello,

I'm looking for details on indexed_kv_limit parameter following an upgrade from 7.x to 8.x.

After an upgrade, I saw a warning message from my indexers saying : 

The search you ran returned a number of fields that exceeded the current indexed field extraction limit. To ensure that all fields are extracted for search, set limits.conf: [kv] / indexed_kv_limit to a number that is higher than the number of fields contained in the files that you index.

When I inspect the job, I noticed many lookup are loaded by AutoLookupDriver not related to my sourcetype. These lookup are configured by various TA (palo alto, cisco et c....).

After, the lookup loading, I noticed more than 400 fields appears in Final required field list.

Are Fields required related to lookup loaded ? (from my understanding : yes)

However, I don't understand why the search doesn't limit the lookup to the sourcetype of the logs. Is-it possible to limit these loading ?

Regards.

 

 

 

 

 

 

 

 

 

 

 

Labels (1)
Tags (2)

_varied
Loves-to-Learn

@Rodelanuit did you manage to resolve the lookup issue?

0 Karma

Rodelanuit
Explorer

I checked release notes and saw a known issue in 8.0.3 :

2020-04-24SPL-186424, SPL-185211indexed_kv_limit related warning messages

 

I'm in 8.0.6, so normally, i'm not affected by this issue, but for the moment, didn't see anything else which could explain my issue.

0 Karma

Rodelanuit
Explorer

After looking into my other indexes, i noticed the same lookup are always selected but the number of  command.search.kv doesn't seem linked (and so the limit indexed_kv_limit is not triggered).

I will continue my debug :).

 

0 Karma

thambisetty
SplunkTrust
SplunkTrust

I have automatic lookups but I am not seeing AutoLookupDriver. But I am seeing like below:

09-23-2020 11:57:17.664 INFO  CsvDataProvider - Reading schema for lookup table='identity_lookup_default_fields', file size=21, modtime=1572594789
————————————
If this helps, give a like below.
0 Karma

Rodelanuit
Explorer

I'm aware of limits.conf file, however, i would like to try to resolve my lookup issue instead of rising the limit.

 

When, I inspect the search.log file, I noticed all lookup are loaded :

09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-action
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-dest_interface_name
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-facility_categories
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-icmp_code
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-messages
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-severity
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-src_interface_name
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_analytics_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_aperture
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_hipmatch
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_system_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_threat_traps
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_traps4
09-23-2020 13:45:43.293 INFO  AutoLookupDriver - Will use Lookup: LOOKUP-vendor_traffic_action
0 Karma

thambisetty
SplunkTrust
SplunkTrust

Automatic lookups are configured based on sourcetype. They are applied only to data which is matching with their configured sourcetype.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Limitsconf

[kv]

avg_extractor_time = <integer>
* Maximum amount of CPU time, in milliseconds, that the average (over search
  results) execution time of a key-value pair extractor will be allowed to take
  before warning. Once the average becomes larger than this amount of time a
  warning will be issued
* Default: 500 (.5 seconds)

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have data with a large
  number of columns and want to ensure that searches display all fields extracted
  from an automatic key-value field (auto kv) configuration.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time and search time.
* Default: 100

indexed_kv_limit = <integer>
* The maximum number of fields that can be extracted at index time from a data source.
* Fields that can be extracted at index time include default fields, custom fields,
  and structured data header fields.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype', 'linecount',
  'splunk_server', and 'splunk_server_group' do not count against this limit and are
  always returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Set this value to 0 if you do not want to limit the number of fields
  that can be extracted at index time.
* Default: 200

 

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...