Hello,
I'm looking for details on indexed_kv_limit parameter following an upgrade from 7.x to 8.x.
After an upgrade, I saw a warning message from my indexers saying :
The search you ran returned a number of fields that exceeded the current indexed field extraction limit. To ensure that all fields are extracted for search, set limits.conf: [kv] / indexed_kv_limit to a number that is higher than the number of fields contained in the files that you index.
When I inspect the job, I noticed many lookup are loaded by AutoLookupDriver not related to my sourcetype. These lookup are configured by various TA (palo alto, cisco et c....).
After, the lookup loading, I noticed more than 400 fields appears in Final required field list.
Are Fields required related to lookup loaded ? (from my understanding : yes)
However, I don't understand why the search doesn't limit the lookup to the sourcetype of the logs. Is-it possible to limit these loading ?
Regards.
@Rodelanuit did you manage to resolve the lookup issue?
I checked release notes and saw a known issue in 8.0.3 :
2020-04-24 | SPL-186424, SPL-185211 | indexed_kv_limit related warning messages |
I'm in 8.0.6, so normally, i'm not affected by this issue, but for the moment, didn't see anything else which could explain my issue.
After looking into my other indexes, i noticed the same lookup are always selected but the number of command.search.kv doesn't seem linked (and so the limit indexed_kv_limit is not triggered).
I will continue my debug :).
I have automatic lookups but I am not seeing AutoLookupDriver. But I am seeing like below:
09-23-2020 11:57:17.664 INFO CsvDataProvider - Reading schema for lookup table='identity_lookup_default_fields', file size=21, modtime=1572594789
I'm aware of limits.conf file, however, i would like to try to resolve my lookup issue instead of rising the limit.
When, I inspect the search.log file, I noticed all lookup are loaded :
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-action
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-dest_interface_name
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-facility_categories
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-icmp_code
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-messages
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-severity
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-cisco_ios-src_interface_name
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_analytics_traps
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_aperture
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_config_traps
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_hipmatch
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_system_traps
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_threat_traps
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_info_for_pan_traps4
09-23-2020 13:45:43.293 INFO AutoLookupDriver - Will use Lookup: LOOKUP-vendor_traffic_action
Automatic lookups are configured based on sourcetype. They are applied only to data which is matching with their configured sourcetype.
https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Limitsconf
[kv]
avg_extractor_time = <integer> * Maximum amount of CPU time, in milliseconds, that the average (over search results) execution time of a key-value pair extractor will be allowed to take before warning. Once the average becomes larger than this amount of time a warning will be issued * Default: 500 (.5 seconds) limit = <integer> * The maximum number of fields that an automatic key-value field extraction (auto kv) can generate at search time. * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype', 'linecount', 'splunk_server', and 'splunk_server_group' do not count against this limit and will always be returned. * Increase this setting if, for example, you have data with a large number of columns and want to ensure that searches display all fields extracted from an automatic key-value field (auto kv) configuration. * Set this value to 0 if you do not want to limit the number of fields that can be extracted at index time and search time. * Default: 100 indexed_kv_limit = <integer> * The maximum number of fields that can be extracted at index time from a data source. * Fields that can be extracted at index time include default fields, custom fields, and structured data header fields. * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype', 'linecount', 'splunk_server', and 'splunk_server_group' do not count against this limit and are always returned. * Increase this setting if, for example, you have indexed data with a large number of columns and want to ensure that searches display all fields from the data. * Set this value to 0 if you do not want to limit the number of fields that can be extracted at index time. * Default: 200