Getting Data In

Why am I unable to parse logs that are bigger than 10 KB in size?

PCIIT
New Member

Hi All ,

We are using Splunk 6.6.6 version. Whenever we run a query with the log size of each event more than 10 KB in size, we are unable to parse it. We analyzed our search.log and found the following warnings.

11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - buildRegexList provided empty conf key, ignoring.
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.480 INFO  UserManager - Unwound user context: admin -> NULL
Tags (2)
0 Karma

pruthvikrishnap
Contributor

I Think its not the issue with event size, below setting is always set to default unless you specify, which controls the event size.sendEventMaxSize =
* The maximum size, in bytes, that an fschange event can be for the input to
send the full event to be indexed.
* Limits the size of event data that the fschange input sends.
* This limits the size of indexed file data.
* Default: -1 (unlimited).

0 Karma

pragycho
Loves-to-Learn

where need to set this value sendEventMaxSize ?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Those messages are unrelated.

You can change the limit of how much raw data autokv uses in limits.conf:

[kv]
maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Are your events 50mb in size, or is your file containing many events 50mb in size? The maxchars setting applies to event size, not file size.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That depends on your data.

0 Karma

pragycho
Loves-to-Learn

using 2 log file and total size is 50000 KB . what is ideal value for [kv] ?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

10240 is the default, not going to change anything by setting that.

0 Karma

PCIIT
New Member

could you please suggest me ?what is correct value need to set ?
[kv]
maxchars = 20480 --->ok or need to set high value

0 Karma

PCIIT
New Member

i added in limit.conf
[kv]
maxchars = 10240

but still same issue

0 Karma

PCIIT
New Member

what is maximum value for maxchars ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...