Why am I unable to parse logs that are bigger than 10 KB in size?

[PCIIT]

Hi All ,

We are using Splunk 6.6.6 version. Whenever we run a query with the log size of each event more than 10 KB in size, we are unable to parse it. We analyzed our search.log and found the following warnings.

11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - buildRegexList provided empty conf key, ignoring.
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.480 INFO  UserManager - Unwound user context: admin -> NULL

[pruthvikrishnap]

I Think its not the issue with event size, below setting is always set to default unless you specify, which controls the event size.sendEventMaxSize =
* The maximum size, in bytes, that an fschange event can be for the input to
send the full event to be indexed.
* Limits the size of event data that the fschange input sends.
* This limits the size of indexed file data.
* Default: -1 (unlimited).

[pragycho]

where need to set this value sendEventMaxSize ?

[martin_mueller]

Those messages are unrelated.

You can change the limit of how much raw data autokv uses in limits.conf:

[kv]
maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

[martin_mueller]

Are your events 50mb in size, or is your file containing many events 50mb in size? The maxchars setting applies to event size, not file size.

[martin_mueller]

That depends on your data.

[pragycho]

using 2 log file and total size is 50000 KB . what is ideal value for [kv] ?

[martin_mueller]

10240 is the default, not going to change anything by setting that.

[PCIIT]

could you please suggest me ?what is correct value need to set ?
[kv]
maxchars = 20480 --->ok or need to set high value

[PCIIT]

i added in limit.conf
[kv]
maxchars = 10240

but still same issue

[PCIIT]

what is maximum value for maxchars ?