topic Re: Why am I unable to parse logs that are bigger than 10 KB in size? in Getting Data In

Why am I unable to parse logs that are bigger than 10 KB in size?

PCIIT — Mon, 12 Nov 2018 18:57:00 GMT

Hi All ,

We are using Splunk 6.6.6 version. Whenever we run a query with the log size of each event more than 10 KB in size, we are unable to parse it. We analyzed our search.log and found the following warnings.

11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - buildRegexList provided empty conf key, ignoring.
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_hour is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.475 WARN  SearchOperator:kv - date_mday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_minute is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_month is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_second is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_wday is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_year is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - date_zone is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - host is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - index is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - linecount is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - punct is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - source is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - sourcetype is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - splunk_server_group is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timeendpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.476 WARN  SearchOperator:kv - timestartpos is an indexed field, ignoring TOKENIZER
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.478 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.479 INFO  UserManager - Unwound user context: admin -> NULL
11-12-2018 17:38:11.480 INFO  UserManager - Unwound user context: admin -> NULL

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

martin_mueller — Mon, 12 Nov 2018 23:05:24 GMT

Those messages are unrelated.

You can change the limit of how much raw data autokv uses in limits.conf:

[kv]
maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

PCIIT — Tue, 13 Nov 2018 05:12:44 GMT

i added in limit.conf
[kv]
maxchars = 10240

but still same issue

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

PCIIT — Tue, 13 Nov 2018 05:13:24 GMT

what is maximum value for maxchars ?

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

martin_mueller — Tue, 13 Nov 2018 09:03:34 GMT

10240 is the default, not going to change anything by setting that.

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

PCIIT — Tue, 13 Nov 2018 16:21:26 GMT

could you please suggest me ?what is correct value need to set ?
[kv]
maxchars = 20480 --->ok or need to set high value

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

martin_mueller — Tue, 13 Nov 2018 17:16:24 GMT

That depends on your data.

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

pruthvikrishnap — Tue, 13 Nov 2018 23:41:31 GMT

I Think its not the issue with event size, below setting is always set to default unless you specify, which controls the event size.sendEventMaxSize = * The maximum size, in bytes, that an fschange event can be for the input to send the full event to be indexed. * Limits the size of event data that the fschange input sends. * This limits the size of indexed file data. * Default: -1 (unlimited).

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

pragycho — Wed, 14 Nov 2018 18:25:25 GMT

using 2 log file and total size is 50000 KB . what is ideal value for [kv] ?

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

pragycho — Wed, 14 Nov 2018 18:33:54 GMT

where need to set this value sendEventMaxSize ?

Re: Why am I unable to parse logs that are bigger than 10 KB in size?

martin_mueller — Wed, 14 Nov 2018 21:06:43 GMT

Are your events 50mb in size, or is your file containing many events 50mb in size? The maxchars setting applies to event size, not file size.