topic Re: Is there a limit on searchable characters in an event? in Splunk Search

Is there a limit on searchable characters in an event?

t_splunk_d — Fri, 16 Jun 2017 01:59:32 GMT

I am searching on an event with has on an average 25000 - 30000 characters. When I search on the auto extracted fields or regex extracted fields I do not get results for the field value as it is not matching. I think there is a limit for search on the events.

For example:

index=test status=true is returning incorrect match results.
It returns the events if status=true is within first 10000 characters of the event otherwise it does not.

Is there a limit and how this can be changed? Any index specific change or any search keyword can overcome this?

Re: Is there a limit on searchable characters in an event?

MuS — Tue, 29 Sep 2020 14:28:13 GMT

Hi t_splunk_d,

Update after feedback and some more research:

This is a default setting in limits.conf related to the automatic kay value extraction of _raw

maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

After increasing this to a higher number I was able to use KV pairs after 10000 characters.

Another reason could be the event truncation described below.

this is not limit in the search, your data was truncated by Splunk.
Splunk truncates by default events after 10000 bytes or characters, see the docs http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf for more details but here is the important part:

TRUNCATE = <non-negative integer>
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
  otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often
  a sign of garbage data).
* Defaults to 10000 bytes.

to change this, you need to set in props.conf a high truncate value for the source or sourcetype:

 [YourSourceTypeHere]
 TRUNCATE = a higher number than the maximum length of your events

apply this on the parsing instance of Splunk (index or heavy weight forwarder), restart this instance and any new data will no longer be truncated.

Hope this helps ...

cheers, MuS

Re: Is there a limit on searchable characters in an event?

t_splunk_d — Fri, 16 Jun 2017 10:47:33 GMT

It is not the truncation of event, the field/keyword is present in the event but not searchable.
Is there a limitation to search for a keyword if it is located beyond 10000? I consistently see that in my search results that if I search for a field which is located beyond 10000 it is not able to locate it, whereas if the same filed is located before 10000 then it is able to locate. I am sure that there is limitation because when I search for the keyword/field the value is returned truncated. Whereas when i copy the whole event into an editor and search the keyword/field it is present. I see this truncation/no results if the keyword/field values are located beyond 10000.

Re: Is there a limit on searchable characters in an event?

DalJeanis — Fri, 16 Jun 2017 11:26:50 GMT

what version of splunk are you on?

Re: Is there a limit on searchable characters in an event?

t_splunk_d — Fri, 16 Jun 2017 12:31:38 GMT

Splunk Enterprise 6.5.2

Re: Is there a limit on searchable characters in an event?

t_splunk_d — Sat, 17 Jun 2017 21:25:17 GMT

Experts - Any thoughts?
The event is sometimes 20000 or more and appears in the search. But field values does not shows up (if it is above the 10000) or truncates ( when in the 10000 borderline). Is this limitation of splunk?

Re: Is there a limit on searchable characters in an event?

MuS — Tue, 29 Sep 2020 14:28:30 GMT

Hi t_splunk_d, see my updated answer - Can I call myself Expert now 🙂

Re: Is there a limit on searchable characters in an event?

t_splunk_d — Sun, 18 Jun 2017 21:27:30 GMT

Yes!! No doubt you are an Expert! Thank you!