topic Re: Can I change the stats limit in splunk for the max characters? in Splunk Search

Can I change the stats limit in Splunk for the max characters?

Hugues — Tue, 04 Oct 2022 16:29:21 GMT

hello all,

My problem is I thing Splunk have max character accepted for stats command,

when i perform this search

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"

I see 3 event, and now if I perform this request

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar

I received two len, one was lose

172

6277

if I perform this statistic request :

index="bnc_6261_pr_log_conf" | logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

recived:

Max(Length):29886

perc95(Length):275756

The event I lose have effectively 28973 character, I thing the actual limit is 10 000.... I already change TRUNCATE parameter at 80 000. It for that I can load event with up to 10 000 character....

My question is, Can I change the stats limit in splunk for the max characters ? with which parameter ? and where from the web page ? can be change by non admin and for a specific source ?

Thank for your future help.

Hugues

Re: Can I change the stats limit in splunk for the max characters?

richgalloway — Mon, 03 Oct 2022 17:06:22 GMT

@Hugues wrote:
hello all,
My problem is I thing Splunk have max character accepted for stats command,
when i perform this search
index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"
I see 3 event, and now if I perform this request
index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar
I received two len, one was lose
172
6277

We don't have enough information to say this is a problem. If two message fields have the same length then only two values will be displayed. Use stats list(l) to view all lengths rather than just the unique ones.

if I perform this statistic request :
index="bnc_6261_pr_log_conf" | logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)
recived:
Max(Length):29886
perc95(Length):275756

The event I lose have effectively 28973 character, I thing the actual limit is 10 000.... I already change TRUNCATE parameter at 80 000. It for that I can load event with up to 10 000 character....

You're comparing apples to oranges. The previous query gets the length (only) of the message field whereas this query gets the length of the entire event.

I don't see evidence of event truncation, but if it is happening then there will be messages in splunkd.log saying so. To find them use this search

index=_internal sourcetype=splunkd component=linebreakingprocessor message="truncating*"

My question is, Can I change the stats limit in splunk for the max characters ? with which parameter ? and where from the web page ? can be change by non admin and for a specific source ?

The stats command does not have a character limit. Some stats functions have a limit on the number of results they can return, but that does not appear to apply here.

Re: Can I change the stats limit in splunk for the max characters?

PickleRick — Mon, 03 Oct 2022 17:38:06 GMT

Your thinking is wrong.

Check this example:

| makeresults 
| eval a="a,a,a" 
| eval a=split(a,",")
| mvexpand a
| stats values(a)

It generates three different rows, all containing field a with value "a" but in the end stats values(a) returns only one "a", because it lists only unique values. So if any two of your three events have the same length of message field, you'll get just two results at the end of your stats.

And it has nothing to do with _raw message, or anything else so your further "analysis" is pointless.

Oh, and don't use wildcards at the beginning of search terms. They are extremely inefficient.

Re: Can I change the stats limit in splunk for the max characters?

Hugues — Mon, 03 Oct 2022 19:08:18 GMT

Thanks for your reply, Sorry im a newbie , I try give you much detail possible...

I try give you more information. When I try this request,

>index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"

Output of 3 event with different message content and lenght:

Event 11:31:51/

Event 11:31:16:

Event 11:30:46/

If I count of length of each message, I have only two length in the output, The biggest message count are not there. All message are different , time is different and all 3 have the same logstream name.

>index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar

I create this small exemple for try to explain my problem. I have same problem with a more complexe request.

>index="bnc_6261_pr_log_conf" logStreamName="c36be289-86f8-4406-94ea-6933b26f9767/i-0ee87265c1f2f4fb7/PatchLinux/stdout" | stats list(message) as msg by logStreamName

my list msg for each logstram name lose the biggest message when is bigger of some number of caracters, may be 10 000.

Il try your request ,

>index=_internal sourcetype=splunkd component=linebreakingprocessor message="truncating*"

and this one

>index=_internal sourcetype=splunkd component=linebreakingprocessor

and

>index=_internal sourcetype=splunkd

I have no events

Thanks all ,for your help!

Hugues

Re: Can I change the stats limit in splunk for the max characters?

richgalloway — Mon, 03 Oct 2022 19:52:44 GMT

The screenshots show odd behavior. Please run this query to see what Splunk is setting message lengths to.

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | table l message

That no events are returned from index=_internal sourcetype=splunkd means you don't have access to that index. An admin can run that query for you to determine if there are any event truncations.

Re: Can I change the stats limit in splunk for the max characters?

Hugues — Tue, 04 Oct 2022 13:45:18 GMT

Thanks for your help.

only two event output

not see the event for 11:31:56 with 28973 characters

I have many other exemple with this problem.... It is not the only case.... every time I have a message with more 6000 characters, may be 10 000 , it disapear when I use stats , eval and table command....

thanks for your help. I am a newbie in splunk , your help is precious!

Hugues

Re: Can I change the stats limit in splunk for the max characters?

Hugues — Tue, 04 Oct 2022 14:06:41 GMT

with raw option it is work

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"
| rex field=_raw ".*message\":(?<new_message>.*)"
| eval l = len(new_message)
| stats list(l) as NumberOfCar

give this output

Now how can solve the problem with this request

index="bnc_6261_pr_log_conf" logStreamName="c36be289-86f8-4406-94ea-6933b26f9767/i-0ee87265c1f2f4fb7/PatchLinux/stdout" | stats list(ms) as msg by logStreamName

Hugues

Re: Can I change the stats limit in splunk for the max characters?

richgalloway — Tue, 04 Oct 2022 14:28:17 GMT

Glad you got the first request working. What's the problem with the second one? Does the ms field exist?

Re: Can I change the stats limit in splunk for the max characters?

Hugues — Tue, 04 Oct 2022 16:25:53 GMT

yes it is working if I use a raw option with a regex, all events have message field, This link with the number of characters , some max.

I cannot apply this solution (ex field=_raw ".*message\":(?<new_message>.*)") on this more complexe request

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | stats list(ms) as msg by logStreamName

because I have the same issue I lose the event have message with more 10 000 characters...

Hugues

Re: Can I change the stats limit in splunk for the max characters?

richgalloway — Tue, 04 Oct 2022 17:24:15 GMT

The stats list(foo) command requires the foo field to be extracted beforehand, either automatically or as part of the current query. That's why the query started working after the rex command was added to the previous query - it ensured the message field was present in all events.

Re: Can I change the stats limit in Splunk for the max characters?

johnhuang — Tue, 04 Oct 2022 19:37:59 GMT

For Auto KV (key valued) extraction, Splunk has a default limit of 10240 characters. You can increase that limit by adjusting maxchars value in limits.conf.

maxchars = <integer> * When non-zero, truncate _raw to this size and then do auto KV. * Default: 10240 characters source: https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Limitsconf

Re: Can I change the stats limit in splunk for the max characters?

Hugues — Tue, 04 Oct 2022 20:31:58 GMT

All 3 messages have the field message, Check the screenshot I made ...

and I lose the message 11:31:51...

see screenshot

when I perform this request or other type of request with eval on field message , I lose event 11:31:51 in the list msg, only 2 msg appear in List msg.

if I try easy request index="bnc_6261_pr_log_conf" logStreamName="4a0dc28c-98ce-4f60-b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" AND message="*LONGOS*" , this suppose get out only the message 11:31:51, I get nothing ...

The Event:

output of my search on logStreamName="4a0dc28c-98ce-4f60-b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" AND message="*LONGOS*"

Hugues

Re: Can I change the stats limit in splunk for the max characters?

johnhuang — Tue, 04 Oct 2022 20:38:23 GMT

Check my response on maxchars on KV extraction. It looks like your fields are extracted using KV.

Re: Can I change the stats limit in Splunk for the max characters?

Hugues — Tue, 04 Oct 2022 20:47:24 GMT

Thanks ,

it for all app index or it can be only apply on index="bnc_6261_pr_log_conf" ? im a newbie in splunk and im not sure if I use the right word... This change will effect anybody else in our compagnie all other search on other index?

thanks again

Hugues

Re: Can I change the stats limit in Splunk for the max characters?

Hugues — Thu, 06 Oct 2022 15:58:11 GMT

Hello all, for some reason I need pass trought _raw my message before use stats.

index="bnc_6261_pr_log_conf" logStreamName="*/i-*/config_Ec2_CECIO_Linux/stdout"

| sort 0 _time

| rex field=_raw ".*message\":(?<new_message>.*)" (add my message from _raw in new_message)

| rex mode=sed field=new_message "s/\\\n/\n/g" (I lose my asci format, bring back newline)

| stats list(new_message) as msg by logStreamName Machine_name account

with the _raw , I keep all my events with stats command. Some expert from Splunk , start check why I need to do that ... If they find the raison I will communicate the info with you...

Re: Can I change the stats limit in Splunk for the max characters?

richgalloway — Thu, 06 Oct 2022 17:15:07 GMT

The maxchars setting applies to all indexes.