topic Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored." in Splunk Search

how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

su_kumar — Wed, 30 Sep 2020 00:24:40 GMT

Hi,

I am using the stats command with the list() function. , i am getting below error.

Error :
'stats' command: limit for values of field 'xxx' reached. Some values may have been truncated or ignored.

WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored.
ERROR SearchParser - Missing a search command before '|'. Error at position '2086' of search query '| tstats count AS count sum(Web_Access_Event.bytes...{snipped} {errorcontext = main | |dedup user}'.

i have configured limit.conf
[stats]
list_maxsize = 10000
maxresultrows = 50000
maxvalues = 10000
maxvaluesize = 10000
Unfortunately , after setting in limit.conf , unable to fix this issue.
anyone help me on this issue

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

codebuilder — Thu, 02 May 2019 18:52:19 GMT

Did you cycle Splunk after modifying limits.conf?

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

su_kumar — Fri, 03 May 2019 03:50:09 GMT

if you talking about after modify limits.conf , need to restart limilts.conf so after modify limits.conf , i had restarted splunk

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

koshyk — Fri, 03 May 2019 06:30:29 GMT

can u please put your SPL which has stats function?
I don't think it is a limits.conf issue as there might be improvement scope in the SPL

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

DavidHourani — Fri, 03 May 2019 07:03:41 GMT

Hi there,

Try this :

[stats]
list_maxsize = 10000
maxresultrows = 50000
maxvalues = 10000
maxvaluesize = 10000

From here :
https://answers.splunk.com/answers/132521/stats-command-limit-for-values-of-field-xxx-reached-some-values-may-have-been-truncated-or-ignored.html

Seems like they have the same issue.

Cheers,
David

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

codebuilder — Fri, 03 May 2019 16:01:25 GMT

You have an errant pipe in your search between main and dedup:

ERROR SearchParser - Missing a search command before '|'. Error at position '2086' of search query '| tstats count AS count sum(Web_Access_Event.bytes...{snipped} {errorcontext = main | |dedup user}'.

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

su_kumar — Sat, 04 May 2019 15:33:19 GMT

<query> 
|dedup user_id  |stats list("user_id") as User dc(user_id) as Total_No_User sum(bytes_in)  as Total_Bandwidth by  group | eventstats sum(bytes_in) as Total_Bandwidth by group | rename group AS "AD Group"
</query>
    </search>

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

su_kumar — Sat, 04 May 2019 15:37:18 GMT

Hi ,
below solution is not working :
[stats]
list_maxsize = 10000
maxresultrows = 50000
maxvalues = 10000
maxvaluesize = 10000

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

DavidHourani — Sat, 04 May 2019 15:39:40 GMT

Oh, if this is your query then you need to remove the pipe from in front of dedup and instead go for values function not the list function 😄

 |stats values("user_id") as User dc(user_id) as Total_No_User sum(bytes_in)  as Total_Bandwidth by  group | eventstats sum(bytes_in) as Total_Bandwidth by group | rename group AS "AD Group"

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

su_kumar — Wed, 08 May 2019 09:38:40 GMT

I have removed pipe but still see errror and not able to see last column duration value

latest query:

dedup user_id | eval duration = round(duration,2) | eval duration=tostring(duration,"duration") | sort group,user_id | where bytes_in >0 |stats list("user_id") as User,list("dest_domain") as Application,list("bytes_in") as Bandwidth_used, list("duration") as Time by group
| rename group AS "AD Group"
</query>

warning :
19 08:46:33.559 -0700 WARN StatsProcessor - Specified field(s) missing from results: 'duration'
05-08-2019 08:46:33.890 -0700 WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored.
05-08-2019 08:46:34.153 -0700 WARN StatsProcessor - Specified field(s) missing from results: 'duration'
05-08-2019 08:46:36.159 -0700 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
05-08-2019 08:46:36.182 -0700 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

DavidHourani — Wed, 08 May 2019 09:49:45 GMT

@su_kumar did this work for you using values instead of list?

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

koshyk — Wed, 08 May 2019 10:43:01 GMT

can you please try changing your query to

| stats sum(bytes_in) as Total_Bandwidth_User_group dc(user_id) as Total_No_User  by user_id, group
| eventstats sum(Total_Bandwidth_User_group) as Total_Bandwidth by group 
| rename group AS "AD Group"

Re: how to avoid this error "WARN StatsProcessor - 'stats' command: limit for values of field 'user_id' reached. Some values may have been truncated or ignored."

DavidHourani — Wed, 08 May 2019 13:59:05 GMT

Replace with the new query I posted here :

|stats values("user_id") as User dc(user_id) as Total_No_User sum(bytes_in)  as Total_Bandwidth by  group | eventstats sum(bytes_in) as Total_Bandwidth by group | rename group AS "AD Group"