topic Re: Maximum output of fieldsummary (or table *) in Splunk Search

Maximum output of fieldsummary (or table *)

FrankSPL — Fri, 10 Nov 2017 15:50:11 GMT

Hi All,

I have a large data set with lots of fields and I want that in a table. However this is not working correctly.
When I test with | fieldsummary I also get strange results.

First I select data

sourcetype=senssordata sensortype="sensortypeA" | fieldsummary

Gives me all fields related to this selection of one sensortype. The majority of the fields are parameter names, for example: PARA1, PARA2, PARA3, etc...

When I transform this in a table by using
sourcetype=senssordata sensortype="sensortypeA" | table *
The same fields are presented in the table.

Now I add more sensortypes to my search

sourcetype=senssordata sensortype IN ("sensortypeA", "sensortypeB","sensortypeC" | fieldsummary
Still no problem. (also the table * works fine)
I receive more PARAx fields because not every sensortype has the same parameter set.

However when I select more sensortypes it start to go wrong

sourcetype=senssordata sensortype IN ("sensortypeA", ....."sensortypeZ" | fieldsummary

Some PARA values that were in the initial fieldsummary outputs are gone.

The table output also fails:
sourcetype=senssordata sensortype IN ("sensortypeA", ....."sensortypeZ" | table *
The fields presented in the fieldsummary are in the table but it is also incomplete.

Is it possible that both fieldsummary and table* can retrieve a maximum number of extracted fields. It looks like a limit around 100 fields. And can this limit be adjusted?

[EDIT] After some extra testing,
Again with sourcetype=senssordata sensortype IN ("sensortypeA", ....."sensortypeZ" | fieldsummary

I know for sure that I'm missing fields. For example PARA2 is missing in this fieldsummary (as it is available when the search is limited to only a few sensortypes)

However, when I adjust the query to
sourcetype=senssordata sensortype IN ("sensortypeA", ....."sensortypeZ" | fields PARA2 | fieldsummary
Then PARA2 is back again (and only PARA2 because I did not select more)

Re: Maximum output of fieldsummary (or table *)

DalJeanis — Fri, 10 Nov 2017 17:01:39 GMT

Try something like this...

 | rename _time as time 
 | fields - _*
 | rename time as _time 
 | fields *

The rename lines are only required if you have a _time value you want to retain.

For table http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Table

Truncated results
The table command truncates the number of results returned based on settings in the limits.conf file. In the [search] stanza, if the value for the truncate_report parameter is 1, the number of results returned is truncated.

The number of results is controlled by the max_count parameter in the [search] stanza. If truncate_report is set to 0, the max_count parameter is not applied.

Re: Maximum output of fieldsummary (or table *)

FrankSPL — Tue, 29 Sep 2020 16:45:50 GMT

Thanks for your response however it does not lead to an answer.
I did try the rename _time part but that does not seem to make much difference.

The limits parts seems interesting but if I read it correctly the max_count is NOT applied when truncate_report is set to 0, which seems to be the default setting. There is no custom limits.conf that changed this default so I assume this limit is not the one I'm hitting.

Re: Maximum output of fieldsummary (or table *)

FrankSPL — Sun, 12 Nov 2017 09:34:27 GMT

I found out myself! I was hitting the kv limit of 100 (max 100 automated field extractions)

limit.conf
[kv]
limit = 1000

For info:
I did some more testing
sourcetype=senssordata sensortype= "sensortypeA" | fieldsummaryo
versus
sourcetype=senssordata sensortype= "*" | where sensortype= "sensortypeA" | fieldsummary
does also give a different output.

On step back.
sourcetype=senssordata sensortype= "sensortypeA"
versus
sourcetype=senssordata sensortype= "*" | where sensortype= "sensortypeA"

Also lists a different list of automated field extractions!

At the end I fount out the defaut behaviour of splunk is to limit the amount of automated field extractions to 100 !

I adjusted this in the limits.conf to 1000 and now everything seems to work.
All different behavior can be explained by selection various sensortypes and hitting the 100 sooner or later. At the end I have +/- 550 different parameter types / field extractions.