Why is my search head sorting columns when displaying statistics/visualizations?

[gf13579]

We have two search heads, one for general use and one for Enterprise Security.

Any table/stats searches on the ES search head appear with the columns sorted alphanumerically — resulting in _time as the last field if you're doing a timechart. This then messes up line charts etc. that expect _time to be the first column (i.e. the x axis).

The sorting happens as the search head finalizes search results — if you run a long search you'll see columns {_time, b, c, a} then as the search finalizes, you'll see the browser rearrange them into {a, b, c, _time}. Viewing that as a line chart is then a disaster.

Has anyone seen this before?

[gf13579 · Accepted solution]

It turned out that limits.conf was the problem, weirdly.

There was a complete - and possibly modified - copy of the 6.6.2 defaults/limits.conf placed in local/. After removing this the upgraded environment (7.2.1) worked fine - columns were sorted as expected, preventing line charts of timechart data from appearing incorrectly.

[inventsekar]

sorting columns
Yep, i also faced that, at sometimes, splunk developers sorted some columns as per their coding practice(but end users may be facing some troubles).
Core Developers and Software Testing/End Users can never agree at some strange issues like this, often. It happens 😉

[skalliger]

I simply don't want any developers to messs with standard Splunk command behaviours, that's quite simple.
So, we need to check our Apps installed. Guess I have to check all configs from the lesser known ones of our environment. Thanks for the hint.

Skalli

[skalliger]

Oh, wow. I am not the only one who noticed this weird behaviour. For me it seems like a strange bug. You can even reproduce this when using

search ...
| table a, b,c

which should sort the columns in the mentioned order. Instead, it just doesn't.

What ES version are you running? I noticed this behaviour on version 5.1.0. After 5.2.0, the issue was gone. Try upgrading if you can.

Skalli

[gf13579]

ES 5.2.x so sadly it looks unrelated.

[skalliger]

And what Splunk Enterprise version are you using?

[gf13579]

7.2.0 on both Search Heads though we've just upgraded both to 7.2.1 and the ES SH is still showing the same frustrating behaviour.