Splunk Search

Why is my search head sorting columns when displaying statistics/visualizations?

gf13579
Communicator

We have two search heads, one for general use and one for Enterprise Security.

Any table/stats searches on the ES search head appear with the columns sorted alphanumerically — resulting in _time as the last field if you're doing a timechart. This then messes up line charts etc. that expect _time to be the first column (i.e. the x axis).

The sorting happens as the search head finalizes search results — if you run a long search you'll see columns {_time, b, c, a} then as the search finalizes, you'll see the browser rearrange them into {a, b, c, _time}. Viewing that as a line chart is then a disaster.

Has anyone seen this before?

0 Karma
1 Solution

gf13579
Communicator

It turned out that limits.conf was the problem, weirdly.

There was a complete - and possibly modified - copy of the 6.6.2 defaults/limits.conf placed in local/. After removing this the upgraded environment (7.2.1) worked fine - columns were sorted as expected, preventing line charts of timechart data from appearing incorrectly.

View solution in original post

0 Karma

gf13579
Communicator

It turned out that limits.conf was the problem, weirdly.

There was a complete - and possibly modified - copy of the 6.6.2 defaults/limits.conf placed in local/. After removing this the upgraded environment (7.2.1) worked fine - columns were sorted as expected, preventing line charts of timechart data from appearing incorrectly.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

sorting columns
Yep, i also faced that, at sometimes, splunk developers sorted some columns as per their coding practice(but end users may be facing some troubles).
Core Developers and Software Testing/End Users can never agree at some strange issues like this, often. It happens 😉

0 Karma

skalliger
Motivator

I simply don't want any developers to messs with standard Splunk command behaviours, that's quite simple.
So, we need to check our Apps installed. Guess I have to check all configs from the lesser known ones of our environment. Thanks for the hint.

Skalli

0 Karma

skalliger
Motivator

Oh, wow. I am not the only one who noticed this weird behaviour. For me it seems like a strange bug. You can even reproduce this when using

search ...
| table a, b,c

which should sort the columns in the mentioned order. Instead, it just doesn't.

What ES version are you running? I noticed this behaviour on version 5.1.0. After 5.2.0, the issue was gone. Try upgrading if you can.

Skalli

0 Karma

gf13579
Communicator

ES 5.2.x so sadly it looks unrelated.

0 Karma

skalliger
Motivator

And what Splunk Enterprise version are you using?

0 Karma

gf13579
Communicator

7.2.0 on both Search Heads though we've just upgraded both to 7.2.1 and the ES SH is still showing the same frustrating behaviour.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...