Table command losing field names in non-Verbose searches

[howyagoin]

Hi,

One of my users reported a bit of an odd issue that spontaneously developed recently. He's got a very long and complicated query which had worked fine for months, but, for some reason, no longer displays results if he uses a "| table a b c d" towards the end of his search, but only if he doesn't run the command in Verbose search mode.

Every other part of the search works fine in Fast/Smart mode, and the fields he wants to display are visible in the Selected Fields area, and I can even "| fields a b c d" to ensure that only those fields are present, but, the use of the table command returns 0 results.

He can run the search in Verbose mode and it works as expected, but, this is new behaviour to us.

The sourcetype he's using is defined in the app's props.conf, and it's just doing a field extraction of a number of key fields, and our setup is a 6.6.3 Search Head talking to a 6.6.2 index cluster.

Many of the fields he's trying to put into the table are created out of using eventstats, rename, lookups, or evals -- but, from his perspective, this all worked in non-Verbose mode just fine, but something "changed" and now it doesn't.

I realise there's not a lot to go on here, but, I'm curious if this is familiar to anyone else.

Note that I can create a Table in output-format (Raw/List/Table) with the selected fields just fine, and, as I say, the Fields command works fine, so we can modify his alert to still get the same output he was getting last week, but I'm at a loss to understand why everything in his query works except for the |table even though the fields are selected and displayed and can be manipulated just fine in other search modes.

[corerosoc]

I can get this to happen on one record search.
| a few rexes here -----

Then I tried three things to end the query.

| table a b --- > lists a and b
| table a ----> lists a
| table b ----> no results

verbose mode solves this, but I did need this in an alert so then I had to carry values (a*) all through the query just to be able to use b in the end.

[woodcock]

The table command is a finalizing command (maybe that's not the correct term) that causes everything up to that point and thereafter to be pulled up to the Search Head, stopping all map-reduction efficiency on the Indexers. It can be as performance-destructive and RAM-consuming as transaction if used "too soon" (with too many events at that point of the SPL pipeline). What is happening is that you are running out of RAM on your Search Head and Splunk is silently finalizing the search early (just like it does when transaction eats up all the RAM, which is why you should NEVER use transaction other then a quick-and-dirty throw-away search on a small dataset). You should be able to find a warning or error log in index=_* about this, but I have never looked (if not, I am sure if you turn on debug logging it will be there). It works without verbose because the extra baggage of all of those extra unnecessary fields, you have just barely enough RAM, but when you pull those in, you cross the tipping point. One mitigation is to push table as far to the right (pipes-wise) as possible. Another is to switch to using fields instead. If it MUST stay the way that it is, swap table a b c d to fields a b c d | fields - _* | table a b c d and that should help (it will at least make verbose do nothing at that point of the SPL).

[howyagoin]

Great tips - and normally I'd be fully alongside you in nodding in agreement, however in this case, I can't help but think something else is amiss. The dataset can be extremely small, RAM is 512GB and consistently showing well over 100GB free.

Splunk isn't using that much, and, the table command is the very last thing in the pipeline.

Fields works, the Table view-mode works, everything else works, it's just the very last thing in the search pipeline, which was being used so that the email alert being sent out would create a very simple display that was easy to read of just the essential fields for an alert.

I did try your suggestion though, but, sadly, it had no effect. I do wonder if there are limits on the indexers which are being tripped - but I will also start checking debug output. Frustrating one, this.

[woodcock]

You have nothing to lose by opening a support case.