topic Re: Problem with custom search command vs timechart in results chart in Splunk Search

Problem with custom search command vs timechart in results chart

gcoles — Tue, 30 Apr 2013 01:49:08 GMT

I've written a custom (generating) Splunk command that retrieves data from Carbon/Graphite, a numerical data-logging tool similar to RRD (but better), for the time period specified in the time-picker. The command works as I had hoped, quickly retrieving the data and providing it in columnar format, with _time, _span, and each requesed column of data provided as outputs.

The problem is that when I click the "Results chart" button above the search results, three out of four times the chart doesn't use _time as the x-axis, instead choosing one of the other columns for this, and tries to use _time as the value being plotted. This problem shows up consistently for the same results over a fixed time-period (same number of points, same columns, field values, and _time and _span fields). To validate my work, I wrote another command that dumps the raw field data to a log file, and can't see anything changing in the output of my command between each execution, yet the chart handler behaves inconsistently. I also compared my results to those of timechart with the same span, and they look identical in the raw data dump.

Does anyone of of any magic that timechart does to prepare data for the Chart module, outside of the result fields themselves?

EDIT: The answer was to use the fields= parameter with splunk.Intersplunk.outputResults() to specify column order. I just needed to create a list of the columns, starting with _time and _span, and append the names of each generated column to the list, then provide that as the second argument:

outputResults(results, fields=column_list)

Re: Problem with custom search command vs timechart in results chart

sideview — Tue, 30 Apr 2013 07:01:59 GMT

There are some little arcane things that aren't fields but that are passed down the pipeline, but I don't think the difference is that. I think it's a more mundane weirdness around the custom search command stuff. Can you look at the fieldOrder in the results? A tool such as firebug can allow you to see the http traffic and you can then see the actual search results. I wonder if the fieldorder is inconsistent for some reason. Also does the problem go away if you throw a | table _time _span foo bar baz on the end of your search?

UPDATE: Indeed this was the problem and the answer was to use the optional fields argument to splunk.Intersplunk.outputResults()

Re: Problem with custom search command vs timechart in results chart

gcoles — Tue, 30 Apr 2013 17:57:23 GMT

Thanks sideview, that definitely is the issue. The chart module only works properly when _time is the first field returned. The problem is that the custom command is using the splunk API to return the results as a list of dictionaries (a dict for each row of results), and dictionaries do not allow fieldorder to be set.

Do you (or anybody) know of any way to specify the fieldorder via the API as results are returned, aside from switching to csv output?

Re: Problem with custom search command vs timechart in results chart

gcoles — Tue, 30 Apr 2013 18:09:01 GMT

Aha, I just found the optional fields= parameter for splunk.Intersplunk.outputResults() -- will give that a go to see if I can use it to specify column order.

outputResults(results, messages=None, fields=None, mvdelim='\n', outputfile=<open file '<stdout>', mode 'w'>)

Re: Problem with custom search command vs timechart in results chart

sideview — Tue, 30 Apr 2013 19:34:54 GMT

(see the comments on the question for further details, because this "answer" was originally a comment up there)