topic Re: query in fast mode showing incorrect data and generate wrong result in chart in Splunk Search

query in fast mode showing incorrect data and generate wrong result in chart

shreyans — Fri, 23 Jan 2015 14:51:54 GMT

Hi,

I have two fields in events, with which I am preparing line chart
both the fields information
1. All the events have both these fields
2. Both the fields have some value in it (It's non-empty and Non NULL)
-- readIOps has numbers
-- cLabel has alphanumeric value (with which I have to group by - this column has 2 distinct values)

Line chart query looks like
index=indexName sourcetype=somessourcetype | timechart mean(readIOps) AS reads by cLabel

Since this query is for chart it by default runs on fast mode and running this query on fast mode shows wrong data
(In line chart, It shows one line with name "NULL" and result I notice was clubbing two distinct values of cLabel)

When I open this query in search, it produce wrong table (with one NULL Column) and it was showing fast mode in search mode
"changing fast mode to verbose mode shows correct result in the search" again reverting back to fast mode shows issue

I did not find any way to force chart search on verbose mode
so I made a fix by changing chart query as below, which produce correct result in chart as well as in fast mode searches
index=indexName sourcetype=somessourcetype cLabel=* | timechart mean(readIOps) AS reads by cLabel

See that cLabel is not null and not empty so ideally cLabel=* should not have any impact in event fetching

My question is, Is this a splunk bug because
fast mode, smart mode and verbose mode ideally should show same results for chart?

Is there any guideline for this kind of issues or chart preparation to avoid such issues

Thanks and Best Regards
- Shreyans Soni

Re: query in fast mode showing incorrect data and generate wrong result in chart

shreyans — Fri, 23 Jan 2015 14:57:35 GMT

this looks like similar issue in advanced xml at
http://answers.splunk.com/answers/177962/how-to-enable-verbose-mode-as-default-in-advanced.html

but unanswered

Re: query in fast mode showing incorrect data and generate wrong result in chart

acharlieh — Fri, 23 Jan 2015 15:37:46 GMT

Yes it's unexpected, But if you think of your search as a series of steps, the next step can only operate what's given to it by the previous step. Therefore extractions for cLabel are only useful to timechart if they are extracted by the base search. Now part of how fast mode works is that it only attempts to extract fields as needed ( see the doc ). To successfully search for all events where index=x sourcetype=y Splunk needs to do no extractions as index and sourcetype are default fields. However adding the condition that the cLabel field exists, Splunk now needs to do the extraction of cLabel in order to determine if a result meets this additional criteria. (Remember, Splunk is schemaless, and as a result most fields are extracted at search time only. While you know that the field is always filled in, that is something Splunk needs to see for itself)

Re: query in fast mode showing incorrect data and generate wrong result in chart

sideview — Mon, 26 Jan 2015 20:16:50 GMT

Can you confirm that the two fields are in all cases non-null? by running this search? You kinda stated this already I know, but I'm just double checking. Definitely sounds like a bug - I'm just trying to narrow it down.

index=indexName sourcetype=somessourcetype | fillnull cLabel readIOps value="NO VALUE" | stats count by cLabel readIOps | search cLabel="NO VALUE" OR readIOps ="NO VALUE"

Re: query in fast mode showing incorrect data and generate wrong result in chart

sideview — Mon, 26 Jan 2015 20:21:23 GMT

However there's two problems here and this answer doesn't really address either. a) If all the events have values for both fields, (and the field extractions are working as expected) then NULL should be impossible. b) The search language optimization around extracting only the necessary fields, (ie one part of what "fast mode" does) should absolutely see both the "mean(readIOps)" and also the "by cLabel", and it should just based on that know that extractions on those fields are required. If you have the fields referenced in the search like that, there should be no need to additionally reference them in the search clause as a workaround.

Re: query in fast mode showing incorrect data and generate wrong result in chart

shreyans — Tue, 27 Jan 2015 05:42:38 GMT

Hi, Thanks for looking into it

I have executed above query
index=indexName sourcetype=somessourcetype | fillnull cLabel readIOps value="NO VALUE" | stats count by cLabel readIOps | search cLabel="NO VALUE" OR readIOps ="NO VALUE"

and verify that 145000 events have been filtered in Events tab and 0 in statistics tab. This say no events with cLabel and readIOps with value "NO VALUE" and there by no null in actual events

Let me know if you need any other info to debug it further

Re: query in fast mode showing incorrect data and generate wrong result in chart

shreyans — Tue, 27 Jan 2015 06:36:48 GMT

Hi Acharlieh,

thanks for your time and looking into this

As you said "search as a series of steps" then see that my query is below
index=indexName sourcetype=somessourcetype | timechart mean(readIOps) AS reads by cLabel

that means I am extracting complete event with index=indexName and sourceType=somesourcetype and passing the same to timechart (I am not extracting few columns using table command and passing it to timechart) so Ideally timechart should get complete event and adding to that I am using readIOps and cLabel column in timechart thereby if timechart receives complete event then field extraction should happen at timechart level command on complete event.

In a way you are right that if I explicitly extract field then search result is correct means below query worked fine
index=indexName sourcetype=somessourcetype cLabel=* | timechart mean(readIOps) AS reads by cLabel

but if you think of other splunk search commands then that is not how entire splunk search queries behavior is
For example
index=indexName sourcetype=somessourcetype
executing above command does not mean that after executing command you can only use index and sourcetype fields. you still have choice of using other fields which meet above condition

Re: query in fast mode showing incorrect data and generate wrong result in chart

shreyans — Tue, 27 Jan 2015 06:37:11 GMT

I probably am very beginner in splunk but as a splunk app developer, Ideally "Verbose mode and fast mode should produce same result" and if not by xyz reason then I am certainly looking for a way to pass search mode from my dashboard chart to explicitly search on verbose mode.
adding to that we have at least two report of same issue already on answers.splunk.com and even that developer has fixed the issue like me (explicitly extracting field)

If whatever I said make sense and worth investigating then can you guys please file a splunk bug here and let me know if I can continue with my query with cLabel=* ? and when would the fix be available

please correct me if I am making any mistake here. You guys have better splunk understanding and insight than me

Thanks and Best Regards
- Shreyans Soni