Timout issue with CLI queries?

[kevintelford]

I am running a simple query over a large index via the CLI. My search completes but does not give me the expected results. For example, I may know that the string "foo" occurs in 30 sources, but when I run my search it will only (but consistently) return a subset of those sources.

nohup /opt/splunk/bin/splunk search 'index=my_index | search foo | stats count by source' -preview false -maxout 0 -auth username:password > myresults.out 2>&1 &

I'm wondering if perhaps there is some internal timeout could be causing my search not to return the full set of results?

And yes, a time range would help but I need to search the whole corpus, and all of the "timestamps" are within the short time period it took to index the data.

I am also currently attempting to re-run my queries with preview set to true to see if this will force a different behavior.

Thanks, Kevin

[araitz · Accepted solution]

Try:

index=my_index foo | stats count by source

Your original search told Splunk to return all events in my_index to the search command 'search', and then for 'search' to filter out events with the term 'foo'. You can see how this is inefficient and can lead to inconsistent results if you hit a search and/or disk quota before events from all the searches are returned.

The above search tells Splunk to return all events in my_index that contain the term 'foo', which is going to perform much better than the original search and will make it far less likely that you hit a search and/or disk quota.

[araitz]

It happens! Regarding normalization, it is really hard to guess the user's intentions, so Splunk assumes that they mean what they typed. Still, it seems like a cool research project to put on the list.

[kevintelford]

I feel like such an idiot. I was always under the impression that Splunk would normalize the query for me behind the scenes. I just liked to make my query more verbose for reading purposes.

Ah well, lesson learned. Thanks dude.