This is the first time I'm setting up a summary search and I must be missing something.
If I click "view recent" I can see that my search actually ran at the scheduled time and had a "resultCount" of 4649 so it was successful.
I can see the "search" string ends with
| summaryindex spool=t ... index="my_index" file="somefile.stash_new" ...
So that means it knew it was supposed to index the results.
I have compared the configuration with other summary searches that I know work and I can't really see a difference.
I have logged onto all the indexers and the search head and run the following commands in /opt/splunk:
find . -name *.stash_new
ls var/spool/splunk/
Both return nothing.
I have tracked down the relevant search.log on the search head and haven't seen anything that jumped at me. There's a SummaryIndexProcessor INFO line that says "successfully wrote file to 'xxxxx.stash_new'".
I have checked for "warnings from the spooler" with:
index=_internal source=*splunkd.log "*spool*"
But that returned nothing.
So surely all went well and the stash file got indexed and then deleted... so why is my summary index empty?
I have run out of things to look at. Anybody has an idea?
When creating any index, the index stanza must be added to the indexes.conf on all indexers by way of the cluster master in a clustered environment or manually on each indexer in a non-clustered environment. Assuming, of course, the search head is not indexing locally and forwarding its data to the indexers.
If you have proper permissions and the search head is not indexing locally, you should see an error similar to the one below in _internal or the Messages drop-down within the web UI. The warning would be reported by each indexer that receives the summary index data for an index not configured.
Received event for unconfigured/disabled/deleted index=<-index-> with source="<-source->" host="<-host->" sourcetype="<-sourcetype->". So far received events from 'X' missing index(es).
Managing indexes:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setupmultipleindexes
Peer configuration:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Configurethepeerindexes
I think I might have figured it out...
We have search head forwarding setup, but the index I created doesn't appear to exist on any of the indexers... It looks like I'll have to create it manually (not via the search head's splunk web).
My initial question still holds: how could I have had a clue that this is my problem? Where's the error message?
By default, every SI exists only on the Search Head where the populating search runs. Why are you checking with non-Splunk CLI? Why not just do a search like this on the Search Head for All time
:
index=my_index
That was the first thing I did. Like I said, the index is empty.
Then you should say so; this is a very important detail.
What if you use the collect command instead?
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Collect
Still nothing in the index.
As I tried that, I was repeatitively running "ls" in /opt/splunk/var/spool/splunk/ and for a second there I could see the stash_new file. So it was created successfully, and then something got rid of it.
so when you search for the index are you searching for the index on your indexers or on your search heads?
Do you have forwarding for all enabled on search head? etc.
Does the index exist on the search head?
oh I see you've found you are forwarding therefore you need the indexes on your indexers 😜 Sorry i was late but that's where I was headed next!
The stash files would be available in the Search Head (where the summary index search is created).