Knowledge Management

Summary search runs, has results, yet summary index remains empty

Contributor

This is the first time I'm setting up a summary search and I must be missing something.

If I click "view recent" I can see that my search actually ran at the scheduled time and had a "resultCount" of 4649 so it was successful.
I can see the "search" string ends with

| summaryindex spool=t ... index="my_index" file="somefile.stash_new" ...

So that means it knew it was supposed to index the results.

I have compared the configuration with other summary searches that I know work and I can't really see a difference.

I have logged onto all the indexers and the search head and run the following commands in /opt/splunk:

find . -name *.stash_new
ls var/spool/splunk/

Both return nothing.

I have tracked down the relevant search.log on the search head and haven't seen anything that jumped at me. There's a SummaryIndexProcessor INFO line that says "successfully wrote file to 'xxxxx.stash_new'".

I have checked for "warnings from the spooler" with:

index=_internal source=*splunkd.log "*spool*"

But that returned nothing.

So surely all went well and the stash file got indexed and then deleted... so why is my summary index empty?

I have run out of things to look at. Anybody has an idea?

0 Karma

Engager

When creating any index, the index stanza must be added to the indexes.conf on all indexers by way of the cluster master in a clustered environment or manually on each indexer in a non-clustered environment. Assuming, of course, the search head is not indexing locally and forwarding its data to the indexers.

If you have proper permissions and the search head is not indexing locally, you should see an error similar to the one below in _internal or the Messages drop-down within the web UI. The warning would be reported by each indexer that receives the summary index data for an index not configured.

Received event for unconfigured/disabled/deleted index=<-index-> with source="<-source->" host="<-host->" sourcetype="<-sourcetype->". So far received events from 'X' missing index(es).

Managing indexes:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setupmultipleindexes

Peer configuration:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Configurethepeerindexes

Contributor

I think I might have figured it out...

We have search head forwarding setup, but the index I created doesn't appear to exist on any of the indexers... It looks like I'll have to create it manually (not via the search head's splunk web).

My initial question still holds: how could I have had a clue that this is my problem? Where's the error message?

0 Karma

Esteemed Legend

By default, every SI exists only on the Search Head where the populating search runs. Why are you checking with non-Splunk CLI? Why not just do a search like this on the Search Head for All time:

index=my_index
0 Karma

Contributor

That was the first thing I did. Like I said, the index is empty.

0 Karma

Esteemed Legend

Then you should say so; this is a very important detail.

0 Karma

SplunkTrust
SplunkTrust
0 Karma

Contributor

Still nothing in the index.

As I tried that, I was repeatitively running "ls" in /opt/splunk/var/spool/splunk/ and for a second there I could see the stash_new file. So it was created successfully, and then something got rid of it.

0 Karma

SplunkTrust
SplunkTrust

so when you search for the index are you searching for the index on your indexers or on your search heads?

Do you have forwarding for all enabled on search head? etc.

Does the index exist on the search head?

0 Karma

SplunkTrust
SplunkTrust

oh I see you've found you are forwarding therefore you need the indexes on your indexers 😜 Sorry i was late but that's where I was headed next!

0 Karma

SplunkTrust
SplunkTrust

The stash files would be available in the Search Head (where the summary index search is created).

0 Karma