Knowledge Management

Summary search runs, has results, yet summary index remains empty

gabriel_vasseur
Contributor

This is the first time I'm setting up a summary search and I must be missing something.

If I click "view recent" I can see that my search actually ran at the scheduled time and had a "resultCount" of 4649 so it was successful.
I can see the "search" string ends with

| summaryindex spool=t ... index="my_index" file="somefile.stash_new" ...

So that means it knew it was supposed to index the results.

I have compared the configuration with other summary searches that I know work and I can't really see a difference.

I have logged onto all the indexers and the search head and run the following commands in /opt/splunk:

find . -name *.stash_new
ls var/spool/splunk/

Both return nothing.

I have tracked down the relevant search.log on the search head and haven't seen anything that jumped at me. There's a SummaryIndexProcessor INFO line that says "successfully wrote file to 'xxxxx.stash_new'".

I have checked for "warnings from the spooler" with:

index=_internal source=*splunkd.log "*spool*"

But that returned nothing.

So surely all went well and the stash file got indexed and then deleted... so why is my summary index empty?

I have run out of things to look at. Anybody has an idea?

0 Karma

mcgrawb
Engager

When creating any index, the index stanza must be added to the indexes.conf on all indexers by way of the cluster master in a clustered environment or manually on each indexer in a non-clustered environment. Assuming, of course, the search head is not indexing locally and forwarding its data to the indexers.

If you have proper permissions and the search head is not indexing locally, you should see an error similar to the one below in _internal or the Messages drop-down within the web UI. The warning would be reported by each indexer that receives the summary index data for an index not configured.

Received event for unconfigured/disabled/deleted index=<-index-> with source="<-source->" host="<-host->" sourcetype="<-sourcetype->". So far received events from 'X' missing index(es).

Managing indexes:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setupmultipleindexes

Peer configuration:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Configurethepeerindexes

gabriel_vasseur
Contributor

I think I might have figured it out...

We have search head forwarding setup, but the index I created doesn't appear to exist on any of the indexers... It looks like I'll have to create it manually (not via the search head's splunk web).

My initial question still holds: how could I have had a clue that this is my problem? Where's the error message?

0 Karma

woodcock
Esteemed Legend

By default, every SI exists only on the Search Head where the populating search runs. Why are you checking with non-Splunk CLI? Why not just do a search like this on the Search Head for All time:

index=my_index
0 Karma

gabriel_vasseur
Contributor

That was the first thing I did. Like I said, the index is empty.

0 Karma

woodcock
Esteemed Legend

Then you should say so; this is a very important detail.

0 Karma

jkat54
SplunkTrust
SplunkTrust
0 Karma

gabriel_vasseur
Contributor

Still nothing in the index.

As I tried that, I was repeatitively running "ls" in /opt/splunk/var/spool/splunk/ and for a second there I could see the stash_new file. So it was created successfully, and then something got rid of it.

0 Karma

jkat54
SplunkTrust
SplunkTrust

so when you search for the index are you searching for the index on your indexers or on your search heads?

Do you have forwarding for all enabled on search head? etc.

Does the index exist on the search head?

0 Karma

jkat54
SplunkTrust
SplunkTrust

oh I see you've found you are forwarding therefore you need the indexes on your indexers 😜 Sorry i was late but that's where I was headed next!

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The stash files would be available in the Search Head (where the summary index search is created).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...