topic Re: Summary Index Not Updating in Knowledge Management

Summary Index Not Updating

srussellnpr — Thu, 25 Nov 2010 02:58:32 GMT

I'm trying to debug issues with a scheduled search that writes to the summary index and the backfill script. My assumption was that the following happens in sequence:

1) Scheduled Search Runs (search is designed to run as a summary index, summary indexing is enabled, etc. etc.)

2) Files are added/modified in /var/lib/splunk/summarydb

3) A search of index="summary" will show those results

I'm finding that when 1 happens, 2 happens immediately, but 3...not so much.

What's going on? Is there some mysterious other process that puts delays between something getting written to the summary index and something being available for search from the summary index?

Re: Summary Index Not Updating

sfleming — Thu, 25 Nov 2010 05:15:15 GMT

I'm assuming you're doing this, but just to make sure... When you search against a summary index, the syntax should be:

index="summary" search_name="savedSearchName" | stats count ....

The search following the first pipe must match your populating search (minus 'si'). So, if your populating search is:

...| sistats count by fieldName

your search against the index must be:

...| stats count by fieldName | more stuff...

Re: Summary Index Not Updating

srussellnpr — Tue, 30 Nov 2010 00:16:26 GMT

Yes. In fact, right now the summary index is totally clean so I'm just doing:

index="summary"

I've found that if I restart splunk, the index data is visible again. I also find this error in the log:

11-29-2010 10:00:05.226 ERROR databasePartitionPolicy - unable to open file: /usr/local/splunk/var/lib/splunk/summarydb/db/.metaManifest (No such file or directory)

Thanks!
-S.

Re: Summary Index Not Updating

gkanapathy — Tue, 30 Nov 2010 13:26:44 GMT

More precisely, the steps are:

Scheduled search runs, uses the collect command either implicitly (via "enable summary indexing" checkbox" or explicitly in the search string.
collect command (with default settings) gets output, transforms, and writes it to $SPLUNK_HOME/var/spool/splunk in an intermediate file
Splunk default batch input reads the intermediate file from there, writes it to the summary index
Data is searchable

When you see the index files being modified, that is not done directly by the summary indexing search job, only indirectly. How long a delay are you seeing? The longest delay would normally be the pause for the batch monitor to notice and index the new output file generated by the search.

Re: Summary Index Not Updating

srussellnpr — Tue, 30 Nov 2010 23:36:57 GMT

Ah! So helpful! I was seeing a significant pause, often resolved by a splunk reboot. If I backfill the summary index using the backfill script, it sometimes just doesn't show up until I reboot. However, sometimes it does. It's zen that way. 🙂

Re: Summary Index Not Updating

grio — Fri, 10 Feb 2012 07:11:09 GMT

link text

I also have this problem, what is the solution, thank you

Re: Summary Index Not Updating

goncalocoelho — Tue, 22 Oct 2019 12:00:40 GMT

I had the same problem and found that if I restart the SH, the index data is visible again.
Don't know why though or if it will happen again 😞