Knowledge Management
Highlighted

Summary Index Not Updating

Explorer

I'm trying to debug issues with a scheduled search that writes to the summary index and the backfill script. My assumption was that the following happens in sequence:

1) Scheduled Search Runs (search is designed to run as a summary index, summary indexing is enabled, etc. etc.)

2) Files are added/modified in /var/lib/splunk/summarydb

3) A search of index="summary" will show those results

I'm finding that when 1 happens, 2 happens immediately, but 3...not so much.

What's going on? Is there some mysterious other process that puts delays between something getting written to the summary index and something being available for search from the summary index?

Tags (1)
0 Karma
Highlighted

Re: Summary Index Not Updating

Splunk Employee
Splunk Employee

I'm assuming you're doing this, but just to make sure... When you search against a summary index, the syntax should be:

index="summary" search_name="savedSearchName" | stats count ....

The search following the first pipe must match your populating search (minus 'si'). So, if your populating search is:

...| sistats count by fieldName 

your search against the index must be:

...| stats count by fieldName | more stuff...
0 Karma
Highlighted

Re: Summary Index Not Updating

Explorer

Yes. In fact, right now the summary index is totally clean so I'm just doing:

index="summary"

I've found that if I restart splunk, the index data is visible again. I also find this error in the log:

11-29-2010 10:00:05.226 ERROR databasePartitionPolicy - unable to open file: /usr/local/splunk/var/lib/splunk/summarydb/db/.metaManifest (No such file or directory)

Thanks!
-S.

Highlighted

Re: Summary Index Not Updating

Splunk Employee
Splunk Employee

More precisely, the steps are:

  1. Scheduled search runs, uses the collect command either implicitly (via "enable summary indexing" checkbox" or explicitly in the search string.
  2. collect command (with default settings) gets output, transforms, and writes it to $SPLUNK_HOME/var/spool/splunk in an intermediate file
  3. Splunk default batch input reads the intermediate file from there, writes it to the summary index
  4. Data is searchable

When you see the index files being modified, that is not done directly by the summary indexing search job, only indirectly. How long a delay are you seeing? The longest delay would normally be the pause for the batch monitor to notice and index the new output file generated by the search.

0 Karma
Highlighted

Re: Summary Index Not Updating

Explorer

Ah! So helpful! I was seeing a significant pause, often resolved by a splunk reboot. If I backfill the summary index using the backfill script, it sometimes just doesn't show up until I reboot. However, sometimes it does. It's zen that way. 🙂

Highlighted

Re: Summary Index Not Updating

Engager

link text

I also have this problem, what is the solution, thank you

0 Karma
Highlighted

Re: Summary Index Not Updating

I had the same problem and found that if I restart the SH, the index data is visible again.
Don't know why though or if it will happen again 😞

0 Karma