Knowledge Management

Summary Index Not Updating

srussellnpr
Explorer

I'm trying to debug issues with a scheduled search that writes to the summary index and the backfill script. My assumption was that the following happens in sequence:

1) Scheduled Search Runs (search is designed to run as a summary index, summary indexing is enabled, etc. etc.)

2) Files are added/modified in /var/lib/splunk/summarydb

3) A search of index="summary" will show those results

I'm finding that when 1 happens, 2 happens immediately, but 3...not so much.

What's going on? Is there some mysterious other process that puts delays between something getting written to the summary index and something being available for search from the summary index?

Tags (1)
0 Karma

goncalocoelho
Path Finder

I had the same problem and found that if I restart the SH, the index data is visible again.
Don't know why though or if it will happen again 😞

0 Karma

grio
Engager

link text

I also have this problem, what is the solution, thank you

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

More precisely, the steps are:

  1. Scheduled search runs, uses the collect command either implicitly (via "enable summary indexing" checkbox" or explicitly in the search string.
  2. collect command (with default settings) gets output, transforms, and writes it to $SPLUNK_HOME/var/spool/splunk in an intermediate file
  3. Splunk default batch input reads the intermediate file from there, writes it to the summary index
  4. Data is searchable

When you see the index files being modified, that is not done directly by the summary indexing search job, only indirectly. How long a delay are you seeing? The longest delay would normally be the pause for the batch monitor to notice and index the new output file generated by the search.

0 Karma

srussellnpr
Explorer

Ah! So helpful! I was seeing a significant pause, often resolved by a splunk reboot. If I backfill the summary index using the backfill script, it sometimes just doesn't show up until I reboot. However, sometimes it does. It's zen that way. 🙂

sfleming
Splunk Employee
Splunk Employee

I'm assuming you're doing this, but just to make sure... When you search against a summary index, the syntax should be:

index="summary" search_name="savedSearchName" | stats count ....

The search following the first pipe must match your populating search (minus 'si'). So, if your populating search is:

...| sistats count by fieldName 

your search against the index must be:

...| stats count by fieldName | more stuff...
0 Karma

srussellnpr
Explorer

Yes. In fact, right now the summary index is totally clean so I'm just doing:

index="summary"

I've found that if I restart splunk, the index data is visible again. I also find this error in the log:

11-29-2010 10:00:05.226 ERROR databasePartitionPolicy - unable to open file: /usr/local/splunk/var/lib/splunk/summarydb/db/.metaManifest (No such file or directory)

Thanks!
-S.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...