Why Is My Summary Index Not Being Populated?

[bill_bartlett]

I've been scratching my head for a couple of days trying to figure this out without success.

In following the Summary Index Knowledge Manual article titled "Usesummaryindexing" I believe I have the search scheduled correctly. I don't have the karma to post screenshots or links, so I'll do my best to describe the problem.

• search name "summary_prod_5min"
• search: index=prod | rex "(?<"LOG">.{300})" | sitop limit=100 LOG (I had to quote "LOG" for the wiki text parser. the actual search contains no quotes around LOG)
• start time: -6m end time: -1m (to account for the slight lag in indexing..)
• scheduled every 5 minutes (2,7,12,17,22,27,etc etc)
• all default alert settings with no email, script, triggered alerts, etc.
• summary indexing enabled to summary index called "ops_summary"
• added field "report" with value of "prod_5min"

As far as I can tell, this is correctly configured. I also made sure that the ops_summary index is configured on the search head:

(root@splunkSH) splunk > bin/splunk cmd btool indexes list | sed -n -e '/ops_summary/,/\[/ p'
[ops_summary]
assureUTF8 = false
blockSignSize = 0
blockSignatureDatabase = _thefishbucket
bucketRebuildMemoryHint = auto
coldPath = $SPLUNK_DB/ops_summary/colddb
coldPath.maxDataSizeMB = 0
coldToFrozenDir =
coldToFrozenScript =
compressRawdata = true
defaultDatabase = _thefishbucket
enableOnlineBucketRepair = true
enableRealtimeSearch = true
frozenTimePeriodInSecs = 188697600
homePath = $SPLUNK_DB/ops_summary/db
homePath.maxDataSizeMB = 0
hotBucketTimeRefreshInterval = 10
indexThreads = auto
maxBloomBackfillBucketAge = 30d
maxBucketSizeCacheEntries = 0
maxConcurrentOptimizes = 1
maxDataSize = auto
maxHotBuckets = 3
maxHotIdleSecs = 0
maxHotSpanSecs = 7776000
maxMemMB = 5
maxMetaEntries = 1000000
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
maxTimeUnreplicatedNoAcks = 300
maxTimeUnreplicatedWithAcks = 60
maxTotalDataSizeMB = 100000
maxWarmDBCount = 300
memPoolMB = auto
minRawFileSyncSecs = disable
minStreamGroupQueueSize = 2000
partialServiceMetaPeriod = 0
processTrackerServiceInterval = 1
quarantineFutureSecs = 2592000
quarantinePastSecs = 77760000
rawChunkSizeBytes = 131072
repFactor = 0
rotatePeriodInSecs = 60
serviceMetaPeriod = 25
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
streamingTargetTsidxSyncPeriodMsec = 5000
suppressBannerList = _audit
sync = 0
syncMeta = true
thawedPath = $SPLUNK_DB/ops_summary/thaweddb
throttleCheckPeriod = 15
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
warmToColdScript =

The next step was to make sure the search returned results. I went to the search, and clicked "view recent" where I see each run does produce results; in the neighborhood of ~400k per run. I clicked "inspect" and the things that I would suspect to be there, are there. (summary index name is correct, input and output counts are there, summary_index action, etc.)

From what I can tell, everything indicates the summary index is configured appropriately. When I do a simple search, I get no results:

• earliest=-1h index=ops_summary

I tried to be more specific, and it's still empty:

• earliest=-1h index=ops_summary report=prod_5min

Can anyone lend some advice on where to look for trouble? I'm happy to provide any additional details as well.

[bill_bartlett · Accepted solution]

Well, this has been solved.

First, using the summary index commands only works with the default "summary" index. If, as I had, you try to use a non-default summary index, nothing works.

My other problem was that my search head was not forwarding any data to the indexers. Once I configured that, and stopped using si* commands, my summary index of "ops_summary" started populating fine.

Thanks again to everyone who contributed.

[bhawkins1]

When you tried to use a non-default summary index, are you sure you first created the index? I'm having a similar issue to the one you describe above, but have not solved it using the solution you describe here. Best guess is that I have a bigger problem than you did.

[bbartlett_splun]

This is a year and a half old, so I don't remember exactly how I set this up. In reading the other comments, it looks like initially I made the mistake of not creating the new index on the indexers.

Can you elaborate on your problem, or create a new question with the steps you've already went through so we can try to assist you?

[bhawkins1]

Thanks for the response. I think that a rename * in my query is mangling timestamps, which means that I can find the results of my report by searching the summary index with index=, but, confusingly, see "There are no results because the first scheduled run of the report has not completed." when I open the report itself. If I don't solve this myself I'll surely make a new post. Thanks again.

[nayakr_r]

Hi All,

I have the similiar problem. I have configured the indexers in indexer servers and search head and forwarding from search to indexer is enabled.

Problem statement 1: Summary indexing view recent jobs are showing the results but those are not getting forwareded to indexer layer. As an aleternative sometime I run the jobs manually from view recent and it will push the data. Manual run is working fine but auto excecution is not working.

Problem statement 2: Lets say all the summary indexing working fine. I have noticed after restart of search head summary indexing is stopped working.

As an alternative whenever I restart search head I go and refresh the summary indexing jobs manually.

These 2 are critical problems I'm facing in splunk summary indexing jobs.

Could someone guide me resolve the issue

Thanks,

Nayak

[bill_bartlett]

UPDATE: I've tried with the default index and it behaves in the same way. I've also tried using '| top | collect index=xxxx' instead of sitop, and it still behaves the same way with nothing being written to the summary index.

Checking the results on "view recent" runs, the table is generated, and the output is what I'd expect to see.

[NOUMSSI]

Hi;
At the level of choice of the start time and end time of your summary indexing, instead to use this:
start time: -6m end time: -1m
Use this
start time: -6m@m end time: -1m@m

[bill_bartlett]

Thanks, NOUMSSI. That's a good point on my oversight. I don't think that's contributing to my problem, but I should clean up the start & end times as you suggest.

[NOUMSSI]

Ok.
try and let me know if there is something else

[bill_bartlett]

I changed the time, but the problem still persists. I see nothing when I search the "ops_summary" index.

[dolivasoh]

Are you doing this in a distributed environment?

[bill_bartlett]

Yes, there are two indexers configured in distributed search.

[bill_bartlett]

following up on this -- I had assumed I could summarize on the search head and search there as well without forwarding to the indexers. Is that not the case?

This is a fairly small deployment (~50GB/day) and everything is run within EC2 on EBS volumes, so data availability/replication isn't a major concern. (backed up to S3 via snapshots)

[harsmarvania57]

Are you forwarding data to indexers from search head?

If you are forwarding data to indexers from search head then "ops_summary" index must be present on indexers as well.

[bill_bartlett]

harsmarvania57 - I am not forwarding data on the search head. I created those indexes on the two indexers as well just in case I missed something, but that hasn't helped either.