Splunk Search

Why is the count of indexed events shown under “What to Search” on the default page of the Search & Reporting app much lower than expected?

sat94541
Communicator

Upon login to Splunk, when we navigate to default page for the Search & Reporting app, the count under “What to Search” is incorrect. The number showing is 721,000+ when it should in the billions.

Refer picture to see the issue
all text

Tags (3)
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

1)First thing to eliminate will be that caused by the fact that "| metadata" (and therefore, that UI Panel) is scoped by default to the default set of indexes that the user in question is set to search - as per http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Metadata :

As mentioned by the following section in above link

Optional arguments[edit]
index-specifier
Syntax: index=
Description: Specify the index from which to return results. Wildcards () can be used. index= matches non-internal indexes. index=_* matches internal indexes.
Default: The default index, which is usually main
So I'm pretty sure that this is governed by the following role-specific setting in authorize.conf:

srchIndexesDefault =
* Semicolon delimited list of indexes to search when no index is specified
* These indexes can be wildcarded, with the exception that '' does not
match internal indexes
* To match internal indexes, start with ''. All internal indexes are
represented by '
'
* Defaults to none, but the UI will automatically populate this with 'main'
in manager

2)One very simple way to check this is to check what kind of event count you get with:

| metadata type=sourcetypes | stats sum(totalCount) AS total_event_count_of_indexes_searched_by_default

and

| metadata type=sourcetypes index=* | stats sum(totalCount) AS total_event_count_of_all_searchable_indexes

3) In case the search in (2) both return incorrect count, next you can try and rebuild metadata - using the following command

splunk cmd splunkd recover-metadata /localhome/igor/local/core/var/lib/splunk/defaultdb/db/db_1442694953_1442694953_0

This command is referred at link --http://docs.splunk.com/Documentation/Splunk/6.3.0/Troubleshooting/Buckets

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

1)First thing to eliminate will be that caused by the fact that "| metadata" (and therefore, that UI Panel) is scoped by default to the default set of indexes that the user in question is set to search - as per http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Metadata :

As mentioned by the following section in above link

Optional arguments[edit]
index-specifier
Syntax: index=
Description: Specify the index from which to return results. Wildcards () can be used. index= matches non-internal indexes. index=_* matches internal indexes.
Default: The default index, which is usually main
So I'm pretty sure that this is governed by the following role-specific setting in authorize.conf:

srchIndexesDefault =
* Semicolon delimited list of indexes to search when no index is specified
* These indexes can be wildcarded, with the exception that '' does not
match internal indexes
* To match internal indexes, start with ''. All internal indexes are
represented by '
'
* Defaults to none, but the UI will automatically populate this with 'main'
in manager

2)One very simple way to check this is to check what kind of event count you get with:

| metadata type=sourcetypes | stats sum(totalCount) AS total_event_count_of_indexes_searched_by_default

and

| metadata type=sourcetypes index=* | stats sum(totalCount) AS total_event_count_of_all_searchable_indexes

3) In case the search in (2) both return incorrect count, next you can try and rebuild metadata - using the following command

splunk cmd splunkd recover-metadata /localhome/igor/local/core/var/lib/splunk/defaultdb/db/db_1442694953_1442694953_0

This command is referred at link --http://docs.splunk.com/Documentation/Splunk/6.3.0/Troubleshooting/Buckets

aljohnson_splun
Splunk Employee
Splunk Employee

Those are just the indexes that are searched by default.

If you go to:

  • Settings > User Access > Roles
  • chose a specific role
  • Scroll to the bottom

You will see:
alt text

The counts you're talking about in the data summary are only reflective of the indexes that are searched by default.


Here are a few other ways to get an event count:

| eventcount index=*

Or internal indexes:

| eventcount index=_*

Or try using the metadata command:

| metadata type=sourcetypes index=*

There are many other ways of course.

Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...