Getting Data In

Why does our License Report show an increase for a "default" index after upgrade to Splunk 6.3? How do we investigate?

stevepraz
Path Finder

I recently upgraded Splunk to 6.3. Our environment has 1 search head, 2 indexers and 1 deployment/licensing server all on Windows.

When reviewing my license usage by index, I saw that starting on the day we did the upgrade, there is a new entry for a "default" index which is taking up ~10GB a day. When I try to dig into what is causing this and search index=default, there is a warning on search that "Search uses index=default this setting has been deprecated".

How can I dig in and determine what is causing this license utilization? Is there any what to see what is making up the data assigned to the "default" index?

1 Solution

ejharts2015
Communicator

From your License Master check out:

 index=_internal source=*license_usage.log type="Usage" idx=default

This resulted in some logs which helped us identify the offending systems/hosts as show in the modified logs below:

11-18-2015 19:52:57.182 +0000 INFO  LicenseUsage - type=Usage s="/var/log/windows/123.123.123.123/syslog.log" st="syslog-141" h="the_host_name" o="" idx="default" i="85293027-217B-40FF-8D1E-D9177AFECEB1" pool="auto_generated_pool_enterprise" b=402 poolsz=53687091200

We logged into the_host_name syslog box and saw that a few of the stanzas in the inputs.conf file did not have an index specified, so we added our default index (which is main)

[monitor:///var/log]
disabled = false
followTail = 1
sourcetype = syslog
whitelist = .log$
index = main

After this fix, we've had no further logging to index=default. Why this suddenly started to happen after the upgrade to 6.3... no idea.

View solution in original post

ejharts2015
Communicator

From your License Master check out:

 index=_internal source=*license_usage.log type="Usage" idx=default

This resulted in some logs which helped us identify the offending systems/hosts as show in the modified logs below:

11-18-2015 19:52:57.182 +0000 INFO  LicenseUsage - type=Usage s="/var/log/windows/123.123.123.123/syslog.log" st="syslog-141" h="the_host_name" o="" idx="default" i="85293027-217B-40FF-8D1E-D9177AFECEB1" pool="auto_generated_pool_enterprise" b=402 poolsz=53687091200

We logged into the_host_name syslog box and saw that a few of the stanzas in the inputs.conf file did not have an index specified, so we added our default index (which is main)

[monitor:///var/log]
disabled = false
followTail = 1
sourcetype = syslog
whitelist = .log$
index = main

After this fix, we've had no further logging to index=default. Why this suddenly started to happen after the upgrade to 6.3... no idea.

Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...