Why does our License Report show an increase for a "default" index after upgrade to Splunk 6.3? How do we investigate?

[stevepraz]

I recently upgraded Splunk to 6.3. Our environment has 1 search head, 2 indexers and 1 deployment/licensing server all on Windows.

When reviewing my license usage by index, I saw that starting on the day we did the upgrade, there is a new entry for a "default" index which is taking up ~10GB a day. When I try to dig into what is causing this and search index=default, there is a warning on search that "Search uses index=default this setting has been deprecated".

How can I dig in and determine what is causing this license utilization? Is there any what to see what is making up the data assigned to the "default" index?

[ejharts2015 · Accepted solution]

From your License Master check out:

 index=_internal source=*license_usage.log type="Usage" idx=default

This resulted in some logs which helped us identify the offending systems/hosts as show in the modified logs below:

11-18-2015 19:52:57.182 +0000 INFO  LicenseUsage - type=Usage s="/var/log/windows/123.123.123.123/syslog.log" st="syslog-141" h="the_host_name" o="" idx="default" i="85293027-217B-40FF-8D1E-D9177AFECEB1" pool="auto_generated_pool_enterprise" b=402 poolsz=53687091200

We logged into the_host_name syslog box and saw that a few of the stanzas in the inputs.conf file did not have an index specified, so we added our default index (which is main)

[monitor:///var/log]
disabled = false
followTail = 1
sourcetype = syslog
whitelist = .log$
index = main

After this fix, we've had no further logging to index=default. Why this suddenly started to happen after the upgrade to 6.3... no idea.