I recently upgraded Splunk to 6.3. Our environment has 1 search head, 2 indexers and 1 deployment/licensing server all on Windows.
When reviewing my license usage by index, I saw that starting on the day we did the upgrade, there is a new entry for a "default" index which is taking up ~10GB a day. When I try to dig into what is causing this and search index=default
, there is a warning on search that "Search uses index=default this setting has been deprecated".
How can I dig in and determine what is causing this license utilization? Is there any what to see what is making up the data assigned to the "default" index?
From your License Master check out:
index=_internal source=*license_usage.log type="Usage" idx=default
This resulted in some logs which helped us identify the offending systems/hosts as show in the modified logs below:
11-18-2015 19:52:57.182 +0000 INFO LicenseUsage - type=Usage s="/var/log/windows/123.123.123.123/syslog.log" st="syslog-141" h="the_host_name" o="" idx="default" i="85293027-217B-40FF-8D1E-D9177AFECEB1" pool="auto_generated_pool_enterprise" b=402 poolsz=53687091200
We logged into the_host_name syslog box and saw that a few of the stanzas in the inputs.conf file did not have an index specified, so we added our default index (which is main)
[monitor:///var/log]
disabled = false
followTail = 1
sourcetype = syslog
whitelist = .log$
index = main
After this fix, we've had no further logging to index=default. Why this suddenly started to happen after the upgrade to 6.3... no idea.
From your License Master check out:
index=_internal source=*license_usage.log type="Usage" idx=default
This resulted in some logs which helped us identify the offending systems/hosts as show in the modified logs below:
11-18-2015 19:52:57.182 +0000 INFO LicenseUsage - type=Usage s="/var/log/windows/123.123.123.123/syslog.log" st="syslog-141" h="the_host_name" o="" idx="default" i="85293027-217B-40FF-8D1E-D9177AFECEB1" pool="auto_generated_pool_enterprise" b=402 poolsz=53687091200
We logged into the_host_name syslog box and saw that a few of the stanzas in the inputs.conf file did not have an index specified, so we added our default index (which is main)
[monitor:///var/log]
disabled = false
followTail = 1
sourcetype = syslog
whitelist = .log$
index = main
After this fix, we've had no further logging to index=default. Why this suddenly started to happen after the upgrade to 6.3... no idea.