Search head not writing to internal or summary indexes

[pj]

We recently migrated a search head off an indexer onto a dedicated server. However it would seem that none of the internal (e.g. _internal, _audit) or default summary (e.g. summary) indexes are being written to. There is plenty of disk space assigned, so that does not seem to be the issue.

We only migrated over the users, apps and searches, not the indexes.

We did edit the inputs.conf file to not log var logs as this was causing the license to go over (as we dont have an indexing license for the search head - we are simply using the forwarder license as documented for search head implementation).

Any ideas what might be up? Thanks!

[Ellen · Accepted solution]

A Splunk Support case was logged for this issue.

Summary indexing was not occurring on the search head due to an incorrect entry in $SPLUNK_HOME/etc/system/local props.conf which sent the summary index's stash files to the nullqueue.

Removed in props.conf the stanza

[stash]
TRANSFORMS-set = setnull

When you run a saved search with summary indexing turned on, its search results are temporarily stored in a file ($SPLUNK_HOME/var/spool/splunk/<savedsearch_name>_<random-number>.stash). There should not be a need to manipulate these temporary stash files.

For further reference on summary indexing and backfill summary data gaps refer to the following: http://www.splunk.com/base/Documentation/latest/Knowledge/Usesummaryindexing?r=searchtip http://www.splunk.com/base/Documentation/4.1.5/Knowledge/Managesummaryindexgapsandoverlaps.

[pj]

No worries, it appears, we had an outputs.conf file containing, amongst others, the following lines:

[tcpout:lb]
indexAndForward = false
server = index.myserver.com:9997
autoLB = true
forwardedindex.0.whitelist = .*
forwardedindex.1.whitelist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.3.whitelist = _internal
forwardedindex.filter.disable = false

We deleted the outputs.conf file as we are not sending data anywhere and the indexes started repopulating on the search head. THe forwarder app was disabled, so not sure why this outputs.conf would make a difference.

[tpsplunk]

it looks like the $SPLUNK_HOME/etc/system/default/outputs.conf also has those same forwardedindex whitelist/blacklist lines. do you have another outputs.conf that overrides the system/default and allows _internal index data to be forwarded to your indexers? I presume you didn't delete the system/default/outputs.conf?

[Simeon]

If you have enabled the forwarder app, that could turn off local indexing. You can check which apps are enabled by running the following command:

/opt/splunk/bin/splunk display app

[pj]

Forwarder app is disabled.

SplunkForwarder UNCONFIGURED DISABLED INVISIBLE
SplunkLightForwarder UNCONFIGURED DISABLED INVISIBLE