Can't write data to _internal index

[broth1]

Recently, I have been handed the Splunk instance we have in our company. We are running Splunk 4.2.2. There's one indexer/search head, and about 75 forwarders.

As I've been learning about Splunk and checking out the various apps and configurations we currently use, I noticed that when I clicked any of the Forwarders in the Deployment Monitor App for statics, no data appeared. After looking at the code I found that the app was trying to pull the data from the _internal index.

I checked the Indexes in Manager on our indexer/search head, and found that the _internal index had been disabled. I enabled the _internal index, and restarted Splunk for good measure. When data still wasn't being written to the _internal index, I searched this site and found the post below:

http://splunk-base.splunk.com/answers/53848/why-is-no-data-being-written-to-the-_internal-index-for-...

This is why you cannot find any _internal events recorded by your search-head anywhere. To correct this, add the following configuration to $SPLUNK_HOME/etc/system/local/inputs.conf:

[tcpout]

forwardedindex.3.whitelist = _internal

I have added that as specified, restarted Splunk, and still no data is being written to _internal.

I also added the following to inputs.conf in the same folder:

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0
index = _internal

However, no data gets written to _internal. If I remove the index part, the logs are scanned and indexed, but they are placed in the "main" index.

Here's the full inputs.conf from $SPLUNK_HOME/etc/system/local/outputs.conf:

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]

disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-perfmon.path]
disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0

And outputs.conf:

[tcpout]
defaultGroup = 
disabled = false
forwardedindex.3.whitelist = _internal

Again, I am new to Splunk, so there may be other configurations I should be checking, so any help would be greatly appreciated. If you need additional information, please let me know.

[gcoles]

I recently got this working on my heavy forwarders by creating $SPLUNK_HOME/etc/system/local/outputs.conf and adding the following lines:

[tcpout]
forwardedindex.filter.disable = true

You can read more in the outputs.conf documentation.

UPDATE

If data is placed in main rather than _internal, the inputs.conf definitions must have either been changed or duplicated, with either no index, or the main index specified (because main is the fallback). Double check $SPLUNK_HOME/etc/system/default/inputs.conf -- it should look like this:

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

If you're on Linux, find all input stanzas for $SPLUNK_HOME/var/log/splunk with:

sudo find /opt/splunk/etc -name 'inputs.conf' -exec grep -H "/var/log/splunk" {} \;

Do this on the indexer and forwarders and examine the files to ensure that index = _internal

[broth1]

Thank you for the response! I apologize for the delay in my reply.

I checked $SPLUNK_HOME/etc/system/default/inputs.conf as you suggested and it has the same setting you posted in your updated comment. I did a search for other inputs.conf files on the server, but none of them defined a different index.

[default]

index         = default

[monitor://$SPLUNK_HOME\var\log\splunk]
index = _internal

Any other suggestions by any chance?

[broth1]

Thanks for the suggestion, but that did not work.

I am trying to index the Splunk log files on my main Splunk Indexer/Search head server, and even when I add the line you mentioned to $SPLUNK_HOME/etc/system/local/outputs.conf, the data is still being placed into the "main" index instead of "_internal."