Monitoring Splunk

Can't write data to _internal index

New Member

Recently, I have been handed the Splunk instance we have in our company. We are running Splunk 4.2.2. There's one indexer/search head, and about 75 forwarders.

As I've been learning about Splunk and checking out the various apps and configurations we currently use, I noticed that when I clicked any of the Forwarders in the Deployment Monitor App for statics, no data appeared. After looking at the code I found that the app was trying to pull the data from the _internal index.

I checked the Indexes in Manager on our indexer/search head, and found that the _internal index had been disabled. I enabled the _internal index, and restarted Splunk for good measure. When data still wasn't being written to the _internal index, I searched this site and found the post below:

http://splunk-base.splunk.com/answers/53848/why-is-no-data-being-written-to-the-_internal-index-for-...

This is why you cannot find any _internal events recorded by your search-head anywhere. To correct this, add the following configuration to $SPLUNK_HOME/etc/system/local/inputs.conf:


[tcpout]
forwardedindex.3.whitelist = _internal

I have added that as specified, restarted Splunk, and still no data is being written to _internal.

I also added the following to inputs.conf in the same folder:


[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0
index = _internal

However, no data gets written to _internal. If I remove the index part, the logs are scanned and indexed, but they are placed in the "main" index.

Here's the full inputs.conf from $SPLUNK_HOME/etc/system/local/outputs.conf:


[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-perfmon.path]
disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0

And outputs.conf:


[tcpout]
defaultGroup = 
disabled = false
forwardedindex.3.whitelist = _internal

Again, I am new to Splunk, so there may be other configurations I should be checking, so any help would be greatly appreciated. If you need additional information, please let me know.

Tags (2)
0 Karma

Communicator

I recently got this working on my heavy forwarders by creating $SPLUNK_HOME/etc/system/local/outputs.conf and adding the following lines:

[tcpout]
forwardedindex.filter.disable = true

You can read more in the outputs.conf documentation.

UPDATE

If data is placed in main rather than _internal, the inputs.conf definitions must have either been changed or duplicated, with either no index, or the main index specified (because main is the fallback). Double check $SPLUNK_HOME/etc/system/default/inputs.conf -- it should look like this:

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

If you're on Linux, find all input stanzas for $SPLUNK_HOME/var/log/splunk with:

sudo find /opt/splunk/etc -name 'inputs.conf' -exec grep -H "/var/log/splunk" {} \;

Do this on the indexer and forwarders and examine the files to ensure that index = _internal

0 Karma

New Member

Thank you for the response! I apologize for the delay in my reply.

I checked $SPLUNK_HOME/etc/system/default/inputs.conf as you suggested and it has the same setting you posted in your updated comment. I did a search for other inputs.conf files on the server, but none of them defined a different index.


[default]
index = default

[monitor://$SPLUNK_HOME\var\log\splunk]
index = _internal

Any other suggestions by any chance?

0 Karma

New Member

Thanks for the suggestion, but that did not work.

I am trying to index the Splunk log files on my main Splunk Indexer/Search head server, and even when I add the line you mentioned to $SPLUNK_HOME/etc/system/local/outputs.conf, the data is still being placed into the "main" index instead of "_internal."

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!