topic How can I forward only _internal index data from indexer to the new indexer? in Getting Data In

How can I forward only _internal index data from indexer to the new indexer?

benazir — Tue, 20 Feb 2018 10:20:16 GMT

I am facing a problem in forwarding the _internal data to the new indexer.

my case is I have to forward only _internal data from all the indexers to new indexer servers because in our environment we have dedicated indexer for _internal data.

when i do this below entry in one of the indexer
inputs.conf:

[monitor:///opt/splunk/idx/splunk/var/log/splunk]
_TCP_ROUTING = management

outputs.conf

[tcpout]
forwardedindex.0.blacklist = .*
forwardedindex.1.whitelist = _internal
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false
disabled=false

[tcpout:management]
server = 10.178.48.66:9997

This makes all the data to forward from this particular indexer to the new indexer, I need only _internal data to get forwarded.

I tried using props.conf and transforms.conf too. It's not working. I don't want to store the _internal data in this indexer, it should present only in the new indexers.
Kindly need your help.

Re: How can I forward only _internal index data from indexer to the new indexer?

deepashri_123 — Tue, 20 Feb 2018 14:03:47 GMT

Is there any reason why a particular indexer set for internal indexes only? This is not the best practice to do so.

Re: How can I forward only _internal index data from indexer to the new indexer?

somesoni2 — Tue, 20 Feb 2018 22:45:20 GMT

Try with this outputs.conf (should be etc/apps under some_app/local OR last resort, under etc/system/local)

[tcpout]
indexAndForward = true

[tcpout:management]
server = 10.178.48.66:9997

[indexAndForward]
index=true

Re: How can I forward only _internal index data from indexer to the new indexer?

benazir — Wed, 21 Feb 2018 03:35:18 GMT

I tried this option, what it does it , it keeps a copy of internal logs here in the old indexers and forward to new indexers too.

but my case is , I need to see the _internal data of that particular indexers only in the new indexers, not on the source indexer, when I search data from search head for _internal index..
since we have dedicated search heads , for different cluster of indexers.

Kindly need to your advice, how to just forward, without doing local indexing .

Re: How can I forward only _internal index data from indexer to the new indexer?

benazir — Wed, 21 Feb 2018 04:28:40 GMT

I haven given the outputs.conf file like below :
[tcpout]
forwardedindex.0.blacklist = .*
forwardedindex.1.whitelist = _internal
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false
disabled=false
indexAndForward = true

[tcpout:management]
server = 10.178.48.66:9997

[indexAndForward]
index = true

Now this is how it works, I cant find any other data forwarded to new management indexer ( that's good)
but the problem is _internal data is routed to main index in the new server - 10.178.48.66 and missing few logs like splunkd,metrics all.
meantime in the old indexer I am still seeing the data from main as well as _internal indexes.