Split forwarding - locally indexing Splunk internal audits; forwarding system operating logs

[grijhwani]

I should probably know the answer to this, but it eludes me.

The search head of my deployment also acts as the enterprise licence server. I want to forward all the operating system logs to be indexed on one of the main indexers along with all our other management host system logs, but I want the internal audit files to remain indexed locally. If I simply enable forwarding of all logs to the indexer the internal audit detail (used for Splunk licence usage/performance reports) disappears, even if the appropriate index is created and enabled on the indexer, and besides I prefer to have the audit detail exactly where it always has been. Aggregate outputs.conf is below:

splunk@searchhead[pts/0 2013-08-07 12:31:54]:~$ splunk cmd btool outputs list --debug
system     [tcpout]
system     autoLB = true
system     defaultGroup =
system     disabled = false
system     forwardedindex.0.whitelist = .*
system     forwardedindex.1.blacklist = _.*
system     forwardedindex.2.whitelist = _audit
system     forwardedindex.3.blacklist = summary_.*
system     forwardedindex.4.blacklist = linux_.*
system     forwardedindex.filter.disable = false
system     indexAndForward = 0
system     maxQueueSize = 500KB
system     [tcpout:indexer.domain_9997]
system     disabled = false
system     forwardedindex.0.blacklist = .*
system     forwardedindex.0.whitelist =
system     forwardedindex.1.blacklist =
system     forwardedindex.1.whitelist = linux_.*
system     forwardedindex.2.whitelist =
system     forwardedindex.3.blacklist =
system     forwardedindex.4.blacklist =
system     server = indexer.domain:9997

This has now been changed to

splunk@searchhead[pts/0 2013-08-07 16:49:02]:~$ /opt/splunk/bin/splunk btool outputs list --debug
system     [tcpout]
system     autoLB = true
system     defaultGroup =
system     disabled = true
system     forwardedindex.0.whitelist = .*
system     forwardedindex.1.blacklist = _.*
system     forwardedindex.2.whitelist = _audit
system     forwardedindex.3.blacklist = summary_.*
system     forwardedindex.4.blacklist = linux_.*
system     forwardedindex.filter.disable = false
system     indexAndForward = 0
system     maxQueueSize = 500KB
system     [tcpout:indexer.domain_9997]
system     disabled = false
system     forwardedindex.0.whitelist = linux_.*
system     forwardedindex.1.blacklist = .*
system     server = indexer.domain:9997

I had already read the documentation; but just to be sure I read it again. I still don't see that I am missing anything. It that TCP forwarding is all or nothing, in that if you are forwarding anything to an external indexer, then everything has to be forwarded, although you can use transforms to determine which external router it is. If I disable the TCP forwarding then everything is indexed locally and all the _internal and _audit data returns.

The one answer I have received so far did not really help any.

[gkanapathy]

you can filter which indexes are forwarded in outputs.conf, using the forwardedindex..whitelist and .blacklist settings.

[grijhwani]

I read the documentation (again) and I'm none the wiser.

[grijhwani]

That was the basic problem to begin with. I want the _internal indexes to remain on the machine, and everything else indexed externally. When I enable TCP forwarding, everything which is whitelisted is indexed externally, but the remainder is simply not indexed.

[gkanapathy]

Please look at the outputs.conf docs. If you want to split different indexes to different places, there is a different mechanism.

[grijhwani]

That is contrary to something I am sure I read somewhere else. How would you, for instance, target different indexes at multiple different indexers if not with the blacklist/whitelist mechanism?

[gkanapathy]

seems to me that you want to remove _audit and/or _internal from the whitelist, or blacklist those indexes so they do not get forwarded. note also that the forwardedindex parameters are ignored except in the global tcpout [tcpout] stanza, and ignored in specific output stanzas [tcpout:blahblah]

[grijhwani]

Thanks. I thought I had already done this. I have added the existing outputs.conf parameters to the original question.