topic Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index? in Getting Data In

How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

Federica_92 — Tue, 01 Dec 2015 15:50:38 GMT

Hi everyone,

I'm trying to use splunk as heavy forwarder to send out only 1 index, but it doesn't work. Could someone please help me? I think there is something wrong in the outputs.conf.

[tcpout]
deafultGroup = nothing

[tcpout:alerts]
server = 10.28.100.121:9998
indexAndForward = 1

[tcpout:alerts]
indexAndForward = 1
#Forward data for the "alerts" index
forwardedindex.0.whitelist = alerts

transforms.conf:

[alerts]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=alerts

props.conf

[index::alerts]
TRANSFORMS-routing = alerts

Thanks,
Federica

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

stmyers7941 — Tue, 01 Dec 2015 15:56:09 GMT

I believe you need a single stanza for [tcpout:alerts] with all the settings underneath it. With 2 stanzas, It's going to use the latest and ignore the former.

[tcpout]
 deafultGroup = nothing
[tcpout:alerts]
server = 10.28.100.121:9998
indexAndForward = 1
#Forward data for the "alerts" index
forwardedindex.0.whitelist = alerts

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

Federica_92 — Tue, 01 Dec 2015 16:12:22 GMT

No, I already tried, doesn't filter the data. It sends everything.

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

stmyers7941 — Tue, 01 Dec 2015 16:31:34 GMT

Have you tried Selective Indexing?

#
# Perform selective indexing and forwarding
#
# With a heavy forwarder only, you can index and store data locally, as well as
# forward the data onwards to a receiving indexer. There are two ways to do
# this:

# 1. In outputs.conf:
[tcpout]
defaultGroup = indexers

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:indexers]
server = 10.1.1.197:9997, 10.1.1.200:9997

# 2. In inputs.conf, Add _INDEX_AND_FORWARD_ROUTING for any data that you want
#    index locally, and
_TCP_ROUTING=<target_group> for data to be forwarded.

[monitor:///var/log/messages/]
_INDEX_AND_FORWARD_ROUTING=local

[monitor:///var/log/httpd/]
_TCP_ROUTING=indexers

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

Federica_92 — Tue, 01 Dec 2015 17:07:15 GMT

I can't monitor, because the file aren't stored anywhere, I need to send out data that splunk is producing and indexing. The only way to access directly to the data, it's to access to the kv store

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

stmyers7941 — Tue, 01 Dec 2015 18:42:48 GMT

What's your input on the heavy forwarder in question?

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

lguinn2 — Wed, 02 Dec 2015 00:24:54 GMT

First, your outputs.conf is wrong. It has misspellings and as has been pointed out, you have duplicate stanzas. Plus, most of the settings must appear in the tcpout stanza at the global level.

[tcpout]
defaultGroup = alerts
indexAndForward = true
forwardedindex.0.whitelist = alerts

[indexAndForward]
index = true
selectiveIndexing = false

 [tcpout:alerts]
server = 10.28.100.121:9998

Your props.conf has no effect, because there is no such stanza as [index::xyz]. Therefore, your transforms.conf was never invoked, which is just as well because the regular expression in the REGEX would not have filtered anything. Just remove the props.conf and transforms.conf entries; you don't need them.

I am not sure that the whitelist is going to work. If it does not, then replace it with these two lines instead:

forwardedindex.0.blacklist = *
forwardedindex.1.whitelist = alerts

In the future, you might want to run

splunk btool check

from the command line, which may identify syntax errors in your configuration files. btool can't catch everything, but it can help.

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

Federica_92 — Wed, 02 Dec 2015 09:40:40 GMT

I tried, with both, whitelist only and whitelist and blacklist.
I got the same issue as before, the traffic is being sent, but not filtered...

Re: How do I edit my single-machine deployments outputs.conf to send out data for only 1 index?

Federica_92 — Wed, 02 Dec 2015 10:10:01 GMT

I have an alert manager ( the app) that is executing a script, whenever an alerts is being triggered. At that stage it produce some data, using the rest API that are saved on the kv stores