How to forward data to a remote app from a Splunk instance that is currently both a search head and indexer?

[arkadyz1]

We have a well established Splunk app on an instance which is serving as a Search Head and an Indexer. However, there are some data there which needs to be forwarded to some other site, which hosts a different application. Some of the data comes from a modular input (receiving some TCP traffic), but there are others, like *hix TA, which we would also like to forward to that other app at a different site.

Is there any trick to do that? Any special settings I need to have in inputs.conf and outputs.conf to work properly and not disturb the main operation, which has quite a few indexes and wants its data locally?

[arkadyz1]

Just to recap my experience, and as a word of caution to others who might read this question:

As soon as you define a forwarding server, the whole Splunk instance turns into a Heavy Forwarder, and everything, including the stuff normally going into _internal, gets forwarded (extremely counterintuitive to me). This is actually documented, but you might easily skip over that part if you search the docs for the specific instructions without reading the whole chapter.

So, what one needs in the case like mine:

1. Enable selective forwarding. For that you need to add the following stanza somewhere among your outputs.conf files:

   [indexAndForward]
   index=true
   selectiveIndexing=true

2. Mark each and every input you want to be kept locally (and yes, this includes the files listed in etc/system/default/inputs.conf) as such. Use _INDEX_AND_FORWARD_ROUTING property in the corresponding input stanza for that.

A simple but an error-prone procedure - it's easy to forget about an input you want to index locally.

[s2_splunk]

Can you review the documentation here and let us know if this doesn't give you the information you are looking for?

[muebel · Accepted solution]

Hi arkadyz1, You'll want to reference the documentation here : http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Routeandfilterdatad

Please let me know if this helps!

[arkadyz1]

Sorry I can't accept both answers - the links provided gave me all the info I needed. It's a non-trivial switch from universal forwarder's _TCP_ROUTING (in inputs.conf) to the heavy forwarder, with the info spread between four (!) files - inputs.conf, outputs.conf, props.conf and transforms.conf).

In addition, we need to rename the sourcetypes from Splunk_TA_nix - being an OEM, we are allowed only the sourcetypes from a predefined list. Is it best done on the receiving system?