Sending two inputs from One universal forwarder to Two different Indexers (not load balancing)

[aholzer]

I have a universal forwarder with two different apps installed, App A and App B. The inputs for App A need to be sent and indexed at Indexer 1, while the inputs for App B need to be sent and indexed at Indexer 2. Notice that I DO NOT want redundancy.

The use case if you are wondering is as follows: I have an Infrastructure environment to monitor all the infrastructure, and I have a PROD environment to monitor our PROD applications. I don't want the data mixing between the two indexers.

How do I do this? What configurations do I need to set in what files?

My research lead me to try the following so far:

1- Tried using the _TCP_ROUTING attribute in the inputs.conf together with defining the tcpout stanza in outputs.conf. This should have lead to the wanted behavior, but it did not work. My setup still sent data only to the indexer defined in the defaultGroup. See below for my configurations.

I have a single app that defines my outputs.conf as followed:
[tcpout]
defaultGroup = PROD
disabled = false

[tcpout:INFRA]
server = Index1:<port>

[tcpout:PROD]
server = Index2:<port>

The inputs.conf for the PROD are defined without a _TCP_ROUTING attribute, which makes them default to the "defaultGroup". This part works great. Now when I define a _TCP_ROUTING on the inputs for the INFRA app, as "_TCP_ROUTING = INFRA" the data continues to be sent to the defaultGroup rather than the INFRA indexer.

2- I also found that with props.conf and transforms.conf I might be able to do what I need, but in order for that to work I'd have to upgrade my universal forwarders to Heavy forwarders and I would much rather avoid that scenario.

Thanks in advance.

[norbert_hamel · Accepted solution]

OK, then you should consider the following:
The outputs.conf defines the "physical" target server addresses (server=xyz) and assigns a logical name [tcpout:foobar].
The inputs.conf defines, to which of the logical targets the data should be send (_TCP_ROUTING = foobar).
outputs.conf and inputs.conf don't need to be in the same app. You could have different outputs.conf in each environment in /etc/system/local, defining different servers as "physical" targets and common generic logical names:

in DEV:

###########
/etc/system/local/outputs.conf

[tcpout:server1]
server=MyDevelopmentIndexer01.prod:9997

in PROD:

###########
/etc/system/local/outputs.conf

[tcpout:server1]
server=MyProductionIndexer01.prod:9997

in the app in DEV and PROD:

###########
/etc/apps/MyApp/default/inputs.conf

[monitor:///path/to/my/file1.log]
_TCP_ROUTING = server1

So you don't have to change the app when pushing it from DEV to PROD. The resulting configuration will take the outputs.conf from the system directory and the inputs.conf from the app directory.

And on top of this example, you are not limited to have the complete outputs.conf either in system OR in apps directory. You could also have 2 output.conf, defining separate parts of your final configuration.

In some cases it is a bit confusing to have several versions of one single config file on a Splunk instance. But once you are familiar with this concept you can do fancy things. For example in my environment I have UF instances which are related to a certain country, but I want to use one app with a inputs.conf that fits all countries.

So I have defined a source = FooBar_CountryCode in the default stanza of inputs.conf in the system/local or system/default directory on each UF only one time. Now I can roll out the app to all countries, and the if the monitor stanza in the app is missing the source line, the source line from my "master" inputs.conf will be used.

The precedence of the config files is described here:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Wheretofindtheconfigurationfiles

1. System local directory -- highest priority
2. App local directories
3. App default directories
4. System default directory -- lowest priority

[aholzer]

@norbet.hamel

I'm marking your answer as correct because you did in fact answer my original question. I have created a new Splunk answers question for the _TCP_ROUTING and winEventLogs, and perfmon follow up I had.

Here's the link to the new question:
http://answers.splunk.com/answers/109101/wineventlogs-and-perfmon-data-inputs-_tcp_routing

[aholzer]

Ok so here's the problem. For the regular monitoring stanzas this is a viable solution and works great.

However, for the winEventLogs and the perfmon stanzas the _TCP_ROUTING attribute does not get recognized.

Do you, or anybody else, know how to get these two stanza types to send data to a particular server? Is there another stanza type that I need to define in the outputs.conf?

Thanks again!

[aholzer]

I'm testing this, but I seem to be getting data cloning between the INFRA and PROD environments. I'll send my configurations after a bit more testing.

[norbert_hamel]

Hi,

for me the setup below is working for 2 destinations on an UF (PreProduction and Production).

I am using Heavy Forwarders as targets, but this should work with Indexers as well.

Cheers
Norbert

###########
outputs.conf

[tcpout:Forwarder_Production]
server=MyHeadyForwarder01.prod:9997, MyHeadyForwarder02.prod:9997
useACK=true

[tcpout:Forwarder_PerProduction]
server=MyHeadyForwarder01.preprod:99977
useACK=true

###########
inputs.conf

[monitor:///path/to/my/file1.log]
_TCP_ROUTING = Forwarder_Production
disabled = false
index = idx_myindex
sourcetype = MySourceType1

[monitor:///path/to/my/file2.log]
_TCP_ROUTING = Forwarder_Production,Forwarder_Preproduction
disabled = false
index = idx_myindex
sourcetype = MySourceType2

[dkeesling]

Is this correct -
[tcpout:Forwarder_PerProduction]

It seyz perPreduction -
I think this is typo.

[aholzer]

Continuation...
Because the outputs.conf are already present defining the correct group to send the data to.

If we have a defaultGroup defined and a _TCP_ROUTING, the _TCP_ROUTING seems to be ignored and it's sent to the defaultGroup no matter what.

Is this a bug?

[aholzer]

@norbert.hamel,
If we removed the "defaultGroup" from our outputs.conf, it would mean that we would have to have separate apps containing the inputs.conf for the different environments.

Say I'm developing app A. At first I have an inputs.conf that would be defined with _TCP_ROUTING = DEV. Once I'm done with my changes, I want to push it to production, I'd need to change the inputs.conf now to point to _TCP_ROUTING = PROD. If we have a defaultGroup and don't define a _TCP_ROUTING, then the inputs would be sent to the correct indexer group by default, which is what we do now. (continued...)

[norbert_hamel]

Hi, I don't get the point here. What "apps" do you have to create? And the numbers of environments DEV, PROD and INFRA will not change too often, am I wrong? Please leave a bit more details about your Splunk infrastructure, I will try to help.

BTW: I have Splunk 5 on the Indexers and 4.3 to 5 on the Universal Forwarders.

[aholzer]

@norbert.hamel I was just playing around with your solution and realized there is a mayor drawback in my case for removing the defaultGroup.

We have a development cycle for our Splunk where we index and develop dashboards on a "DEV" indexer. Which we then promote to our "PROD" and "INFRA" servers. Without the defaultGroup, we'd be stuck having to create new apps for every environment so that we can define the _TCP_ROUTING accordingly.

Any suggestions around this problem would be greatly appreciated.

[aholzer]

Ah interesting, the main difference between our configurations is that you don't have a defaultGroup defined.

I'll see if this works for me.

On a side note, what Splunk version are you using? Both for the forwarder and the indexer? I'm using 6.0 for the indexers, but I'm still on 5.4 for the forwarders.

Thanks for the suggestion, I'll give it a try and get back to you.