Getting Data In

How to send all received traffic on a specific port from Heavy Forwarders to a clustered index?

Path Finder

Environment:
2x heavy forwarders (6.4.1) in a load balanced pool (sitting behind haproxy) and using indexer_discovery
1x cluster master, 3x indexer peers
2x search heads

Question:
I am receiving data in a specific clustered index when using a universal forwarder's inputs.conf:

[monitor:///opt/splunk/etc/system/local/]
disabled = 0
index=clustered_index

outputs.conf:

[tcpout]
defaultGroup = default-autolb-group-g0
[tcpout:default-autolb-group-g0]
server = 10.10.10.10:9997
[tcpout-server://10.10.10.10:9997]

So that's fine and I can search the data.

What I want to do, is as I have multiple receive ports on the HF's, I would like to just send everything received on port 9997, 5220 etc. to a specific index on the back-end cluster - And this is from sources that do not have universal forwarders.

Am I able to just send all data from heavy forwarders to an index on a cluster that is not defaultdb/main?

I have tried setting inputs.conf on the HF's to:

[monitor://9997]
disabled = 0
index=clustered_index

but I do not seem to be seeing the traffic?

Thanks in advance!
Bry

1 Solution

Splunk Employee
Splunk Employee

Do you have a requirement that mandates the use of heavy forwarders? If not, consider using universal forwarders instead.

To your question: You are trying to use the monitor stanza to listen on a port and that's not going to work, because monitor is used by the file TailingProcessor.

If you want to listen on a network (TCP or UDP) port, you need to use the appropriate stanza instead as documented here.

I would consider it bad practice to use a splunk default port (9997 in this case) to listen for anything other than Splunk2Splunk traffic. If you have some universal forwarders that would send data through your intermediate forwarders, they should use that port.
For any other traffic, I would chose a port not used by Splunk by default. Just saying...

View solution in original post

Splunk Employee
Splunk Employee

Do you have a requirement that mandates the use of heavy forwarders? If not, consider using universal forwarders instead.

To your question: You are trying to use the monitor stanza to listen on a port and that's not going to work, because monitor is used by the file TailingProcessor.

If you want to listen on a network (TCP or UDP) port, you need to use the appropriate stanza instead as documented here.

I would consider it bad practice to use a splunk default port (9997 in this case) to listen for anything other than Splunk2Splunk traffic. If you have some universal forwarders that would send data through your intermediate forwarders, they should use that port.
For any other traffic, I would chose a port not used by Splunk by default. Just saying...

View solution in original post

Path Finder

@ssievert

thanks for the doc pointer, it had the answer - I edited the tcp input for the listener I was using (non- s2s 9997) and added '-index clustered_index' and it indeed went to the correct index.

Thx
Bry

Path Finder

Hi ssievert

thanks for your reply, much appreciated.

Reason for HF's, is that at some point we will be parsing a lot of data to filter out unecessary data before indexing, so we thought appropriate to use heavy forwarders (happy to be shown an alternative way of course 🙂 )

tcp 9997 we are only receiving splunk sources data, so effectively we have universal forwarders on centos for example, and with inputs/outputs.conf sending that data over 9997. We do receive (or want to receive) data from non-UF sources and for this we're listening on another tcp port, 5220 as an example.

Specifying the target index on the universal forwarders send correctly (via proxy and load balanced HF's) but I really want to be able to specify all traffic from the non-splunk sources to a target index other than default/main - is that possible on the HF's?

Again, thanks for your response.
Bry

0 Karma

Splunk Employee
Splunk Employee

You could do your future filtering on the indexers directly, but if you are concerned about the unfiltered event stream eating up your network bandwidth, HF is the way to go. Rule of thumb: Unless you are filtering more than 50% of the raw event stream, do it on the indexer (UFs are more efficient on the wire and less resource-hungry).

To get your HFs to listen on a different port, configure a TCP input for that port, for example:
[tcp://5220]
index=yourTargetIndex
sourcetype=yourSourcetype

The documentation link I had posted prior contains details.
If you have a mixed data stream coming in on 5220 and want to set index/sourcetype based on contents of the events, you'll have to do dynamic overrides of sourcetype/index via props/transforms (documented here

0 Karma

Path Finder

Hi

I should also say that I am receiving data on multiple ports fine - it's just that data that is not targeting a specific index (universal forwarders work well) goes direct into the defaultdb/main and I want to be able to change that to a specific clustered index.

Thx
Bry

0 Karma

Path Finder

on my index peers (/opt/splunk/etc/slave-apps/_cluster/local/inputs.conf) I have:

[splunktcp://9997]

I'm wondering if I add the index here, it would go to the specified index instead of defaultdb? (of course i would add on the master and apply the cluster bundle to the peers)

I.e.
[splunktcp://9997]
index=clustered_index

0 Karma

Splunk Employee
Splunk Employee

I wouldn't do that on the cluster peers for sure. If you do, all data that will ever come in on port 9997 will be written to the same index. What if you need additional indices in the future?

Setting your index in inputs.conf on your forwarder should work just fine. Where did you configure it? It should either be in $SPLUNK_HOME/etc/system/local or in an app directory, e.g. $SPLUNK_HOME/etc/apps/my_inputs/local (I made up my_inputs, name it to your liking).

For information on how to override your index name based on the sourcetype, review this post

0 Karma

Path Finder

Hi ssievert

thanks for your reply, always appreciated!

ok, thats a good call - I shall be looking to send different data surces to different indexes (at some point).

I have set the inputs.conf on the HF's at: /opt/splunk/etc/system/local/inputs.conf

[tcp://5220]
index=clustered_index

I also tried [splunktcp://5220] but i am not seeing any data being sent to the index, only the main..

the HF's outputs.conf look like:
[indexer_discovery:name1]
pass4symmKey = hashed
master_uri = https://ip:8089

[tcpout:group1]
autoLBFrequency = 30
forceTimebasedAutoLB = true
indeerDiscovery = name1
useACK = true

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = ip1:9997,ip2:9997,ip3:9997
sendCookedData = false

The HF's are sending data to the cluster idx peers and universal forwarders are sending to the clustered-index but the HF's are just sending into the defaultdb/main

Thx
Bry

0 Karma

Path Finder

Hi ssievert

Apologies for delay and thanks for responding again, very much appreciated!

Ok, I didn't think I was able to target an index on the HF unless of course i was indexing locally (on the HF)? I believe I tried adding a remote index and didn't see any data going to the backend index but I may have done that incorrect, so will try again. It works fine when setting the target index on a universal forwarder.

1: Can I target a remote index on the HF via a tcp data input?
2: If not, how would I filter the data on the backend clustered index, so that a specific sourcetype goes to a specific clustered index, would this also be via a tcp dta input?

I will of course read further through the link/document you provided previously.

Thx
Bry

0 Karma