topic Re: How split outputs in splunk in Getting Data In

How split outputs in splunk

virginiatech199 — Mon, 10 Aug 2020 13:26:43 GMT

Hello,

Running Splunk Universal Forwarder 7.3.6 (build 47d8552a4d84) on CentOS 7.

I am sending two logs -- suricata and bro - to indexers in AWS. The default splunk group for these two is lbssl

I want to split the two up like so:

suricata goes to lbssl (as it always has)

bro goes to NAD

Based on this thread: https://community.splunk.com/t5/Getting-Data-In/How-can-we-send-data-to-2-different-groups-of-indexers/td-p/280318

I have set my outputs.conf file

#ESG_072114_03
[tcpout]
defaultGroup = lbssl

[tcpout:lbssl]
compressed = true
server = old-url.com:443
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = long-encrypted-password-goes-here
sslRootCAPath = $SPLUNK_HOME/etc/apps/ssl_forwarder/cert/ca_chain.pem
sslVerifyServerCert = false

[tcpout:NAD]
compressed = true
server = new-url-for-bro-NAD-flow:443
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = another-long-encrypted-password-goes-here
sslRootCAPath = $SPLUNK_HOME/etc/apps/ssl_forwarder/cert/ca_chain.pem
sslVerifyServerCert = false

and in inputs.conf for the bro app added routing option:

[default]
_TCP_ROUTING = NAD
host=server-name-goes-here-01

Never get any data for old-url which is the suricata flow that got to splunk before changes.

new-url-for-bro-NAD-flow does appear to get data.

Any thoughts on what is incorrect/misconfigured or additional needed configs would be helpful.

Re: How split outputs in splunk

isoutamo — Mon, 10 Aug 2020 14:00:49 GMT

if this is same UF for both files then you must set output per monitor like it has done on

https://community.splunk.com/t5/Getting-Data-In/Can-single-forwarder-forward-data-to-two-different-indexer-s/td-p/71677

If you are using [default] it's used for all traffic.

r. Ismo

Re: How split outputs in splunk

virginiatech199 — Mon, 10 Aug 2020 14:22:26 GMT

i'll let you know.....

Re: How split outputs in splunk

thambisetty — Mon, 10 Aug 2020 14:36:43 GMT

Where is the input.conf defined for suricata?

Re: How split outputs in splunk

virginiatech199 — Mon, 10 Aug 2020 19:04:03 GMT

suricata is here:

/opt/splunkforwarder/etc/apps/TA-unified2/local/inputs.conf

Added an explicit call to _TCP_ROUTING tho this should not be needed:

[monitor:///nsm/sensors/.../snortlogs/.../json_out.txt]
_TCP_ROUTING = lbssl <<<<< here
initCrcLength = 630
crcSalt = <SOURCE>
disabled = false

(Also tried it without _TCP_ROUTING)

Re: How split outputs in splunk

virginiatech199 — Mon, 10 Aug 2020 19:06:29 GMT

For the moment, stand by on this question. Getting log errors

"08-10-2020 19:04:06.716 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group NAD has been blocked for 200 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data."

Checking with team that manages NAD; this is an AWS load balancer

Re: How split outputs in splunk

thambisetty — Tue, 11 Aug 2020 09:58:17 GMT

can you try monitoring different file from same universal forwarder and this input stanza should use default routing which lbssl (I mean don't define _TCP_ROUTING). check if you see data coming into your AWS indexers. if you see data coming from the new input that means there is issue with fishbucket.