Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

[edoardo_vicendo]

Hello,

Due to a specific requirement we have to install a Splunk Universal Forwarder acting as "intermediate forwarder".

Basically it will receive data via TCP (to leverage persistent queue), and it has to forward them in output in HTTP.

Forwarding data in HTTP is possible since Splunk Universal Forwarder 8.x:

https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Configureforwardingwithoutputs.con...

Here the set-up:

# inputs.conf
[tcp://9997]
persistentQueueSize=1000MB
connection_host=none
disabled=false

# outputs.conf
#Example from Splunk
[httpout]
httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0
uri = https://10.222.22.122:8088

What we also want to achieve is to forward only data received via TCP, and to do not forward the Splunk UF internal logs. I didn't found a sort of _HTTP_ROUTING setting (like for example _TCP_ROUTING) to be put in inputs.conf

Therefore listing all the Splunk UF inputs with that command:

/opt/splunkforwarder/bin/splunk btool inputs list --debug

I was thinking about this configuration:

#props.conf
[source::/opt/splunkforwarder/...]
force_local_processing = true
TRANSFORMS-null = setnull

#transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Do you think it is going to work?

Maybe another option could be tag TCP inputs host based on DNS or IP, and then move to nullQueue all the logs produced by the Splunk UF:

#inputs
[tcp://9997]
persistentQueueSize=1000MB
connection_host=dns
disabled=false

#props.conf
[host::mysplunkUFhostname]
force_local_processing = true
TRANSFORMS-null = setnull

#transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Do you see any other possible configuration?

Thanks a lot,

Edoardo

[PickleRick]

Why wouldn't you want the internal logs in the first place? They can be very helpful in troubleshooting (and are used in forwarder monitoring if I remember correctly). And they don't eat your license.

[edoardo_vicendo]

Because this Splunk UF will receive specific data and have to forward only them out in HTTP

[isoutamo]

Still, how you are planned to debug your input etc. issues w/o internal logs?

[edoardo_vicendo]

Unfortunately old stile, less/tail etc...

Not the best way but currently the only solution as HTTP output does not allow to split the output, it is all or nothing

[PickleRick]

What do you mean by "split the output"? You can normally forward events received from another UF and have them indexed separarely from local internal logs. You can do local log ingestion as well. I have many such setups - for example when a customer has a site from which he doesn't have direct visibility to splunk infrastructure due to network segmentation and filtering issues. We use intermediate forwarders and everything runs smoothly.

[richgalloway]

Transforms don't work on universal forwarders.  You'll need a heavy forwarder for that.

There's an easier answer, however.  Just disable the monitoring of internal logs.  Add these lines to /opt/splunkforwarder/etc/apps/nointernallogs/local/inputs.conf (you'll need to create the directories and file):

[monitor:///opt/splunk/etc/splunk.version]
disabled = true

[monitor:///opt/splunk/var/log/introspection]
disabled = true

[monitor:///opt/splunk/var/log/splunk]
disabled = true

[monitor:///opt/splunk/var/log/splunk/license_usage_summary.log]
disabled = true

[monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
disabled = true

[monitor:///opt/splunk/var/log/watchdog/watchdog.log*]
disabled = true

Then restart the forwarder.

[edoardo_vicendo]

@richgalloway :Thanks for your feedback, initially I though about this solution but wanted to have something "more robust" that does not depend from manual instruct which are the inputs to exclude.

By the way you are right, the transforms.conf does not apply on the UF, I'll try with your suggestion.