topic Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs in Getting Data In

Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

edoardo_vicendo — Wed, 15 Dec 2021 14:09:58 GMT

Hello,

Due to a specific requirement we have to install a Splunk Universal Forwarder acting as "intermediate forwarder".

Basically it will receive data via TCP (to leverage persistent queue), and it has to forward them in output in HTTP.

Forwarding data in HTTP is possible since Splunk Universal Forwarder 8.x:

https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Configureforwardingwithoutputs.conf#Configure_the_universal_forwarder_to_send_data_over_HTTP

Here the set-up:

# inputs.conf [tcp://9997] persistentQueueSize=1000MB connection_host=none disabled=false

# outputs.conf #Example from Splunk [httpout] httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0 uri = https://10.222.22.122:8088

What we also want to achieve is to forward only data received via TCP, and to do not forward the Splunk UF internal logs. I didn't found a sort of _HTTP_ROUTING setting (like for example _TCP_ROUTING) to be put in inputs.conf

Therefore listing all the Splunk UF inputs with that command:

/opt/splunkforwarder/bin/splunk btool inputs list --debug

I was thinking about this configuration:

#props.conf [source::/opt/splunkforwarder/...] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

Do you think it is going to work?

Maybe another option could be tag TCP inputs host based on DNS or IP, and then move to nullQueue all the logs produced by the Splunk UF:

#inputs [tcp://9997] persistentQueueSize=1000MB connection_host=dns disabled=false #props.conf [host::mysplunkUFhostname] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

Do you see any other possible configuration?

Thanks a lot,

Edoardo

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

richgalloway — Wed, 15 Dec 2021 18:49:28 GMT

Transforms don't work on universal forwarders. You'll need a heavy forwarder for that.

There's an easier answer, however. Just disable the monitoring of internal logs. Add these lines to /opt/splunkforwarder/etc/apps/nointernallogs/local/inputs.conf (you'll need to create the directories and file):

[monitor:///opt/splunk/etc/splunk.version] disabled = true [monitor:///opt/splunk/var/log/introspection] disabled = true [monitor:///opt/splunk/var/log/splunk] disabled = true [monitor:///opt/splunk/var/log/splunk/license_usage_summary.log] disabled = true [monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*] disabled = true [monitor:///opt/splunk/var/log/watchdog/watchdog.log*] disabled = true

Then restart the forwarder.

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

PickleRick — Wed, 15 Dec 2021 20:58:34 GMT

Why wouldn't you want the internal logs in the first place? They can be very helpful in troubleshooting (and are used in forwarder monitoring if I remember correctly). And they don't eat your license.

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

edoardo_vicendo — Thu, 16 Dec 2021 09:45:18 GMT

Because this Splunk UF will receive specific data and have to forward only them out in HTTP

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

edoardo_vicendo — Thu, 16 Dec 2021 09:48:28 GMT

@richgalloway :Thanks for your feedback, initially I though about this solution but wanted to have something "more robust" that does not depend from manual instruct which are the inputs to exclude.

By the way you are right, the transforms.conf does not apply on the UF, I'll try with your suggestion.

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

isoutamo — Thu, 16 Dec 2021 09:56:54 GMT

Still, how you are planned to debug your input etc. issues w/o internal logs?

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

edoardo_vicendo — Thu, 16 Dec 2021 16:15:05 GMT

Unfortunately old stile, less/tail etc...

Not the best way but currently the only solution as HTTP output does not allow to split the output, it is all or nothing

Re: Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

PickleRick — Thu, 16 Dec 2021 18:45:47 GMT

What do you mean by "split the output"? You can normally forward events received from another UF and have them indexed separarely from local internal logs. You can do local log ingestion as well. I have many such setups - for example when a customer has a site from which he doesn't have direct visibility to splunk infrastructure due to network segmentation and filtering issues. We use intermediate forwarders and everything runs smoothly.