Getting Data In

Configure Splunk Universal Forwarder with TCP input to send data with HTTP output to HEC filtering internal logs

edoardo_vicendo
Contributor

Hello,

Due to a specific requirement we have to install a Splunk Universal Forwarder acting as "intermediate forwarder".

Basically it will receive data via TCP (to leverage persistent queue), and it has to forward them in output in HTTP.

Forwarding data in HTTP is possible since Splunk Universal Forwarder 8.x:

https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Configureforwardingwithoutputs.con...

 

Here the set-up:

# inputs.conf
[tcp://9997]
persistentQueueSize=1000MB
connection_host=none
disabled=false
# outputs.conf
#Example from Splunk
[httpout]
httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0
uri = https://10.222.22.122:8088

 

What we also want to achieve is to forward only data received via TCP, and to do not forward the Splunk UF internal logs. I didn't found a sort of _HTTP_ROUTING setting (like for example _TCP_ROUTING) to be put in inputs.conf

Therefore listing all the Splunk UF inputs with that command:

/opt/splunkforwarder/bin/splunk btool inputs list --debug

 

I was thinking about this configuration:

#props.conf
[source::/opt/splunkforwarder/...]
force_local_processing = true
TRANSFORMS-null = setnull

#transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

 

Do you think it is going to work?

Maybe another option could be tag TCP inputs host based on DNS or IP, and then move to nullQueue all the logs produced by the Splunk UF:

#inputs
[tcp://9997]
persistentQueueSize=1000MB
connection_host=dns
disabled=false

#props.conf
[host::mysplunkUFhostname]
force_local_processing = true
TRANSFORMS-null = setnull

#transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

 

Do you see any other possible configuration?

 

Thanks a lot,

Edoardo

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Why wouldn't you want the internal logs in the first place? They can be very helpful in troubleshooting (and are used in forwarder monitoring if I remember correctly). And they don't eat your license.

0 Karma

edoardo_vicendo
Contributor

Because this Splunk UF will receive specific data and have to forward only them out in HTTP

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Still, how you are planned to debug your input etc. issues w/o internal logs?
0 Karma

edoardo_vicendo
Contributor

Unfortunately old stile, less/tail etc...

Not the best way but currently the only solution as HTTP output does not allow to split the output, it is all or nothing

0 Karma

PickleRick
SplunkTrust
SplunkTrust

What do you mean by "split the output"? You can normally forward events received from another UF and have them indexed separarely from local internal logs. You can do local log ingestion as well. I have many such setups - for example when a customer has a site from which he doesn't have direct visibility to splunk infrastructure due to network segmentation and filtering issues. We use intermediate forwarders and everything runs smoothly.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Transforms don't work on universal forwarders.  You'll need a heavy forwarder for that.

There's an easier answer, however.  Just disable the monitoring of internal logs.  Add these lines to /opt/splunkforwarder/etc/apps/nointernallogs/local/inputs.conf (you'll need to create the directories and file):

[monitor:///opt/splunk/etc/splunk.version]
disabled = true

[monitor:///opt/splunk/var/log/introspection]
disabled = true

[monitor:///opt/splunk/var/log/splunk]
disabled = true

[monitor:///opt/splunk/var/log/splunk/license_usage_summary.log]
disabled = true

[monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*]
disabled = true

[monitor:///opt/splunk/var/log/watchdog/watchdog.log*]
disabled = true

Then restart the forwarder.

---
If this reply helps you, Karma would be appreciated.

edoardo_vicendo
Contributor

@richgalloway :Thanks for your feedback, initially I though about this solution but wanted to have something "more robust" that does not depend from manual instruct which are the inputs to exclude.

By the way you are right, the transforms.conf does not apply on the UF, I'll try with your suggestion. 

0 Karma
Get Updates on the Splunk Community!

App Building 101 - Build Your First App!

WATCH RECORDING NOW   Tech Talk: App Dev Edition Splunk has tons of out-of-the-box functionality, and you’ve ...

Introducing support for Amazon Data Firehose in Splunk Edge Processor

We’re excited to announce a powerful update to Splunk Data Management with added support for Amazon Data ...

The Observability Round-Up: September 2024

What’s up Splunk Community! Welcome to the latest edition of the Observability Round-Up, a monthly series in ...