Due to a specific requirement we have to install a Splunk Universal Forwarder acting as "intermediate forwarder".
Basically it will receive data via TCP (to leverage persistent queue), and it has to forward them in output in HTTP.
Forwarding data in HTTP is possible since Splunk Universal Forwarder 8.x:
Here the set-up:
# inputs.conf [tcp://9997] persistentQueueSize=1000MB connection_host=none disabled=false
# outputs.conf #Example from Splunk [httpout] httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0 uri = https://10.222.22.122:8088
What we also want to achieve is to forward only data received via TCP, and to do not forward the Splunk UF internal logs. I didn't found a sort of _HTTP_ROUTING setting (like for example _TCP_ROUTING) to be put in inputs.conf
Therefore listing all the Splunk UF inputs with that command:
/opt/splunkforwarder/bin/splunk btool inputs list --debug
I was thinking about this configuration:
#props.conf [source::/opt/splunkforwarder/...] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue
Do you think it is going to work?
Maybe another option could be tag TCP inputs host based on DNS or IP, and then move to nullQueue all the logs produced by the Splunk UF:
#inputs [tcp://9997] persistentQueueSize=1000MB connection_host=dns disabled=false #props.conf [host::mysplunkUFhostname] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue
Do you see any other possible configuration?
Thanks a lot,
Why wouldn't you want the internal logs in the first place? They can be very helpful in troubleshooting (and are used in forwarder monitoring if I remember correctly). And they don't eat your license.
What do you mean by "split the output"? You can normally forward events received from another UF and have them indexed separarely from local internal logs. You can do local log ingestion as well. I have many such setups - for example when a customer has a site from which he doesn't have direct visibility to splunk infrastructure due to network segmentation and filtering issues. We use intermediate forwarders and everything runs smoothly.
Transforms don't work on universal forwarders. You'll need a heavy forwarder for that.
There's an easier answer, however. Just disable the monitoring of internal logs. Add these lines to /opt/splunkforwarder/etc/apps/nointernallogs/local/inputs.conf (you'll need to create the directories and file):
[monitor:///opt/splunk/etc/splunk.version] disabled = true [monitor:///opt/splunk/var/log/introspection] disabled = true [monitor:///opt/splunk/var/log/splunk] disabled = true [monitor:///opt/splunk/var/log/splunk/license_usage_summary.log] disabled = true [monitor:///opt/splunk/var/log/splunk/splunk_instrumentation_cloud.log*] disabled = true [monitor:///opt/splunk/var/log/watchdog/watchdog.log*] disabled = true
Then restart the forwarder.
@richgalloway :Thanks for your feedback, initially I though about this solution but wanted to have something "more robust" that does not depend from manual instruct which are the inputs to exclude.
By the way you are right, the transforms.conf does not apply on the UF, I'll try with your suggestion.