Universal Forwarder 6.4.0 to HEC

[acamilo_2]

Unfortunately, I have a few hundred hosts running 6.4 universal forwarder and I cannot upgrade them. I have a subset of hosts that need to send an application log to HEC on customer's splunk deployment but still send this and all the other logs to our splunk. We already have a few apps for the forwarder to handle different logs in a different way so I decided to create an app for those forwarders to send to the customer so that I can make sure that only the application log gets sent to them.
Hosts are Ubuntu 14.04
Splunk servers are on 16.04 and we run Splunk Enterprise 6.5.2
We don't use HEC in any of our hosts hence why I'm asking for help here.

I'm confident about all the config files except output.conf, I'm not clear on what defaultGroups does or if it's even necessary and the tcpout:app_upstart_logs stanza is also nebulous to me.

output.conf
[tcpout]
defaultGroup = ???
forwardedindex.filter.disable = true
indexAndForward = false
maxQueueSize = 250MB
maxConnectionsPerIndexer = 5

[tcpout:app_upstart_logs]
useACK = false
token = XXXXXXXXXXXXXXXXXXXXXXXXXX
server = customer.splunk-server.net:8080

This is my inputs.conf

[monitor:///var/log/upstart/]
disabled = false
sourcetype = app_upstart_logs
blacklist = (\.gz$|\.0$|.1$|\.2$|.3$|\.4$|.5$|\.6$|.7$|\.8$|\.9$|\.10$|\.report$|lost\+found)

PS: We are running some really old stuff. Feel free to roll your eyes like a teenage valley girl (is that still a thing?) I stand ashamed.

[bandit]

An http event collector (hec) is essentially a heavy forwarder with the special hec listener. You would use the same sort of outputs.conf that you would use on any search head or heavy forwarder to route logs to your downstream indexers.

Example outputs.conf config:

# BASE SETTINGS

# TURN OFF INDEXING ON SEARCH HEAD OR HEAVY FORWARDER
[indexAndForward]
index = false

[tcpout]
# FORWARD ALL INTERNAL LOGS
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

# round robin to all indexers
defaultGroup = all_indexers

[tcpout:all_indexers]
server = host500.acme.com:9997,host501.acme.com:9997,host502.acme.com:9997,host503.acme.com:9997,host504.acme.com:9997