Is is possible to direct Insighs Infrastructure Collectd agent to send to the 9997 of a Universal Forwarder instead using HEC 8088?

[luisbontempo]

Hi, we are trying to use Splunk Insights for Infrastructure in a large company where security policies are quite tight. By default, machines have open connections to port 9997 on Heavy Forwarders, so opening a new port on 8088 for each agent install is not practical.

Is it possible to make insights CollectD agent use the Universal Forwarder, that is also in the agent machine, to send its data instead of directly sending data to the Infrastructure server using HEC on port 8088?

Any other suggestions?

Many thanks in advance!

Luis Bontempo

[luisbontempo]

Hi all,

Thanks to Matthew we were able to sort this one out so here are the steps to do it.
On the machine where you installed the agents do as sysadmin:

Edit the agent machine universal forwarder configuration "inputs.conf" and change it to

[http]
disabled=0
port=8088
enableSSL=0
dedicatedIoThreads=2
maxThreads = 0
maxSockets = 0
useDeploymentServer=0
# ssl settings are similar to mgmt server
sslVersions=*,-ssl2
allowSslCompression=true
allowSslRenegotiation=true


[http://uf_hec_local]
disabled = 0
token = 00000000-0000-0000-0000-000000000000

Restart the universal forwarder
sudo /opt/splunkforwarder/bin/splunk restart

Edit the CollectD agent
vi /etc/collectd/collectd.conf

server "localhost"
port "8088"
token "00000000-0000-0000-0000-000000000000"
ssl true
verifyssl false

Restart the CollectD service
service collectd restart

SECURITY INFO: for simplicity the token was all zeros but for security reasons best to change to another key
WARNING: This is not officially supported !

[mattymo]

would also recommend setting hec to only listen to localhost, unless you are trying to receive remote calls to hec using acceptFrom in inputs.conf

acceptFrom =  ...
* Lists a set of networks or IP addresses from which to accept connections.
* Specify multiple rules with commas or spaces.
* Each rule can be in the following forms:
    1. A single IPv4 or IPv6 address (examples: "10.1.2.3", "fe80::4a3")
    2. A CIDR block of addresses (examples: "10/8", "fe80:1234/32")
    3. A DNS name, possibly with a '*' used as a wildcard (examples:
       "myhost.example.com", "*.splunk.com")
    4. A single '*', which matches anything.
* You can also prefix an entry with '!' to cause the rule to reject the
  connection. The input applies rules in order, and uses the first one that
  matches. For example, "!10.1/16, *" allows connections from everywhere except
  the 10.1.*.* network.
* Defaults to "*" (accept from anywhere)

[stefan_d]

Thanks for sharing!

[mattymo · Accepted solution]

Hey Luis!

Thanks for checking out insights for infra!!

While what your asking is totally doable, - to the best of my knowledge, at least - catching HEC with UF is not officially supported.

That being said, there are customers who do it and have no issues. You can chat with some of them on our slack chat, sign up here: splk.it/slack

I would still like to explore the option with you and help you try it out to see if you can achieve what makes deploying easier for you!

please hit me up at the email in my bio, or at @mattymo on Slack in our #insights-for-infra channel, where myself and the PM and ENG folks hang out!

[stefan_d]

We're also looking for a similar solution!

[mattymo]

we got it working! Will Ask Luis to share his experience 🙂