thx for your advice! I already know the pdf - a really good presentation, but it doesn't solve this particular problem...
The status quo looks like this:
Every server has its own certificates (signed by our private CA) - one for Inter-Splunk communication and one for web. All our Universial Forwarders are using one single certificate (signe by our private CA too).
We configured the Universial Forwarder's outputs.conf to use our certificate (certificate-path and password). The outputs.conf is located in an app which is deployed via the Deployment Server. Funnily enough this password will be encrypted at restart of the Universial Forwarder. Therefore for the communication between Forwarder and Indexer our certificate is used and everything works fine.
However the communication between the Forwarders and the Deployment Server is the problem. For this connection the Forwarder uses the default Splunk certificate (the server.conf in $SPLUNK_HOME/etc/system/default refers to it). Trying to deploy a new server.conf (in which our certificate is configured) via the app mentioned above didn't solve the problem because the provided cleartext password wasn't encrypted... Providing an encrypted password in the new server.conf is useless as well, because every Forwarder uses its own splunk.secret file to decrypt the password.
... View more
we are using self-signed certificates in our Splunk environment. In general everything works fine, but at a closer look we found that the Universial Forwarders aren't using our self-signed forwarder-certificate. Instead they are using the Splunk default certificates. Examining the conf-files I found out, that the forwarder's server.conf ($SPLUNK_HOME/etc/system/default) refers to that certificate.
Changing the server.conf via an app doesn't work because the certificate password will not be encrypted.... 😞
Has anyone an idea how we can deploy a new server.conf to our Universial Forwarder in an (Splunk supported/recommended) automated way...? Using of System Center or such other tools isn't a good option, because of lack of privileges but for several hundred Universial Forwarders it would be "a little bit" annoying to fix that problem manually.
... View more
By examining the _internal logs I found the following, Metric Error:
ERROR Metrics - Metric with name thruput:thruput already registered
It is reported by Universal Forwarders of several Clients spread over the entire day (with peaks in the morning hours - so I suppose that it's related to the client's start-up)
The interesting thing is, that all of these clients are still reporting events to the Indexers...
Why does this happen?
And how can I avoid this?
... View more