Has anyone else seen that installing a Universal Forwarder turned on remote performance monitoring for the receiving Splunk 4.2.2 build 101277 instance monitoring localhost?
I’ve been using a free version of Splunk over the past few months and have had to reinstall numerous times due to exceeding the license due to my exuberance in adding data inputs. My latest instance has been up and running quite nicely for 3 weeks now with an average indexing volume well below 100 MG a day.
On Wednesday of this week I installed a Universal Forwarder on an AD machine and set up the default performance monitoring (huge mistake but totally my fault) and that night I received a daily volume limit exceeded message.
Splunk made it fairly easy to figure out where all the data was coming from, the 3 years of historical logs plus the performance monitoring reports gobbled up my indexing allocation. After struggling with the documentation looking for a configuration switch to turn off the performance monitoring and not getting anywhere I uninstalled the Universal Forwarder and then deleted all the logs that had been sent to Splunk expecting to reinstall the Universal Forwarder again today.
Much to my surprise when I logged into my Splunk server this morning to be greeted by another daily volume limit exceeded message. Splunk to the rescue, I immediately noticed that the machine I was running my Splunk server on jumped from the bottom of the list of hosts based on events to the top.
I had not set up performance monitoring on the machine hosting my Splunk instance. But since installing the Universal Forwarder on another machine on Wednesday, Splunk has been capturing Remote Performance information from localhost. Needless to say I’ve disabled it now but it cost me 2 of my 3 index volume limits for the next 30 days.
