topic Re: Setting up splunk monitors in Monitoring Splunk

Setting up splunk monitors

fsrodriguez — Wed, 13 Dec 2017 13:49:19 GMT

At my job whenever they set up a Splunk forwarder they add only one monitor. "/var/logs". Does anybody do it this way?

Shouldn't we adding monitors with stanzas on the /SPLUNK_HOME/etc/system/local/intputs.conf ?

We have forwarders installed on 29 servers. Our licence usage is currently at 8GB. Does this sound like its too much for the amount of servers?

Thanks in Advance

Re: Setting up splunk monitors

nickhills — Wed, 13 Dec 2017 14:49:28 GMT

if this is as you say, the chances are high that your ingesting duplicate data (such as when your log files role) and you likely have very few sourcetypes.

Whilst obviously this will (and does) work, its not a very sensible way to use Splunk.

In terms of estimating the licence usage, its difficult to say without knowing what sort of logs your collecting but 8GB/day for 30 servers seems like a lot unless they are quite busy.

Re: Setting up splunk monitors

fsrodriguez — Wed, 13 Dec 2017 16:13:35 GMT

Yeah that's what I was thinking. All of the instances have the Splunk Add-on for Unix and Linux. Some alerts are just set up to check if a service is running. I don't think we even need to add that monitor directory in order for those alerts to work.

Re: Setting up splunk monitors

nickhills — Wed, 13 Dec 2017 16:51:26 GMT

The TA for nix comes configured to collect a number of common logs from *nix systems, so its possible that's how your environment has been configured. If so you probably have sourcetype=messages or sourcetype=dmesg.

If this is the case then it may not be as you fear.
If however, all of your data is in one sourcetype I shall weep for you. 🙂

Re: Setting up splunk monitors

fsrodriguez — Wed, 13 Dec 2017 20:05:14 GMT

what do you mean by all of my data is once sourcetype?... and I have a feeling it is lol..

So lets say I am trying to monitor only the tomcat service and create a query with ps source. I should go into /etc/system/local/inputs.conf and add:

[monitor:/opt/tomcat/logs/catalina.out]

then this should work and it shouldn't return any results when the service is down correct?

host="server1" source=ps tomcat| stats latest(_time) as latest by host