Setting up splunk monitors

[fsrodriguez]

At my job whenever they set up a Splunk forwarder they add only one monitor. "/var/logs". Does anybody do it this way?

Shouldn't we adding monitors with stanzas on the /SPLUNK_HOME/etc/system/local/intputs.conf ?

We have forwarders installed on 29 servers. Our licence usage is currently at 8GB. Does this sound like its too much for the amount of servers?

Thanks in Advance

[nickhills]

if this is as you say, the chances are high that your ingesting duplicate data (such as when your log files role) and you likely have very few sourcetypes.

Whilst obviously this will (and does) work, its not a very sensible way to use Splunk.

In terms of estimating the licence usage, its difficult to say without knowing what sort of logs your collecting but 8GB/day for 30 servers seems like a lot unless they are quite busy.

[fsrodriguez]

Yeah that's what I was thinking. All of the instances have the Splunk Add-on for Unix and Linux. Some alerts are just set up to check if a service is running. I don't think we even need to add that monitor directory in order for those alerts to work.

[nickhills]

The TA for nix comes configured to collect a number of common logs from *nix systems, so its possible that's how your environment has been configured. If so you probably have sourcetype=messages or sourcetype=dmesg.

If this is the case then it may not be as you fear.
If however, all of your data is in one sourcetype I shall weep for you. 🙂

[fsrodriguez]

what do you mean by all of my data is once sourcetype?... and I have a feeling it is lol..

So lets say I am trying to monitor only the tomcat service and create a query with ps source. I should go into /etc/system/local/inputs.conf and add:

[monitor:/opt/tomcat/logs/catalina.out]

then this should work and it shouldn't return any results when the service is down correct?

host="server1" source=ps tomcat| stats latest(_time) as latest by host