Can you help me and suggest me a solution about my context.
I have many servers Jboss, and each server host many instance/JVM (1 to 4).
Example a server jbossserver1 hosts 2 instances jbinstance1 and jbinstance2.
An instance is identified by a search time field JBOSSINSTANCE.
I would like scheduled a search to monitor the daily volume log for each instance and trigger an alert when the volume exceed a daily quota (300 MB for example)
To do this I write a search :
index="jboss" earliest=-0d@d [search index=_internal source=*license_usage.log type=Usage earliest=-0d@d | stats sum(b) as bytes by h | rename h as host | where (bytes/1024/1024)>300 | fields - bytes] NOT ([|inputlookup high_volume_jboss | fields JBOSS_INSTANCE]) | fields + host,CTX,JBOSS_INSTANCE | eval raw_len=len(_raw) | stats sum(raw_len) As TotalSize by host,CTX,JBOSS_INSTANCE | eval TotalSizeMB=round(TotalSize/1024/1024,2), quotaMB=250,newLogLevel="ERROR"| where TotalSizeMB > 250
I make search with a join on the license_usage.log to match the host which log more than 300MB (don't forget i want sizing by instance not by host)
I make a second filter to exclude instance that is matched in a input lookup containing pair host/instance matched by a precedent search in the day
I calculate the size of each raw matched
But I think this is not optimized. What do you think ?
how about using the len function and not touching the license usage logs, something like this:
earliest = -0d@d index = jboss JBOSSINSTANCE=* | eval raw =len(raw) | eval rawMB = raw/1024/1024 | stats sum(rawMB) by JBOSS_INSTANCE
now set an alert on the threshold you want
Thanks for your reply. Without licenseusage the search calculates volume for all instances of all jboss and it is not i want due to performance and duration too long. With licenseusage allow me to have a first filter on host where quota overflow at instance level is possible.
Another idea ?
@pmerlin1... You should check out DMC or Monitoring Console from Settings menu in Splunk.
Indexing > License Usage should allow you to look at License Pool and split the usage by host, source, sourcetype or whatever you need.
Ideally, if you haven't done so already, you should have separate license pool created for JBOSS servers so that you can monitor, ration and control license volume only for those set of servers. Queries in Monitoring Console should be already optimized and may allow you to accelerate performance further.
would you like to calculate the license per host or per instance? i understood that each host contains one or more instances and you would like to check those so if a host has 2 instances and one instance use 500mb per day and the other uses 100mb per day you would like to be alerted on the instance only.
did i understand correctly?
I would like calculate per instance (per host would be easier with the source license_usage.log), the alert concern only instance which exceed a quota. Yes you undestand correcly
Since you're calculating daily volume of data using length of raw data, it's going to be on slower side anyways. You may be able to setup summary index on both data to collect what you're collecting at regular interval. See this http://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Knowledge/Usesummaryindexing
absolutely, well aware of the techniques and how to use the license usage data, the challenge here as i understand it is that @pmerlin1 wants to have a report on license per instance and there are multiple instances per host.
if there are naming conventions and each instance has different source you can split your license search by s (for source) and alert on that. something like:
earliest=-1d@d latest=@d index=_internal source=*license_usage.log* type=Usage s=<yourJBOSSnamingConvention> | stats sum(b) AS Bytes by s | eval MB = Bytes/1024/1024 | table s MB | sort -MB