Below is teh query I am using to alert me of excessive hourly usage by a host. Is there a way to modify the first portion of the query "index=_internal source=*license_usage.log" to ignore a specific index?
index=_internal source=*license_usage.log | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc
I have tried:
All of the above modifications have failed. None of them give any results. I am currently assuming this is because the index=_internal statement is covering all indexes and I am unable to create exceptions for that statement. Can anyone speak to this? thoughts? comments? questions? concerns? Any ideas are greatly appreciated.
thanks!
I believe i have found my answer. After looking at the output for
index=_internal source=*license_usage.log
i saw that there is a variable called "h" that contains the hostname.
5/6/13
11:22:24.171 AM
05-06-2013 11:22:24.171 -0400 INFO LicenseUsage - type=Usage s="udp:514" st=syslog h="otherhost" o="" i="<removed>" pool="auto_generated_pool_enterprise" b=21067 poolsz=5368709120
host=<removed> Options| sourcetype=splunkd Options| source=<removed>
I then edited my original query to ignore the hostnames i wanted to ignore (index=_internal source=*license_usage.log) NOT(h="ignoreME")
and below is the full results.
(index=_internal source=*license_usage.log) NOT(h="ignoreME") | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc
It sounds like you've got where you want to be now though it's worth also checking out this link for some useful ways of troubleshooting license usage - some more searches.
In addition to this, Splunk Deployment Monitor app and Splunk on Splunk apps provide some useful dashboards for tracking usage and splitting it by host, sourcetype etc. These apps are free and available on Splunkbase.
Another place to look is in the search app under the Status > Index Activity menu option. There are some useful dashboards there also, notably Index Volume.
thank you for the information dkuk. I have been using those apps for information gathering in the past but they are not able to generate alerts which is what i really needed, so i had to create this manual queries.
I believe i have found my answer. After looking at the output for
index=_internal source=*license_usage.log
i saw that there is a variable called "h" that contains the hostname.
5/6/13
11:22:24.171 AM
05-06-2013 11:22:24.171 -0400 INFO LicenseUsage - type=Usage s="udp:514" st=syslog h="otherhost" o="" i="<removed>" pool="auto_generated_pool_enterprise" b=21067 poolsz=5368709120
host=<removed> Options| sourcetype=splunkd Options| source=<removed>
I then edited my original query to ignore the hostnames i wanted to ignore (index=_internal source=*license_usage.log) NOT(h="ignoreME")
and below is the full results.
(index=_internal source=*license_usage.log) NOT(h="ignoreME") | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc
as per a suggestion from /k i have modified my overall query to:
index=_internal source=license_usage.log NOT h="hosts" | bucket _time span=1h | eval when=strftime(_time, "%F %H:00") | stats sum(b) as mbytes by h, when | eval mbytes = round(mbytes/1024/1024,2) | where mbytes>15 | sort by mbytes desc
I believe there are simpler ways of achieving what you're trying to do.
index=_internal source=*license_usage.log
| bucket _time span=1h
| eval when=strftime(_time, "%F %H:00")
| stats sum(b) as mbytes by h, when
| eval mbytes = round(mbytes/1024/1024,2)
| where mbytes>25
thank you for the suggestion. This is much cleaner and simpler than my current query. I have modified mine and posted the new query in the answer. Thank you very much for showing me a better way.
thank you but that still does not allow me to ignore certain indexes or hosts. but i do think i have found a solution.
If your search only looks in index _internal it will not look into any other index, any other index already is excluded.
thanks! I will try that now and see if i can improve my query.
What goes in the internal index _internal
?
general splunkd messages (splunkd.log)
searches that have been performed (searches.log)
metrics data from forwarders (metrics.log)
...
try this;
index=_internal | dedup 3 source sourcetype
This will give you a small sample of the types of events/sources that will end up in _internal
to be honest I am confused then to what _internal uses. Is my whole query wrong then based off of that first statement? i was think that "(index=_internal source=*license_usage.log) NOT [search index=dogs]" might do what i needed but after you last comment i am not sure anymore. Can you elaborate a bit on what indexes contain what data? I am going to start googling to find more information. Thanks k!
Internal indexes in general start with an underscore. There are several of these, e.g. _audit
, _blocksignature
and _internal
.
If you specify an index as part of the search, only that index will be searched.
main
is not an internal index, and there is no such thing as 'index enveloping'.
/k
does index=_internal envelope all internal indexes like main, dhcp, win_dns, etc?