topic Re: alter index=_internal query to ignore a specific index in Alerting

alter index=_internal query to ignore a specific index

rnolette — Mon, 28 Sep 2020 13:50:18 GMT

Below is teh query I am using to alert me of excessive hourly usage by a host. Is there a way to modify the first portion of the query "index=_internal source=*license_usage.log" to ignore a specific index?

index=_internal source=*license_usage.log | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc

I have tried:

(index=_internal source=*license_usage.log) NOT index=other
(index=_internal NOT index=other) source=*license_usage.log
(index=* NOT index=other) source=*license_usage.log
(index=* source=*license_usage.log) NOT index=other

All of the above modifications have failed. None of them give any results. I am currently assuming this is because the index=_internal statement is covering all indexes and I am unable to create exceptions for that statement. Can anyone speak to this? thoughts? comments? questions? concerns? Any ideas are greatly appreciated.

thanks!

Re: alter index=_internal query to ignore a specific index

martin_mueller — Mon, 06 May 2013 14:51:48 GMT

If your search only looks in index _internal it will not look into any other index, any other index already is excluded.

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 28 Sep 2020 13:50:21 GMT

does index=_internal envelope all internal indexes like main, dhcp, win_dns, etc?

Re: alter index=_internal query to ignore a specific index

kristian_kolb — Mon, 06 May 2013 15:01:55 GMT

Internal indexes in general start with an underscore. There are several of these, e.g. _audit, _blocksignature and _internal.

If you specify an index as part of the search, only that index will be searched.

main is not an internal index, and there is no such thing as 'index enveloping'.

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 28 Sep 2020 13:50:23 GMT

to be honest I am confused then to what _internal uses. Is my whole query wrong then based off of that first statement? i was think that "(index=_internal source=*license_usage.log) NOT [search index=dogs]" might do what i needed but after you last comment i am not sure anymore. Can you elaborate a bit on what indexes contain what data? I am going to start googling to find more information. Thanks k!

Re: alter index=_internal query to ignore a specific index

kristian_kolb — Mon, 06 May 2013 15:23:06 GMT

I believe there are simpler ways of achieving what you're trying to do.

index=_internal source=*license_usage.log  
| bucket _time span=1h
| eval when=strftime(_time, "%F %H:00") 
| stats sum(b) as mbytes by h, when 
| eval mbytes = round(mbytes/1024/1024,2) 
| where mbytes>25

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 06 May 2013 15:24:48 GMT

thank you but that still does not allow me to ignore certain indexes or hosts. but i do think i have found a solution.

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 06 May 2013 15:28:15 GMT

I believe i have found my answer. After looking at the output for

index=_internal  source=*license_usage.log

i saw that there is a variable called "h" that contains the hostname.

5/6/13
    11:22:24.171 AM 
    05-06-2013 11:22:24.171 -0400 INFO  LicenseUsage - type=Usage s="udp:514" st=syslog h="otherhost" o="" i="<removed>" pool="auto_generated_pool_enterprise" b=21067 poolsz=5368709120
    host=<removed>   Options|  sourcetype=splunkd   Options|  source=<removed>

I then edited my original query to ignore the hostnames i wanted to ignore (index=_internal source=*license_usage.log) NOT(h="ignoreME") and below is the full results.

(index=_internal source=*license_usage.log) NOT(h="ignoreME") | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc

Re: alter index=_internal query to ignore a specific index

kristian_kolb — Mon, 06 May 2013 15:28:43 GMT

What goes in the internal index _internal?

general splunkd messages (splunkd.log)
searches that have been performed (searches.log)
metrics data from forwarders (metrics.log)
...

try this;

index=_internal | dedup 3 source sourcetype

This will give you a small sample of the types of events/sources that will end up in _internal

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 06 May 2013 15:36:53 GMT

thanks! I will try that now and see if i can improve my query.

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 28 Sep 2020 13:50:27 GMT

as per a suggestion from /k i have modified my overall query to:

Re: alter index=_internal query to ignore a specific index

rnolette — Mon, 06 May 2013 16:01:59 GMT

thank you for the suggestion. This is much cleaner and simpler than my current query. I have modified mine and posted the new query in the answer. Thank you very much for showing me a better way.

Re: alter index=_internal query to ignore a specific index

dkuk — Tue, 07 May 2013 06:54:03 GMT

It sounds like you've got where you want to be now though it's worth also checking out this link for some useful ways of troubleshooting license usage - some more searches.

In addition to this, Splunk Deployment Monitor app and Splunk on Splunk apps provide some useful dashboards for tracking usage and splitting it by host, sourcetype etc. These apps are free and available on Splunkbase.

Another place to look is in the search app under the Status > Index Activity menu option. There are some useful dashboards there also, notably Index Volume.

Re: alter index=_internal query to ignore a specific index

rnolette — Tue, 07 May 2013 12:37:26 GMT

thank you for the information dkuk. I have been using those apps for information gathering in the past but they are not able to generate alerts which is what i really needed, so i had to create this manual queries.