Yeah, I guess what I am getting at here is that this search is still just telling us what the OS knows in terms of how much storage is used. But that is not a guarantee that the storage is necessarily being consumed entirely by indexed data. I mean, it is quite possibly being consumed entirely be indexed data.    What does your CMC tell you about the size of your indexed data on disk (the size of the data in the indexes themselves)? ... View more

It might also be possible for you to get rid of the first streamstats and just construct the table you need using stats, and then doing the second streamstats (as you can see in my suggestion above) to get the difference in the values. ... View more

I think part of your problem is that you are losing the reference to the "combination" field by doing your first timechart.    You could try something like the following: (Note: I have not tested this as I do not have access to your data)   eventtype=cacti:mirage host="onl-cacti-02" host_id=193 ldi IN("8835","8836","8837","8839","8840","8841","8846","8847","8848","8843","8844",) | reverse | eval combination=rrdn+"#"+name_cache | streamstats current=t window=2 global=f range(_time) as deltaTime range(rrdv) AS rrd_value_delta by combination | eval isTraffic = if(like(rrdn,"%traffic%"),1,0) | eval kpi = if(isTraffic==1,rrd_value_delta*8/deltaTime,rrd_value_delta/deltaTime) | bucket _time span=5m | stats last(kpi) as last_kpi by combination _time | sort combination _time | streamstats current=t window=2 range(last_kpi) as last_kpi_diff by combination | eval change_percent=(last_kpi/(last_kpi-last_kpi_diff))*100 | where change_percent>30 AND last_kpi_diff!=0       ... View more

Were you ever seeing data in this index for any time range, using a normal search?   I am wondering if you are encountering an access problem for the specific index, and that the tstats search is returning something as a result of a bug.  ... View more

When you expand the time range for your regular (non-tstats) search, do you get any results? Like over a 24 hour or 7 day time range? ... View more

Are you trying to measure for a 30% difference for any one of the values?  In other words: "If any of these values (combination) have a difference of >30%, then fire an alert" ... View more

Is the value for latest(_time) returning a time that is within your time range? ... View more

When you say that the "disk usage" is 95%, presumably this is total disk usage on the indexers as reported by the OS.   What does the monitoring console tell you about the actual index sizes on disk?  In other words, what is Splunk saying its storage usage is for the indexes (buckets)?      ... View more

I would recommend breaking up your rex statement into a few different regexes. This way, you can anchor on the items that are closer to the data you want to extract.  For example: | rex field=_raw "<TYPE regex here>" | rex field=_raw "<EType regex here>" | rex field=_raw "<TCode regex here>" etc.   Alternatively you can construct 2 or 3 large regexes that can accommodate the different event structures you have, and in each regex, call the fields slightly different names.  I.E. Regex 1 would extract TCode1, and regex 2 would extract TCode2.  Then you can use the eval command with the coalesce function to merge these fields together later on to TCode. For example: | eval TCode=coalesce(Tcode1,TCode2) ... View more

If your dropdown is a non-static list of choices that come from a search query, how do you have a "respective panel" for each dropdown choice? If you have a respective panel for each dropdown choice, that implies that you know upfront how many dropdown options there can be.  ... View more

You need to make sure that you have the same field name for "user" between both indexes. Assuming that the "user field" in both indexes is called "user", and assuming that you have fields extracted which are called "client" and "count_field" and "requestID" then something like the following should work:   index=indexA "Received from client" OR ("request id:" "completed") | transaction startsWith="Received from client" endsWith="completed" requestID | stats values(client) as client values(user) as user by requestID | join type=outer user [| search index=indexB "has total sent items count" | stats sum(count_field) as count_total by user]   The thing to watch out for here is that the count that is returned from index2 will essentially be a sum of the count for the user. Because there is no "requestID" present in Index2. So the count_field that is seen in the index2 is not directly tied to the requestID that is seen in index1.  ... View more

You can do it using a match() statement within your case statement as follows:   ...< original search > ... | eval action=case(Err="server timeout, try after sometime","Ignore",match(Err,"Web service error"),"follow-up",Err="Address element not found","Ignore")   Hope this helps! ... View more

That will work too.   However if there is a lot of data in the index, then it will mean you have to search double the data by expanding the time range from 1 day to 2 days.  Additionally, it essentially means that you are "double searching" at least one day's worth of data.    On the other hand, there are also technical constraints to using the subsearch and join method. Just worth noting the tradeoffs between both options.  ... View more

The link below is to a similar question with a good answer that also contains helpful links to Splunk documentation that references best practices. This should have what you need.  https://community.splunk.com/t5/Getting-Data-In/What-are-the-best-practices-for-defining-source-types/m-p/423052 ... View more

Set your cron scheduled as follows for the scheduled report and it should work.   0 */1 1,2,3,4,5,26,27,28,29,30,31 * * ... View more

The reason that we are using _time for bin is because this allows us to use the first stats command in a similar way that the timechart works, BUT without losing the field called my_count.    Timechart is very similar to running "stats" with "by _time" at the end. Hence, I wanted to use a "stats" with "by _time", to create a similar result set. But in order to do that, I needed to use "bin" with _time first before replacing the timechart with the stats command.  The main thing you need to think about is, regardless of what you are putting in your query, you need to look at the fields that are produced in the RESULTS at each step of the query. Only the fields that are produced/outputted after each phase of the query will be available to the next phase of the query.    So, in your example, you were using timechart. And if you just ran that part of the search and checked the statistics table of the results, you would see that there is no field called "my_count". Which means that the field called "my_count" will not be available for the next part of the search (which is your last stats command).    So the reason that I used _time with bin, is to replace your timechart command with a stats command, which preserves the my_count field to be used in the next part of the search (which is your last stats command).   I hope that helps! ... View more

Ah I see. Then your best bet will be to do something like the search I had posted. This will essentially group all of the events into a single event, for each value of workflowname, and then you can send your email based off of those values for workflowname and suppress based on the same field.    Let me know if this makes sense and if the search which I provided gives you the outcome you are looking for.  ... View more

A few things to note here:   When you configure the alert to trigger "when the result are greater than 0" this means that it will trigger as long as there is at least one result returned from the search. This is NOT looking at your field called "count". In your example, the alert fires and has 6 results, and hence the condition for "when the results are greater than 0" will be triggered. Unfortunately, if you are suppressing based on the workflowname, then you cannot have 3 separate emails triggered the way you want, using only 1 alert. Because as soon as Splunk sees a THE FIRST result for WorkFlow1, then it will only trigger an email for that result, before checking for other values for workflowname.    You have two options in this situation:   1. Create 3 different alerts. Each one will look for a specific workflowname and trigger an alert for the entire alert result set.  2. Adjust the search to collect all of the "records" for EACH workflowname into a single "event", and then run your alert the same way you are doing it. You can do this by running a search like the following (ignore the first part, which is just creating some data for me to work with)   | makeresults count=6 | streamstats count | eval workflowname=case(count<=3,"workflow1",count=4,"workflow3",count>=5,"workflow2"), runid=case(count<=3,123,count=4,555,count>=5,678), count_field=case(count=1,3,count=2,8,count=3,2,count=4,10,count=5,12,count=6,4), status=case(count=1,"completed",count=2,"paused",count=3,"completed",count=4,"running",count=5,"running",count=6,"completed") ```IGNORE EVERYTHING ABOVE THIS LINE. THIS IS JUST CREATING SOME SAMPLE DATA TO WORK WITH``` | table workflowname count_field runid status | eval concat_field="workflow=".workflowname." ::: runid=".runid." ::: count=".count_field." ::: status=".status | stats values(concat_field) as concat_field by workflowname   ... View more

When you say you are trying to "plot" the Fail_Rate on a chart, I presume you are trying to chart this over time, by ChanID?     source="C:\\Call_logs" termcodeID=1 OR termcodeID=34 OR termcodeID=7 OR termcodeID=9 OR termcodeID=21 OR termcodeID=27 OR termcodeID=30 OR termcodeID=32 OR termcodeID=34 ChanID!=0 | bucket _time span=1h | eval good_call=if(termcodeID=1,1, 0) | eval bad_call=if(good_call=1,0, 1) | eventstats count(termcodeID) as totalcalls sum(bad_call) as bad_call_count by ChanID | eval Fail_Rate=((bad_call_count/totalcalls)*100) | timechart span=1h max(Fail_Rate) as Fail_Rate by ChanID   If so, you can try something like this (but I have not seen your data so I cannot guarantee if this will work for you, but some variation of this should work).   (If this is not what you are trying to do, please clarify what you are trying to do and I will re-work the search to try and accommodate your use case)     ... View more

What do you mean the "threshold" time? How are you calculating this? If a search is configured to run once per hour over 60 mins of data, what would the threshold time be in this case? 60 mins? And if the search is configured to run once per hour looking over 2 hours of data, is the threshold time still 60 mins?  ... View more

You could run a search on a schedule that checks the status of current jobs using the search/jobs API endpoint and then look at the runDuration for those jobs and see when they are running more than a certain threshold.    Something like the following will get you started.  | rest /services/search/jobs search="delegate=scheduler" | fields label savedSearchLabel published dispatchState runDuration delegate   Remember, there are lots more fields available for the results returned from the search/jobs API endpoint, so you can always rework the search to use some of those other fields also.  ... View more