About ToKnowMore

ToKnowMore · ‎09-15-2022

BTW, I did try the following modification, but the result was not what is desired: index=firewall host=10.214.0.11 NOT src_ip=172.26.22.192/26 | stats count by src_ip, dest_ip | appendpipe [| stats sum(count) as count by src_ip |eval keep=1 | eventstats sum(count) as total_log_count ] | appendpipe [| stats sum(count) as count by dest_ip |eval keep=1 | eventstats sum(count) as total_log_count ] |where keep=1| sort -count | head 20 | where total_log_count > 500000 AND count > 30000 I received the following output which eliminated the rest of the IPs I was hoping to see: src_ip dest_ip count keep total_log_count 192.168.23.5 39035 1 785936 192.168.23.5 31974 1 785936

ToKnowMore · ‎09-15-2022

This is a search string I inherited and for the most part has worked fine. There is a desire to modify it and thought I would seek help. index=firewall host=10.214.0.11 NOT src_ip=172.26.22.192/26 | stats count by src_ip, dest_ip | appendpipe [| stats sum(count) as count by src_ip |eval keep=1 | eventstats sum(count) as total_log_count ] | appendpipe [| stats sum(count) as count by dest_ip |eval keep=1 | eventstats sum(count) as total_log_count ] |where keep=1| sort -count | head 20 | where total_log_count > 1000000 Below example outputs received, separate instances: src_ip dest_ip count keep total_log_count 192.168.14.11 39164 1 1008943 192.168.14.11 32239 1 1008943 10.80.0.243 31880 1 1008943 143.251.111.100 30773 1 1008943 156.33.250.10 15544 1 1008943 192.242.214.186 13793 1 1008943 172.253.63.188 12359 1 1008943 192.168.5.46 12346 1 1008943 192.168.10.146 10987 1 1008943 192.168.3.19 9079 1 1008943 192.168.3.195 8970 1 1008943 192.168.3.18 8074 1 1008943 172.18.3.42 7709 1 1008943 192.168.14.23 7647 1 1008943 192.168.5.46 7583 1 1008943 172.253.63.188 6549 1 1008943 172.33.250.10 5806 1 1008943 192.168.24.65 5654 1 1008943 172.253.115.188 5494 1 1008943 192.168.24.134 4388 1 1008943 src_ip dest_ip count keep total_log_count 87.114.132.220 45441 1 1005417 192.168.35.6 39597 1 1005417 192.168.14.15 31629 1 1005417 172.30.5.9 16348 1 1005417 10.80.0.243 15444 1 1005417 196.199.95.18 13883 1 1005417 172.253.62.139 12703 1 1005417 192.168.12.45 11957 1 1005417 172.253.115.188 10010 1 1005417 192.168.3.19 9676 1 1005417 192.168.35.16 9641 1 1005417 192.168.5.146 9290 1 1005417 192.168.25.46 7440 1 1005417 172.253.115.188 7292 1 1005417 192.168.3.18 6163 1 1005417 192.168.39.18 6063 1 1005417 176.155.19.207 5818 1 1005417 4.188.95.188 4947 1 1005417 5.201.73.253 4942 1 1005417 45.225.238.30 4938 1 1005417 Is there a way to modify the query such that it only triggers if there is a single entity causing logs greater than a certain number (e.g. 50000) in combination with the total logs also being over a certain threshold? There is still a desire to see an output reporting the top 20 IPs. Your time, consideration and helpful suggestions is appreciated. Thank you.

ToKnowMore · ‎10-13-2020

Hi @ITWhisperer Hope this message finds you well. The email alert worked. I received an email alert in the early morning hours with the expected reason event identified. Thank you!! For the sake of those who may find this configuration useful, could you please explain what the below search string means and the additional configuration components (e.g. the blue arrows) needed for a scheduled alert with a rolling window? index=firewall host=10.10.10.10 earliest="-7m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | where count > 100 | head 20 Example email alert received: It is my hope your explanation of the config would be better articulated from you. Once done, I will gladly report your concise explanation as the solution. For sure you have earned that. Thanks again for your time, patience, and input throughout this topic!

ToKnowMore · ‎10-08-2020

Hi @ITWhisperer Hope this entry finds you well. I have "normalized" the search string, or what I think is how the search string will be based on your input: index=firewall host=10.10.10.10 earliest="-7m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | where count > 100 | head 20 The time range is set for last 8 mins and the "trigger alert when" is set for 75 (search percent>75). I have not received any email alerts since the above in place yesterday and am unsurprised. I suspect that minum count of 100 and the percent threshold of 75 has not been met. It is my hope that the true test will be next week when we've seen the event of interest take place. I just did not want to discount any other time, hence the rolling trigger alert. If I may, I will provide an update on Tuesday, 10/13/2020. I hope this is acceptable. Thanks for your time, help and patience!

ToKnowMore · ‎10-07-2020

Hi @ITWhisperer Sorry I missed that early suggestion, I apologize. I have made the suggested change: I suspect with "where count > 20" in play only those REASON codes hitting that metric of 20 would be listed in the email alert, no others is that correct?

ToKnowMore · ‎10-07-2020

Hi @ITWhisperer Thanks for the prompt reply! I apologize for not making the recommended "time range" change from 3 minutes to 8 minutes as I missed that suggestion. Have updated accordingly and will advise results.

ToKnowMore · ‎10-07-2020

Hi @ITWhisperer Yuppers, changed the percent trigger as directed. I received email alerts where the 75 percent threshold was met or surpassed. I was surprised how many I received over night. To mitigate the early AM email alerts in particular, may I include a "where count > 100"? This would be in addition to the trigger alert of "search percent>=80"? I ask this because reviewing the emails received in the early AM hours revealed that ANY reason count did not meet or exceed 100. Is there a way to single out the top reason to have at least a hit count of 100? I modified the search string with the following to see what results would render in addition to lowering ther percent trigger threshold to 20 to see what email alert values would be generated. At present, I have not received any email alert, so I clearly did not use the "where count > 20" command properly. index=firewall host=156.33.226.83 earliest="-7m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | where count > 20 | head 20 The introduction of "where count > 20" broke the alert, and I say this because no alert has since been generated. I performed a manual 3 minute window check of top reasons, please see below. There are counts above 20 and percent above 20, so was hoping an alert would be generated, but that did not happen. How can the TOP reason count be taken into consideration in that only take action if that TOP reason account is above 100 (arbitrary number) while maintaining the trigger threshold of 80? What adjustment is need in the search string? There is light at the end of the tunnel! Thanks for your help!!

ToKnowMore · ‎10-06-2020

Hi @ITWhisperer Thanks for your responses! Made requested adjustments. I am getting email alerts, however, the highest percent figure is under the trigger threshold of 75%, please see trigger setting (blue arrow). Should the trigger be percent >= 75 ? Below is an example of an email alert received with the trigger being percent>=75% Your time and input is appreciated. Thanks!

ToKnowMore · ‎10-06-2020

@ITWhisperer Update. Though I increased the trigger threshold to 75%, I am receiving an alert about every minute, similar to the one below: Also, it the time range of 3 minutes seen in the alert configuration somewhat in conflict of what's in the search field? Sorry for the tremendous amount of questions in the communique and my last post to you. Thank you.

ToKnowMore · ‎10-06-2020

Hi @ITWhisperer Thanks for your response. No worries at all. I've made progress working with you and am appreciative. I made the adjustment you suggested but tweaked the enlarged number seen below: index=firewall host=10.10.10.10 earliest="-6m@m" latest="-3m@m" | bin span=2m _time | stats count by _time, reason | eventstats sum(count) as total by _time | eval percent=count * 100 / total | sort - percent | head 20 The reason for the change was the following example email alert below, I found the results going back too far in time: WIth the adjustment mentioned above, I then received an alert with the following result, please note the alert trigger time versus time results reported. I think it looks better. I upped the trigger percent threshold to 75% so that I do not get frequent emails as I have been with the setting at 20%. Sorry, from the image directly above, how is the total getting calculated? Also what refinements if any do you recommend? Also, is it unrealistic of me to do a manual 3 minutes window check at any given time to compare the automated email alert received if and when one gets generated? Does that make sense? Thanks for HELP thus far!

ToKnowMore · ‎10-06-2020

@ITWhisperer Hi ITWhisperer, I implemented your suggestions and then made a few adjustments, currently have the following syntax: index=firewall host=10.10.10.10 earliest="-75m@m" latest="-2m@m" | bin span=2m _time | stats count by _time, reason | top limit=20 reason Purposely have the trigger threshold set low - 20% The email generated from the string above is seen below. The count under the count column did increase, but I am bewildered that there is no disparity between the reason codes as seen for count and percentage and/or percentage. Another example: When doing a manual check of top reasons, there is a discernable difference between count/percentage amongst the reason codes. Is there some additional info that I may provide that could aid in this effort? I saw a job inspector: Would it be worthwhile pulling info from the "search job properties" of the manual search seen above and leverage it for what I am trying to do? Below is a screen shot of the "request" portion of the "search job properties" Thoughts? Is a snickers bar needed? Your help, time and patience is appreciated!

ToKnowMore · ‎10-06-2020

Hi @ITWhisperer Sorry ITWhisperer, for whatever reason I did not get an email alert that you responded otherwise I would have replied to you immediately. As of this entry, I have implemented your latest suggestion and will advise results pending some time. Thanks again for your continued assistance!

ToKnowMore · ‎10-05-2020

Hi @ITWhisperer Hope things are well. Progress report: I have the trigger percent threshold low (30%), so I may receive the alert to see what results are rendered. As of now, I like what I am seeing as progress is being made, but I think fine tuning is still needed. I received an email alert below with the following info, please see image below. I was drawn to the "count" column with the figures being so low for a 3 minute window (if understanding the syntax correctly). For comparison, I did a manual "top value" check using the time frame of 10/5/2020 11:25am to 11:28am as the email trigger seen from the above image is that of 11:28:09. Please see image below and one can notice the count difference between the two different images: Is there away to mitigate the difference in the count reporting? If so, how? I would expect the count mount per each reason to be higher during a 3 minute window. Your time, help and response is appreciated.

ToKnowMore · ‎10-05-2020

Hi @ITWhisperer Have implemented your suggestion and will advise. Thanks for your assistance thus far as progress has definitely been made! Again will advise, so some time may go by before I provide an update.

ToKnowMore · ‎10-04-2020

Hi @ITWhisperer Progress! Thanks very much. I am getting alerts. More than I anticipated. It seems that the email is generated once the total percentage sum is more than 75%. How may I get the email to generate when the TOP reason is more than 75%? BTW, I changed from "percent>=.07" as the emails were coming in droves. Once I changed the trigger to "percent>=75%" the emails were less frequent, but I do not see the top reason hitting 75%. Please see images below. Your help is appreciated. Thanks!

ToKnowMore · ‎10-04-2020

Hi @ITWhisperer Hope this message finds you well. Sorry have had to shuttle from one place to another today. Finally got home. Thanks very much for your response. I found the provided image to be helpful! I have implemented your suggestions and Splunk didn't bark at me (I suspect as you anticipated). Will see once the event takes place early tomorrow morning and the desired associated email alert is sent. Will advise accordingly! Thanks again for your time, knowledge and patience.

ToKnowMore · ‎10-04-2020

Hello ITWhisperer, Thank you much for taking the time to respond, its much appreciated. Sorry ITWhisperer, I did not know how to appropriately incorporate your suggestions. Below is how I got the result mentioned in my original post, please see screenshots. It was a different tool that was used to see (graphically) a precipitous drop in activity. This drop took place during a 1 minute window, so a splunk search was used with a 1min before and after the reported event, hence 3 minutes window. So I am trying to see if we can receive an email alert with similar info provided in image #3 for any given time for any given reason. I feel like this is doable as I get the result I am looking for by using the steps seen in images 1 thru 3 manually. How can I automate this using a floating 3 minute window? Is it best to do a real time or a schedule (cron job) alert with a delay in running the result as mentioned out of indexing concerns? I am open to suggestions. FYI, image #4 is my feeble attempt to create a "real time alert". What steps may I follow get the desired results that I get from image #3 for a floating time period where any reason passes a 70% threshold? Again, thanks for your time, feedback and help. Below

ToKnowMore · ‎10-03-2020

Hello, I would like to create an alert based email on the following manually entered search string below. The time frame used was for a 3 minute period, say from 1:02am to 1:05am, 10/2/2020. As one can see from the result only has 3 reasons are present and that is fine as I know more would be reported if there were other reasons to be reported. index=firewall host=10.10.10.10 | top limit=20 reason Below is an example of the output: reason count percentage Idle Timeout 582 88.7197512 Transport Closing 42 6.402439 DPD Failure 32 4.878049 It is my desire to have this alert be generated any time a reason is equal to or greater than 70% for a 3 minute period. The trigger would be any reason passing that threshold percentage of 70%. I understand this is considered "rolling window triggering" as such the following document was referred to me: docs.splunk.com/Documentation/SplunkCloud/latest/Alert/DefineRealTimeAlerts#Create_a_real-time_alert_with_rolling_window_triggering That said, I did not find those instructions to be helpful for a percentage threshold trigger alert. Perhaps what I am hoping to do cannot be done. Nonetheless, I thought I would inquire with the Splunk community. FYI, we are on code 7.3.5 and have no idea when an upgrade is taking place and to what code version. Your time, help, patience and feedback is appreciated.

Posts	18
Solutions	0
Karma Given	0
Karma Received	2
Member Since	‎10-03-2020

Online Status	Offline
Date Last Visited	‎09-23-2022 12:58 AM

Help with additional criteria for an email alert t...

Create an alert email from a splunk search result

Re: An additional criteria for an email alert to b...

Help with additional criteria for an email alert t...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Re: Create an alert email from a splunk search res...

Create an alert email from a splunk search result