About JosephSwann

JosephSwann · ‎07-11-2019

Hello All, So i cloned an alert and then shortly after the cloned alert Actually fired off an email once and showed as adding it to the triggered alerts! Could my system be overloaded? The skip ratio is 0 and nothing else incredibly offensive popped out to me. What could i check to see if it's choking the alerts from firing off? I tried: index=_internal search_type=scheduled when i click on "alert _actions" summary_index is 81% of all alerts, 18% appears to be ITSI Related and only .087% is email (count of 3) in my mind i'm thinking it could be a performance issue due to all these saved searches but i'm not seeing it in the monitoring console. I guess i don't know where to look or what to do to stop the summary index from hoggin all saved search real estate anyways, really appreciate some help if possible

JosephSwann · ‎07-10-2019

Hi all, I've been banging my head up against the wall for the last day or so trying to figure out why my alerts aren't firing off any more. I'm on Splunk Version 7.1.2. All the alerts worked fine up until now. I haven't done any new configurations so it's odd that it stopped working. The environment maybe a tad bit overloaded so as troubleshooting steps i've scaled back just about everything yet it's still not working. There are only 2 servers within this setup (Search head and indexer) Neither can send email. These servers are Virtual and i notice the issue one day when I tried to perform a search and the indexer was froze up. I logged into vsphere and rebooted it and all was well... Before you go down the line of things to check here's what i've tried I've disabled all alerts except 1 test alert, i've disabled / turned down anything else on that server that could be competing for resources, I've looked into the Monitoring Console, i've checked search activity, skip search ratio, looked into every link within the Monitoring console and all looks healthy. To rule out this being an SMTP issue i tried testing and alert from the command line and THAT WORKS! *| top 5 host | sendmail to="myemail.com" ^ The email comes through every time. However with my set alerts aren't working. For each alert i have it send email + Add to Triggered alerts 1) It's not sending the SMTP alert 2) It's stating "There are no fired events for this alert" however when you perform the search behind the alert you clearly see it should be working.. I looked into the python logs and didn't see anything bad there. I'd really appreciate some solid help with this one. Thanks

JosephSwann · ‎04-08-2017

Thank you! What's funny is i put in several support tickets as well as worked directly with our splunk reps and their support engineer and I lie to you not! Not one of them knew how to do this. Now I know ITSI isn't as popular or well accepted as Enterprise security but what has happened is its bleed all real support to their money maker and us ITSI folks have a much smaller support group and knowledge base. Their is a huge knowledge gap at splunk and in general of the Splunk ninja masters vs people who copy stuff off splunk base to survive each and every day. This seems like it should of been promoted along with Itsi. . Why they don't have these apps available as apart of the configuration bundle IIke they did with the universal forwarer install asking if you want to install the windows module during setup (there giving you options, people like options! ) where you can say select a check box for sql, vmware etc if your monitoring those type of devices or logs is beyond me. It's stupid to have a product that ships in a fashion where it won't work without a great deal of other apps of knowledge and configuration. .. installing this in a search head cluster and indexer cluster is of no smaller effort seeing how you have got to make sure you identify all your TAs or apps across your enterprise. .. adding space and configuration load. This has been my toughest challenge yet but I'm thankful

JosephSwann · ‎04-03-2017

After checking permissions on file and user account running the script I saw It looked good. I simply added the file back to the script dir and tried once again, it works. So I was must have fat fingered something. Thanks @iguinn

JosephSwann · ‎03-29-2017

I'm fully aware that Linux is the "preferred" platform to use with Splunk, they only start preaching it in Splunk 101! So thank you for pointing that out Captain Obvious!! :)......However it was absolutely NO help to my problem. I'm a System Architect and have worked on many complex systems (Linux and windows) and I can tell you from experience The customer's STACK isn't always going to be Linux weather you like it or not You have to be versatile in IT! As a I disclaimer I'd like to clear up the false narrative that splunk can't be ran on Windows sigh. I've been using Splunk on a Windows Platform for over 3 Years in a distributive setup and Haven't had "M*A*N*Y avoidable regrets"... in fact I haven't had any major problems at all the entire time I've used it. Different stokes for different folks! But largely in part the luxury of choice is on the customer and their current STACK, NOT your cold war Linux vs Windows attitude... I'm sure Splunk doesn't feel the way you do about their product. @Iguinn Thanks for your response, I'll try looking into what you've said. I cannot explain in full detail what I'm doing because of the nature of my work however I think there's confusion because you don't have the full details. I can't go into them but it involves multiple subnets, enclaves and OS Platforms... Just know the proof of concept isn't being done this way because we want to. It's the only options available... Thank you

JosephSwann · ‎03-28-2017

Hi all, I'm using Splunk 6.5.1 on a Windows Platform and simply trying to get the "Run as a Script" trigger working under Alerts. I see that a version of this is now deprecated however “run as a script” is still available via Alert Triggers. What I'm trying to do now is just for proof of concept, so i'm using veryy minimal easy script and paths so I can get the understanding down. It states under trigger actions (when setting up an alert) for us to put our scripts in “C:\Splunk\bin\scripts”, when I look in the scripts directory I see nothing but *.PATH files (they all reference exe's it seems). so I created one myself called PING.Path. Within the path file, I placed the path of where the windows batch file is located: F:\SimplePing.BAT Inside the windows Batch file is just a simple ping command that pipes the results to a text file (so that I’ll know the script really fired off). Ping hostname > F:\PingStuff\PingResults.txt ^ This works fine no problem when I launching the bat file manually, but never fires when the Alert is triggered within Splunk, I’ve also tried copying the .bat file directly to the scripts directory and referenced it in the alert that didn’t work either. What am I doing wrong? I’m only using the windows batch file for proof of concept so I can grasp understanding of how. In the end we like to use Secure Copy to move files from windows to Linux with PSCP Script with something like this in the syntax: pscp local-file-name username@remote-host:/directory/name

Posts	6
Solutions	0
Karma Given	4
Karma Received	1
Member Since	‎03-28-2017

Online Status	Offline
Date Last Visited	‎03-23-2022 11:29 AM

Alerts Suddenly stopped working?!?

Can Someone Please explain in detail how to "Run a...

Re: Alerts Suddenly stopped working?!?

Alerts Suddenly stopped working?!?

Re: Splunk IT Service Intelligence: How to get on-...

Re: Can Someone Please explain in detail how to "R...

Re: Can Someone Please explain in detail how to "R...

Can Someone Please explain in detail how to "Run a...