I have an app server running a custom application that is, unfortunately, a bit buggy.  This bug causes it's service to spike in CPU usage and degrade performance.  There's a fix in the works but because I can manually resolve it by restarting the service it is lower on the priority list. I currently use Splunk to send me an alert when CPU usage gets to 80% or more - this lets me get in there to do the reset before performance degrades. It looks like Splunk used to have a simple feature to run a script on the UF's /bin/ directory, which would have made this pretty simple - but it is deprecated and I assume doesn't work at all.  Now, however, we're supposed to create a custom alert action to reinvent this alert action. Following the basic directions here, I've come to find I don'thave the ability to create a new Alert Action:  Create alert actions - Splunk Documentation I can "Browse More" and view the existing ones, but there's no ability to create anything new.  Is there some sort of pre-requisite before these can be done?  It does not appear to be mentioned in this documentation if that's the case. Alternatively, does Splunk still trigger scripts even though the feature is deprecated?  The above needs learned but seems like a lot of overhead to have one specific server run net stop [service] && net start [service]. ... View more

I am new to Splunk and am teaching myself how to us it as I integrate it with my environment. I inherited an existing Splunk Enterprise instance that, at one point, apparently used to work to some degree but by the time I joined the team and took over had fallen into disuse. After getting it upgraded from 9.0 to 9.3.2, rolling out Universal Forwarders, tinkering with inputs.conf, and fixing some network issues, I found myself finally able to get Windows Event Log data into my indexer from a couple of different test machines. The inputs.conf I was using was something I had found on one of the existing machines before reinstalling the UF, and I noticed that it had a lot more stuff in it than Windows Event Log stanzas.  Some of it was suggesting it monitored stuff I was interested in right now, such as CPU utilization.  However, I noticed that exactly nothing outside of Windows Event data was ever making it across the wire, no matter how I reconfigured the inputs.conf stanzas. The one I honed in on first was the CPU utilization, and through research I discovered that when I invoke a stanza in inputs.conf it has to exist in some degree within the Settings > Data Inputs library (?) present on my Splunk instance. perfmon://CPU, perfmon://CPULoad, and perfmon://Processor were all stanzas I found online for (among other things) checking to see what % CPU utilization a target server was at.  None of them worked.  Looking into these Data Inputs, it looks like something is broken - when I select these three (as an example) Splunk's web UI throws up an error saying that "Processor is not a valid object".   Following some guidance online, I was able to make my own custom Data Input just called testCPU, pointing at a custom index I call testWindows, and basically make it a clone of CPU (taking in % Processor Time and % User Time as counters and whatnot).  For the required object, I noticed that "Processor Information" was an option I could pick rather than "Processor", so I went with that one.  I then deployed a stanza in inputs.conf that says perfmon://testCPU on one of my UFs, and it absolutely works.  My Indexer is now pulling in CPU % use information.  I suspect if I went back to the three CPU-related entries above and set it to "Processor Information", it would work and any of the existing Apps I inherited that invoke those stanzas would themselves start pulling in data through it. However, I do not know why my built-in Data Inputs are broken - it isn't just limited to the CPU ones I used as an example above.  For example, the "System" input claims "System is not a valid object" and the available objects dropdown does not have an obvious replacement (there's no "System Information" to follow the pattern above).  The "PhysicalDisk" DI claims "PhysicalDisk is not a valid object" but has nothing obvious to replace it either.  Available Memory claims "Memory" is not a valid object with no obvious replacement, etc. Does anyone know what might be going on here?  Looking at how the Stanzas are configured online the examples I see for the handful above I have looked into do in fact invoke object = "xxx" that matches the names of things my Splunk says isn't valid.  Some of these might have some obvious replacements ("Network" might be "Physical Network Card Activity" or something like that) but a lot of them don't. How should I go fix these?  My first assumption was that I would find some kind of "Objects" config file that may have clues to how these got redefined, but that wasn't the case. I have a ticket in with support, but I am broadening the scope here to see if anyone else has familiarity with something like this (and also to create something for another user with the same issue to find in the future). ... View more