topic Re: Question around Alerts and Automation through Splunk in Alerting

Question around Alerts and Automation through Splunk

dnv007 — Wed, 30 Sep 2020 05:34:15 GMT

Hello!

I have multiple questions around the topic "Alerts" in Splunk. Here is what i am trying to achieve.. I am trying to automate a couple of Macros to run one after the other. For example:
1)My first Macro runs to extract data for a period of 6 months from another index(lets call this Complete_Data_index) into my new index( lets call it Data_Teir1)
2)My second macro runs on Data_Teir1, by generating additional fields along with the original fields as part of the results and collects it into a new index called Data_Tier2.
3)My third macro runs on the index Data_Tier2, where again it generates additional fields along with the original fields and the fields generated by Data_Tier2 as part of the results and collects it into a new index called Data_Tier3.

The requirement now is to generate logs that record if each macro run was successful,errorneous,partially successful etc. Basically to set up a logger to know what is happening at each stage of the Macro.

1)One of the questions I also had was with the feature "Trigger Conditions". If for some reason data was not collected onto Data_Tier1 from Complete_Data_index, and my "Trigger Condition" is set to Number of Results greater than 0.(refer screenshot). Will this trigger an alert to me indicating no data was collected?

2)Can all this be achieved just with Splunk or should I use Python to help me set up logging/loggers?

Please help and suggest!

Thanks in Advance!

Re: Question around Alerts and Automation through Splunk

vessev — Thu, 28 May 2020 10:03:47 GMT

Hi dnv007,

the trigger condition you mentioned works this way:
If your Splunk query (for which you setup this alert) would find more than "0" events (or "results") the alert would be generated.
If you change "is greater that" to "is less than" and change the value to "1" then every time an alert is generated if no events for your Splunk query is found - depending on your "Alert type scheduled/real-time" and the chosen time.
But you can do a lot with alerts. For example you can set your "trigger alert when" to "custom" there you can check for field values and more.

For your log generating problem. Use a universal forwarder for windows/linux. You can monitor file paths and therefore logfiles on text basis. If your macro is able to generate a logfile or append something to a "main" logfile (f.e. windows: Applicaton log) you can pull this and send it to splunk.

BR vess

Re: Question around Alerts and Automation through Splunk

dnv007 — Thu, 28 May 2020 11:27:37 GMT

Thanks @vessev !

On your suggestion for the logs, I meant more of setting up a debugging log file for the Macros(that i have mentioned above) to understand if the macro din run or ran and gave partial results etc. I want to set up a logger to understand how macros that i have setup has functioned. Can i achieve all this with just Splunk and its features (that i dont know about) ? or Would it be better if i set up a logger file through Python?