Alerting

How to get alert when any input from TA stops working?

neerajs_81
Builder

Noob question, can someone pls assist how to get alert when any of the inputs under any TA (Add-on) stops sending logs for last 24 h ?   Lets take Splunk Add-on for AWS as an example.  I have  around 60 inputs configured.  How do i write a search that can alert when either of these 60 inputs stop working ?  if i do a "index=aws" , it shows me various sources and sourcetypes but there isn't any field that has names of these Inputs .

We have run into issues that we had to manually enable/disable inputs quite frequently when they stop logging.

neerajs_81_0-1669530490469.png

 

Labels (1)
Tags (2)
0 Karma
1 Solution

inventsekar
SplunkTrust
SplunkTrust

Hi @neerajs_81 ...

if i do a "index=aws" , it shows me various sources and sourcetypes but there isn't any field that has names of these Inputs.

The "name of the monitor" is simply a basic property of a log source. we should not use the name of the monitor to identify a log events. 

As a new splunk user, i can understand your view, but, splunk provides more clear methods to solve your issue. 

As you said, "if i do a "index=aws" , it shows me various sources and sourcetypes".. you should find out which source and sourcetype you are looking for the alerting. 

Actually On production systems we will need to use many fields to filter out the log events... for example , the sourcetypes...aws:billing, aws:s3.. and then, we may need to use other fields and create the alert. 

On the Splunk_TA_aws, you can select the 4th tab "Health Check", the drop down will list.. Health Overview and S3 Inputs Health Details.. when you select "S3 Inputs Health Details".. it will show you a big dashboard with multiple panels.. the first dashboard panel "SQS-Based S3 Time Lapse between Log Creation and Indexing".. you can click on the "Open in Search".. which will open a search window with the search query pre-loaded ..

index="_internal" (datainput="*")(host="*") sourcetype="aws:sqsbaseds3:log" (message="Sent data for indexing.")
| eval delay=_time-strptime(last_modified, "%Y-%m-%dT%H:%M:%S%Z")
| eval delay = if (delay<0,0,delay)
| timechart eval(round(mean(delay),3)) by datainput

you can save this as an alert.. so, now you can see that we got many fields like..  index, datainput, host, sourcetype, message.. with these fields, we can pin point the log events and create the alert. hope you got it.. let us know if need more details.. thanks. 

View solution in original post

neerajs_81
Builder

Wonderful thank you for the details. I can use your index=_internal query for any log source actually. 

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @neerajs_81 ...

if i do a "index=aws" , it shows me various sources and sourcetypes but there isn't any field that has names of these Inputs.

The "name of the monitor" is simply a basic property of a log source. we should not use the name of the monitor to identify a log events. 

As a new splunk user, i can understand your view, but, splunk provides more clear methods to solve your issue. 

As you said, "if i do a "index=aws" , it shows me various sources and sourcetypes".. you should find out which source and sourcetype you are looking for the alerting. 

Actually On production systems we will need to use many fields to filter out the log events... for example , the sourcetypes...aws:billing, aws:s3.. and then, we may need to use other fields and create the alert. 

On the Splunk_TA_aws, you can select the 4th tab "Health Check", the drop down will list.. Health Overview and S3 Inputs Health Details.. when you select "S3 Inputs Health Details".. it will show you a big dashboard with multiple panels.. the first dashboard panel "SQS-Based S3 Time Lapse between Log Creation and Indexing".. you can click on the "Open in Search".. which will open a search window with the search query pre-loaded ..

index="_internal" (datainput="*")(host="*") sourcetype="aws:sqsbaseds3:log" (message="Sent data for indexing.")
| eval delay=_time-strptime(last_modified, "%Y-%m-%dT%H:%M:%S%Z")
| eval delay = if (delay<0,0,delay)
| timechart eval(round(mean(delay),3)) by datainput

you can save this as an alert.. so, now you can see that we got many fields like..  index, datainput, host, sourcetype, message.. with these fields, we can pin point the log events and create the alert. hope you got it.. let us know if need more details.. thanks. 

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Index This | What travels the world but is also stuck in place?

April 2026 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Discover New Use Cases: Unlock Greater Value from Your Existing Splunk Data

Realizing the full potential of your Splunk investment requires more than just understanding current usage; it ...

Continue Your Journey: Join Session 2 of the Data Management and Federation Bootcamp ...

As data volumes continue to grow and environments become more distributed, managing and optimizing data ...