Monitor AWS Service Status using Splunk and Syndication Input

[marcellodesales]

Problem

• Dependency on AWS Services status

If you depend on Amazon AWS service to operate, you need to keep a close eye on the status of their services. Amazon uses the website http://status.aws.amazon.com/, which provides links to RSS feeds to specific services in specific regions.

• Potential Errors

Our service uses S3, CloudFront, and other services to operate. We'd like to be informed on any service that might go down during hours of operations, and automate what we should do in case something goes wrong.

• Splunk Logging

We use Splunk for Logging all of our services.

Requirement

For instance, if errors occurs in the application while writing to S3, we'd like to know if that was caused by a potential outage in AWS.

• How to monitor the Status RSS feed in Splunk?
• Is there an HTTP client for that? A background service?

[marcellodesales]

Published first at http://stackoverflow.com/questions/39406016/monitor-aws-service-status-using-splunk

[marcellodesales · Accepted solution]

Solution

• You can use the Syndication Input app (https://splunkbase.splunk.com/app/2646/#/overview) to collect the RSS feed data from the AWS Status
• Create a query that fetches the RSS Items that have errors and stores in Splunk indexes under the syndication sourcetype.
• Create an alert based on the query, a since field so that we can adjust the alerts over time.

How

1. Ask your Splunk team to install the app "Syndication Input" on the environments you need.

2. After that, just collect each of the RSS feeds needed and add them to the Settings -> Data Input -> Syndication Feed. Take all the URLs from the Amazon Status RSS feeds and use them as Splunk Data Input, filling out the form with certain interval:

• http://status.aws.amazon.com/rss/cloudfront.rss
• http://status.aws.amazon.com/rss/s3-us-standard.rss
• http://status.aws.amazon.com/rss/s3-us-west-1.rss
• http://status.aws.amazon.com/rss/s3-us-west-2.rss

When you are finished, the Syndication App has the following:

1. Use the search for the errors when the occur, adjusting the “since” date so that you can create an alert for the results. I added a day in the past just for display purpose.

• since should be some start day you will start monitoring AWS. This helps the query to result in any new event when Amazon publishes new errors captured from the text Informational message:.
  • The query should not return anything new because the since will not return any date.
• Since the token RESOLVED is appended to a new RSS feed item, we exclude them from the alerts.

.

sourcetype=syndication "Informational message:" NOT "RESOLVED" 
   | eval since=strptime("2010-08-01", "%Y-%m-%d") 
   | eval date=strptime(published_parsed, "%Y-%m-%dT%H:%M:%SZ") 
   | rex field=summary_detail_base "rss\/(?<aws_object>.*).rss$" 
   | where date > since 
   | table aws_object, published_parsed, id, title, summary 
   | sort -published_parsed

1. Create an Alert with the Query. For instance, to send an email: