I have a distributed setup of Splunk (non-clustered) and am running the Splunk AWS App/Add-on. Overall a lot of the inputs are working, but I have persistent issues with the CloudWatch inputs. I have setup the cloudwatch inputs as instructed in the documentation, and everything looks good, but any dashboard that relies on that data is missing it, and if I do a search for 'index="*" sourcetype="aws:cloudwatch" I get no results. This tells me that no data is being received at all for this input. This is the same for two different AWS accounts that I have tried creating Cloudwatch inputs for.
Is this a known bug, or are there some troubleshooting steps I can take to diagnose why we aren't getting this data?
Thanks for any insight you may have.
EDIT: Not sure if this helps, but I found this in aws:cloudwatch:log:
2017-01-09 15:26:34,546 INFO pid=24869 tid=MainThread file=taawscommon.py:getconfigs:129 | message="Not data collection tasks for awscloudwatch is discovered. Doing nothing and quitting the TA."
I'm not clear what the solution is, but I think it's telling me that it isn't sending cloudwatch data...
Maybe you have already done it, but just a reminder that make sure that the required permissions have been configured and delegated to your AWS accounts for collecting CloudWatch data.
See documentation for details:
Also, it's a best practice to configure the search head tier to directly forward data to the indexer tier, regardless of clustered or not.
Hope this helps.Thanks!
I appreciate the reply! Unfortunately I have already triple checked the AWS Permissions and have our search heads forwarding data to the indexer tier. Any other suggestions you might have would be appreciated!
Which version of AWS app and AWS add-on are you using? Make sure these versions are compatible with each other.
Also, did you use add-on or app to create the data input?
I am on the latest versions of the App and Add-on (5.0 and 4.2 respectively). I created the data input via the app.
Had the same problem. Go to the dimension name and remove all dimensions except the InstanceId. It should look something like this:
Hope this helps.