AWS App missing All Cloudwatch Metrics

[devenjarvis]

I have a distributed setup of Splunk (non-clustered) and am running the Splunk AWS App/Add-on. Overall a lot of the inputs are working, but I have persistent issues with the CloudWatch inputs. I have setup the cloudwatch inputs as instructed in the documentation, and everything looks good, but any dashboard that relies on that data is missing it, and if I do a search for 'index="*" sourcetype="aws:cloudwatch" I get no results. This tells me that no data is being received at all for this input. This is the same for two different AWS accounts that I have tried creating Cloudwatch inputs for.

Is this a known bug, or are there some troubleshooting steps I can take to diagnose why we aren't getting this data?

Thanks for any insight you may have.

EDIT: Not sure if this helps, but I found this in aws:cloudwatch:log:

2017-01-09 15:26:34,546 INFO pid=24869 tid=MainThread file=ta_aws_common.py:get_configs:129 | message="Not data collection tasks for aws_cloudwatch is discovered. Doing nothing and quitting the TA."

I'm not clear what the solution is, but I think it's telling me that it isn't sending cloudwatch data...

[Igor1984]

Hi,

Had the same problem. Go to the dimension name and remove all dimensions except the InstanceId. It should look something like this:
[{"InstanceId": ".*"}]

Hope this helps.

[hunters_splunk]

Hi devenjarvis,

Maybe you have already done it, but just a reminder that make sure that the required permissions have been configured and delegated to your AWS accounts for collecting CloudWatch data.

See documentation for details:
http://docs.splunk.com/Documentation/AWS/5.0.0/Installation/ConfigureyourAWSpermissions#Configure_Cl...

Also, it's a best practice to configure the search head tier to directly forward data to the indexer tier, regardless of clustered or not.

http://docs.splunk.com/Documentation/AWS/5.0.0/Installation/Installon-prem#Configure_the_search_head...

Hope this helps.Thanks!
Hunter

[devenjarvis]

Hello Hunter,

I appreciate the reply! Unfortunately I have already triple checked the AWS Permissions and have our search heads forwarding data to the indexer tier. Any other suggestions you might have would be appreciated!

Thanks again,
Deven

[hunters_splunk]

Hi Deven,

Which version of AWS app and AWS add-on are you using? Make sure these versions are compatible with each other.
Also, did you use add-on or app to create the data input?

Thanks!
Hunter

[devenjarvis]

Hello Hunter,

I am on the latest versions of the App and Add-on (5.0 and 4.2 respectively). I created the data input via the app.

Thanks,
Deven