unable to retrieve stats using AWS-addon, giving me invalid_dimension error

[kvanderhoeven]

I have configured the AWS-addon and I receive/retrieve data for some inputs, like config and description. I want to setup cloudwatch inputs but I receive errors:

2017-05-30 08:55:59,490 INFO pid=9948 tid=Thread-1 file=aws_credentials.py:_load_source_credentials:214 | message="fetch ec2 instance credentials"
2017-05-30 08:55:59,829 INFO pid=9948 tid=Thread-1 file=aws_credentials.py:load:182 | arn="arn:aws:sts::xxxxxxxxxxxxxxxx:assumed-role/Splunk_forwarder_role/i-038271d2ee1d389e6", expiration="2017-05-30 14:35:04+00:00", message="load credentials succeed"
2017-05-30 08:56:03,552 INFO pid=9948 tid=Thread-8 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,606 INFO pid=9948 tid=Thread-5 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,636 INFO pid=9948 tid=Thread-4 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,646 INFO pid=9948 tid=Thread-13 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,674 INFO pid=9948 tid=Thread-3 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,700 INFO pid=9948 tid=Thread-2 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,785 INFO pid=9948 tid=Thread-12 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,791 INFO pid=9948 tid=Thread-1 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,803 INFO pid=9948 tid=Thread-11 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,827 INFO pid=9948 tid=Thread-7 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,837 INFO pid=9948 tid=Thread-14 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,864 INFO pid=9948 tid=Thread-9 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,869 INFO pid=9948 tid=Thread-10 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,883 INFO pid=9948 tid=Thread-6 file=aws_cloudwatch_conf.py:_do_filter_invalid_dimensions:262 | message="test_koen:eu-west-1:AWS/EC2 total=0, valid=0, filtered=0"
2017-05-30 08:56:03,886 INFO pid=9948 tid=MainThread file=aws_cloudwatch_conf.py:list_metrics:434 | message="Finished querying metrics for metric_names=[u'CPUCreditBalance', u'CPUCreditUsage', u'CPUUtilization', u'DiskReadBytes', u'DiskReadOps', u'DiskWriteBytes', u'DiskWriteOps', u'NetworkIn', u'NetworkOut', u'NetworkPacketsIn', u'NetworkPacketsOut', u'StatusCheckFailed', u'StatusCheckFailed_Instance', u'StatusCheckFailed_System'], namespace=AWS/EC2"
2017-05-30 08:56:03,886 ERROR pid=9948 tid=MainThread file=aws_cloudwatch_conf.py:get_tasks:128 | message="No metric/dimension has been found for datainput=test_koen"
2017-05-30 08:56:03,886 INFO pid=9948 tid=MainThread file=ta_aws_common.py:get_configs:129 | message="Not data collection tasks for aws_cloudwatch is discovered. Doing nothing and quitting the TA."

Does anyone know what goes wrong here?

[shwetas]

What should be format to put in particular instance id only like 10 instance ID to limit the data collection?

[drafeld]

Having the same issue on Splunk Enterprise 6.6.1 with Add-On for AWS 4.3.0,Hi,

we're having the same situation with

Splunk Enterprise
Splunk Version: 6.6.1

Splunk Add-on for AWS
App Version: 4.3.0

[nvonkorff]

Hi kmason,

Same issue here.

Removing all other dimensions on ec2 CloudWatch input is the only way to get any metrics through. Using any other combination of dimensions results in: "No metric/dimension has been found for datainput="

This is true of RDS and WorkSpaces CloudWatch inputs as well, i.e. only keeping the Id dimension in the only way to get metrics in.

Interestingly, ELB CloudWatch input contains two dimensions by default and this works fine.

Anyone else having the same issue?

Splunk Enterprise
Splunk Version: 6.6.0
Splunk Build: 1c4f3bbe1aea

Splunk Add-on for AWS
App Version: 4.2.3
App Build: 48

[kmanson]

Changing the Dimension Names got it working, but still not exactly sure why the first list doesn't work

from this

[{"InstanceId": ".*", "AutoScalingGroupName": ".*", "InstanceType": ".*", "ImageId": ".*"}]

to this

[{"InstanceId": ".*"}]

[kmanson]

Any luck on a solution yet?