I'm having a weird problem with the AWS app that I simply can't figure out. I am seeing different numbers of instances reported by sourcetype=aws:config
and sourcetype=aws:description.
I would have thought they would be the same but they're not. After filtering for running state, not deleted and so on, aws:description is showing me 100 running instances at the present moment--which the AWS admins tell me is correct--but aws:config has only 89 collected over all time. I do not see how this is even possible. And the dropdown in 'Individual EC2 Instance' only shows around 40.
I've checked in the AWS console and the config dashboard there is also showing the correct number--100. Somehow Splunk is throwing away 11 instances, but I can't see where or how. Plainly the SNS/SQS plumbing is working or I wouldn't be seeing any config items at all.
Has anyone seen this before?
The CLI command did the trick. It seems that the Add-on did not gather up all the config data when it was first set up, possibly because the AWS policy wasn't set up 100% correctly to begin with--it was done by another team who took a while to get the hang of what was required.
I am now seeing the expected drill-down information from 'Usage Overview' into 'Individual EC2 Instance' which was previously missing due to incomplete config data.
It may be that it is a good idea to run this CLI command after implementing the AWS add-on, to ensure that Splunk has all the configuration records. Certainly recommended to anyone else who comes across this issue.
Thank you for your assistance jzhong
you are welcome
There might be couple reasons:
With the latest AWS App(v4.1.1) and TA(v3.0.0), we increase the use of description API, and only use config data in the topology view. If you have instances across regions, it's quite a lot configuration work to setup all config services with S3/SNS/SQS. So you can trust the data from aws:description.
Hope it helps.
Only one active region at the moment.
How do you manually take a config snapshot? How do you tell if one's been taken or when?
How can you tell if the SQS is used by some other facility?
In the AWS App configuration UI, when you add Config data source, the app will automatically trigger a Config snapshot for you. You can see the timestamp of last snapshot. You can manually trigger config snapshot via CLI: aws configservice deliver-config-snapshot deliver-config-snapshot
There is no easy way to check whether a queue is used by others. I sometimes just create a new queue, subscribe to the SNS topic and use that new queue in my app
"You can manually trigger config snapshot via CLI: aws configservice deliver-config-snapshot deliver-config-snapshot"
When I run this command I get an error saying "--delivery-channel-name is required"
Any chance you could show me an example of how I'm supposed to run the command?