All Apps and Add-ons

AWS app: description vs config

Contributor

I'm having a weird problem with the AWS app that I simply can't figure out. I am seeing different numbers of instances reported by sourcetype=aws:config
and sourcetype=aws:description.

I would have thought they would be the same but they're not. After filtering for running state, not deleted and so on, aws:description is showing me 100 running instances at the present moment--which the AWS admins tell me is correct--but aws:config has only 89 collected over all time. I do not see how this is even possible. And the dropdown in 'Individual EC2 Instance' only shows around 40.

I've checked in the AWS console and the config dashboard there is also showing the correct number--100. Somehow Splunk is throwing away 11 instances, but I can't see where or how. Plainly the SNS/SQS plumbing is working or I wouldn't be seeing any config items at all.

Has anyone seen this before?

0 Karma

Contributor

The CLI command did the trick. It seems that the Add-on did not gather up all the config data when it was first set up, possibly because the AWS policy wasn't set up 100% correctly to begin with--it was done by another team who took a while to get the hang of what was required.

I am now seeing the expected drill-down information from 'Usage Overview' into 'Individual EC2 Instance' which was previously missing due to incomplete config data.

It may be that it is a good idea to run this CLI command after implementing the AWS add-on, to ensure that Splunk has all the configuration records. Certainly recommended to anyone else who comes across this issue.

Thank you for your assistance jzhong

0 Karma

Splunk Employee
Splunk Employee

you are welcome

0 Karma

Splunk Employee
Splunk Employee

There might be couple reasons:

  • Do you have instances in other regions? aws:description will capture all instances in all regions, but aws:config is only for those regions that you've enabled config service with proper SNS/SQS
  • Make sure you've taken Config snapshot. It should be automatically done when you add AWS account in the splunk app. Config history is incremental change, taking a snapshot will capture all existing resources, even they are created before you enable Config service
  • Make sure the SQS queue for the Config topic is not used by the other program/user.

With the latest AWS App(v4.1.1) and TA(v3.0.0), we increase the use of description API, and only use config data in the topology view. If you have instances across regions, it's quite a lot configuration work to setup all config services with S3/SNS/SQS. So you can trust the data from aws:description.

Hope it helps.

Contributor

Only one active region at the moment.
How do you manually take a config snapshot? How do you tell if one's been taken or when?
How can you tell if the SQS is used by some other facility?

0 Karma

Splunk Employee
Splunk Employee

In the AWS App configuration UI, when you add Config data source, the app will automatically trigger a Config snapshot for you. You can see the timestamp of last snapshot. You can manually trigger config snapshot via CLI: aws configservice deliver-config-snapshot deliver-config-snapshot

There is no easy way to check whether a queue is used by others. I sometimes just create a new queue, subscribe to the SNS topic and use that new queue in my app

0 Karma

Communicator

"You can manually trigger config snapshot via CLI: aws configservice deliver-config-snapshot deliver-config-snapshot"

When I run this command I get an error saying "--delivery-channel-name is required"

Any chance you could show me an example of how I'm supposed to run the command?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!