Splunk App for AWS - Topology Dashboard - config events not parsing as expected

[dflodstrom]

I'm having issues troubleshooting the the Topology dashboard in the Splunk App for AWS. I've got my config input configured and am receiving config history events but it looks like the JSON aws:config events have a top level that isn't expected and so topology searches aren't returning any results. Fields like resourceId are nested as configurationItems{}.resourceId. I can use | rename configurationItems{}.* as * to strip configuratioitems{} from my fields but even then the topology history generating search returns a table where every field is multivalued and every row represents multiple resourceId's

I'm using the latest versions of the app and the TA and my config input is configured using the SQS based S3 config input. I've successfully configured this app in my lab environment/VPC but the config events didn't look like they do in production.

I'm just looking for some advice and/or hoping someone has seen something similar in their environment.

[dflodstrom]

We see some global assets like IAM details for our account and it's group and policy and our load balancers. So that gives you an idea that it is somewhat functional. The app is unable to parse out our VPCs, that dropdown returns no results.

[dflodstrom]

Here's an example of what our config events look like in production.

{"fileVersion":"1.0","configurationItems":[{"relatedEvents":[],"relationships":[{"resourceId":"eni-xxxxxxxx","resourceType":"AWS::EC2::NetworkInterface","name":"Contains NetworkInterface"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"subnet-xxxxxxxx","resourceType":"AWS::EC2::Subnet","name":"Is contained in Subnet"},{"resourceId":"vol-xxxxxxxx","resourceType":"AWS::EC2::Volume","name":"Is attached to Volume"},{"resourceId":"vol-xxxxxxxx","resourceType":"AWS::EC2::Volume","name":"Is attached to Volume"},{"resourceId":"vpc-xxxxxxxx","resourceType":"AWS::EC2::VPC","name":"Is contained in Vpc"}],"configuration":{"amiLaunchIndex":0,"imageId":"ami-xxxxxxxxxxxx","instanceId":"i-xxxxxxxxxx","instanceType":"r4.large","keyName":"<Value>","launchTime":"2019-01-24T19:21:59.000Z","monitoring":{"state":"enabled"},"placement":{"availabilityZone":"us-east-1resourceValue","groupName":"","tenancy":"default"},"privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx","productCodes":[],"publicDnsName":"","state":{"code":16,"name":"running"},"stateTransitionReason":"","subnetId":"subnet-0bea2f017db0fe8b9","vpcId":"vpc-xxxxxxxxxx","architecture":"x86_64","blockDeviceMappings":[{"deviceName":"/dev/sda1","ebs":{"attachTime":"2019-01-24T19:22:00.000Z","deleteOnTermination":true,"status":"attached","volumeId":"vol-xxxxxxxxxxxx"}},{"deviceName":"/dev/sdk","ebs":{"attachTime":"2019-01-24T19:22:00.000Z","deleteOnTermination":true,"status":"attached","volumeId":"vol-xxxxxxxxxxxx"}}],"clientToken":"xxxxxxxx","ebsOptimized":false,"enaSupport":true,"hypervisor":"xen","iamInstanceProfile":{"arn":"arn:arnvalue","id":"AIPAJTUJJMCJC3H2CNWUE"},"instanceLifecycle":"spot","networkInterfaces":[{"attachment":{"attachTime":"2019-01-24T19:21:59.000Z","attachmentId":"eni-xxxxxxxx","deleteOnTermination":true,"deviceIndex":0,"status":"attached"},"description":"","groups":[{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"}],"ipv6Addresses":[],"macAddress":"xxx.xxx.xxx.xxx","networkInterfaceId":"eni-xxxxxxxx","ownerId":"xxxxxxxx","privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx","privateIpAddresses":[{"primary":true,"privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx"}],"sourceDestCheck":true,"status":"in-use","subnetId":"subnet-0bea2f017db0fe8b9","vpcId":"vpc-xxxxxxxxxx"}],"rootDeviceName":"/dev/sda1","rootDeviceType":"ebs","securityGroups":[{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"}],"sourceDestCheck":true,"spotInstanceRequestId":"sir-xxxxxxx","tags":[{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue:aws:cloudformation:us-east-1resourceValue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"}],"virtualizationType":"hvm","cpuOptions":{"coreCount":1,"threadsPerCore":2}},"supplementaryConfiguration":{},"tags":{"XXXXXXXX":"im0021","aws:autoscaling:groupName":"groupName","aws:cloudformation:stack-name":"SymphonyComputeNodeTier-AMI-xxxxxxxxxxxx","aws:cloudformation:stack-id":"arn:arnvalue","XXXXXXXX":"ubuntu18","XXXXXXXX":"sf","XXXXXXXX":"xxxxxxxx","aws:cloudformation:logical-id":"Group","XXXXXXXX":"internal-poc","kubernetes.io/cluster/LOGICAL_NULL":"owned","XXXXXXXX":"private","Name":"ComputeNode-im0021-test"},"configurationItemVersion":"1.3","configurationItemCaptureTime":"2019-01-24T19:23:33.648Z","configurationStateId":XXXXXXXX"awsAccountId":"XXXXXXXXXXXX","configurationItemStatus":"ResourceDiscovered","resourceType":"AWS::EC2::Instance","resourceId":"i-xxxxxxxx","ARN":"arn:arnvalue","awsRegion":"us-east-1","availabilityZone":"us-east-1resourceValue","configurationStateMd5Hash":"","resourceCreationTime":"2019-01-24T19:21:59.000Z"},{"relatedEvents":[],"relationships":[],"supplementaryConfiguration":{},"tags":{},"configurationItemVersion":"1.3","configurationItemCaptureTime":"2019-01-24T19:53:39.437Z","configurationStateId":XXXXXXXX"awsAccountId":"XXXXXXXXXXXX","configurationItemStatus":"ResourceDeleted","resourceType":"AWS::EC2::Instance","resourceId":"i-xxxxxxxx","ARN":"arn:arnvalue","awsRegion":"us-east-1","configurationStateMd5Hash":""},{"relatedEvents":[],"relationships":[{"resourceId":"eni-xxxxxxxx","resourceType":"AWS::EC2::NetworkInterface","name":"Contains NetworkInterface"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"sg-xxxxxxxx","resourceType":"AWS::EC2::SecurityGroup","name":"Is associated with SecurityGroup"},{"resourceId":"subnet-xxxxxxxx","resourceType":"AWS::EC2::Subnet","name":"Is contained in Subnet"},{"resourceId":"vol-xxxxxxxx","resourceType":"AWS::EC2::Volume","name":"Is attached to Volume"},{"resourceId":"vol-xxxxxxxx","resourceType":"AWS::EC2::Volume","name":"Is attached to Volume"},{"resourceId":"vpc-xxxxxxxx","resourceType":"AWS::EC2::VPC","name":"Is contained in Vpc"}],"configuration":{"amiLaunchIndex":0,"imageId":"ami-xxxxxxxxxxxx","instanceId":"i-xxxxxxxxxx","instanceType":"r4.large","keyName":"<Value>","launchTime":"2019-01-24T20:02:38.000Z","monitoring":{"state":"enabled"},"placement":{"availabilityZone":"us-east-1resourceValue","groupName":"","tenancy":"default"},"privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx","productCodes":[],"publicDnsName":"","state":{"code":16,"name":"running"},"stateTransitionReason":"","subnetId":"subnet-0ced7ba8f9f4be8ab","vpcId":"vpc-xxxxxxxxxx","architecture":"x86_64","blockDeviceMappings":[{"deviceName":"/dev/sda1","ebs":{"attachTime":"2019-01-24T20:02:39.000Z","deleteOnTermination":true,"status":"attached","volumeId":"vol-xxxxxxxxxxxx"}},{"deviceName":"/dev/sdk","ebs":{"attachTime":"2019-01-24T20:02:39.000Z","deleteOnTermination":true,"status":"attached","volumeId":"vol-xxxxxxxxxxxx"}}],"clientToken":"xxxxxxxx","ebsOptimized":false,"enaSupport":true,"hypervisor":"xen","iamInstanceProfile":{"arn":"arn:arnvalue","id":"XXXXXXXXXXXX"},"instanceLifecycle":"spot","networkInterfaces":[{"attachment":{"attachTime":"2019-01-24T20:02:38.000Z","attachmentId":"eni-xxxxxxxx","deleteOnTermination":true,"deviceIndex":0,"status":"attached"},"description":"","groups":[{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"}],"ipv6Addresses":[],"macAddress":"xxx.xxx.xxx.xxx","networkInterfaceId":"eni-xxxxxxxx","ownerId":"xxxxxxxx","privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx","privateIpAddresses":[{"primary":true,"privateDnsName":"ip-xxxxxxxx","privateIpAddress":"xxx.xxx.xxx.xxx"}],"sourceDestCheck":true,"status":"in-use","subnetId":"subnet-0ced7ba8f9f4be8ab","vpcId":"vpc-xxxxxxxxxx"}],"rootDeviceName":"/dev/sda1","rootDeviceType":"ebs","securityGroups":[{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"},{"groupName":"groupName","groupId":"sg-xxxxxxxxxxxxxxxxx"}],"sourceDestCheck":true,"spotInstanceRequestId":"sir-254i5fpg","tags":[{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue:aws:cloudformation:us-east-1resourceValue"},{"key":"<keyvalue>","value":"privateresourcevalue"},{"key":"<keyvalue>","value":"privateresourcevalue"}],"virtualizationType":"hvm","cpuOptions":{"coreCount":1,"threadsPerCore":2}},"supplementaryConfiguration":{},"tags":{"XXXXXXXX":"im0021","aws:autoscaling:groupName":"groupName","aws:cloudformation:stack-name":"XXXXXXXX-AMI-xxxxxxxxxxxx","aws:cloudformation:stack-id":"arn:arnvalue","XXXXXXXX":"sf","XXXXXXXX":"ubuntu18","XXXXXXXX":"xxxxxxxx","XXXXXXXX":"internal-poc","aws:cloudformation:logical-id":"Group","XXXXXXXX":"private","kubernetes.io/cluster/LOGICAL_NULL":"owned","Name":"ComputeNode-im0021-test"},"configurationItemVersion":"1.3","configurationItemCaptureTime":"2019-01-24T20:04:10.190Z","configurationStateId":XXXXXXXX"awsAccountId":"XXXXXXXXXXXX","configurationItemStatus":"ResourceDiscovered","resourceType":"AWS::EC2::Instance","resourceId":"i-xxxxxxxx","ARN":"arn:arnvalue","awsRegion":"us-east-1","availabilityZone":"us-east-1resourceValue","configurationStateMd5Hash"...truncated due to input box limit}

[ccl0utier]

I assume you've configured the AWS Config service in both environment in the same way (recorder, etc...)?

If so I would open a support ticket with us if not already done.

[dflodstrom]

Here's an example of what an aws:config event looks like in my lab:

{"awsRegion": "us-east-1", "relatedEvents": [], "tags": {}, "resourceType": "AWS::EC2::NetworkInterface", "relationships": [{"resourceId": "i-xxxxxxxxxxx", "resourceType": "AWS::EC2::Instance", "name": "Is attached to Instance"}, {"resourceId": "sg-xxxxxxxxxxxx", "resourceType": "AWS::EC2::SecurityGroup", "name": "Is associated with SecurityGroup"}, {"resourceId": "subnet-xxxxx", "resourceType": "AWS::EC2::Subnet", "name": "Is contained in Subnet"}, {"resourceId": "vpc-xxxxxx", "resourceType": "AWS::EC2::VPC", "name": "Is contained in Vpc"}], "configurationItemVersion": "1.3", "configurationItemStatus": "OK", "configuration": {"sourceDestCheck": true, "subnetId": "subnet-xxxxxx", "availabilityZone": "us-east-1d", "status": "in-use", "attachment": {"instanceId": "i-xxxxxx", "status": "attached", "attachTime": "2019-01-11T02:47:21.000Z", "instanceOwnerId": "812653813035", "attachmentId": "eni-attach-xxxxxxxxx", "deleteOnTermination": true, "deviceIndex": 0}, "networkInterfaceId": "eni-xxxxxxx", "privateDnsName": "ip-172-xx-xx-xx.ec2.internal", "interfaceType": "interface", "requesterManaged": false, "description": "", "macAddress": "12:ee:c2:e2:dd:0c", "ipv6Addresses": [], "vpcId": "vpc-xxxxxx", "privateIpAddress": "172.xx.xx.xx", "ownerId": "XXXXXXXXXXXX", "privateIpAddresses": [{"privateIpAddress": "172.xx.xx.xx", "primary": true, "privateDnsName": "ip-172-xx-xx-xx.ec2.internal"}], "groups": [{"groupId": "sg-xxxxxxxxxxxxxxxxx", "groupName": "launch-wizard-2"}], "tagSet": []}, "supplementaryConfiguration": {}, "configurationItemCaptureTime": "2019-01-14T16:23:04.206Z", "ARN": "arn:aws:ec2:us-east-1:xxxxxxxxxxxx:network-interface/eni-xxxxxxxxxxxxxxxx", "resourceId": "eni-xxxxxxxxxxxxxxx", "configurationStateId": 1547482984206, "configurationStateMd5Hash": "", "availabilityZone": "us-east-1d", "awsAccountId": "XXXXXXXXXXXX"}

These topology searches have no issue digesting this and anything I see in config I can see in my topology dashboard.

[ccl0utier]

Sadly, I don't have a Splunk AWS environment to compare with anymore which make this a little harder, but here's a few questions:

• I assume you have the relevant saved searches enabled to generate the topology data? See https://docs.splunk.com/Documentation/AWS/latest/Installation/Savedsearches
• If you believe the issue is from the Splunk AWS application or add-on, you can and should open a ticket with Splunk (it doesn't sound like you believe so, however)
• Alternatively, if the issue appears to come from the AWS Config service (or account) configuration, you can hopefully open a ticket with AWS

That being said, if you can provide data about your issue from both your environments (one works the other doesn't if I understand correctly), that would help us help you.

[dflodstrom]

I appreciate the comment. I've configured the app successfully in my own AWS environment and yes I've enabled the searches in my production environment to make the dashboard work. The issue I'm facing is that the searches populating the snapshot indexes do not work with the config data we're receiving. I've definitely opened a support case but figured I'd try my luck here while support ... does their thing.