Splunk For AWS Problem

[bruceav]

When I try running the Splunk For AWS app I get the following error:

Splunk cannot find the "AWSCloudTrail-overview" view.

As far as I know the aws.conf is configured correctly and my Cloudtrail bucket is configured correctly. What am I missing?

[zsanaa]

I am having trouble in configuring the AWS Cloudtrail overview in the splunk app for AWS. Not all dashboards are being populated.
I am unable to see Network Configuration actions & Start/stop instances.
I have tried all solutions mentioned above.

Thanks in advance

[atanasoffa]

it appears the reason for these errors was because of a misconfiguration between the data input and the aws-cloudtrail stanza in inputs.conf - no issues with the aws-cloudtrail.py script. Thank you for your time and help!

[atanasoffa]

Thanks for your reply. Here are some of my errors after I applied your suggestion:

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" File "/apps/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py", line 219, in run

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" logging.debug("reading message with id %s at %s",envelope["MessageId"],envelope["Timestamp"])

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'MessageId'

I added in a debug line and I do get similar output as you, just in a different order (the "s3bucket" object and value is before the s3ObjectKey) but then I get the errors above...

[grinabms]

That's the only change that I made. Can you post your error message?

One suggestion is to add a debugging line to see exactly what is in the "envelope"... here is how it should look:
logging.info("envelope: %s",json.dumps(envelope))
#message = json.loads(envelope["Message"])
message = envelope

When you save the edit, then your splunkd.log file should contain log entries like this:

03-25-2014 23:25:54.726 +0000 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" envelope: {"s3ObjectKey": ["AWSLogs/123412341234/CloudTrail/us-east-1/2014/03/24/123412341234_CloudTrail_us-east-1_20140324T1645Z_pUiRsGvGTkwgBOoL.json.gz"], "s3Bucket": "my-log-bucket"}

[grinabms]

I got my Cloudtrail logs into SplunkAppforAWS with a small change in aws-cloudtrail.py.

Background: Cloudtrail data wasn't feeding into my dashboards, and I saw a steady stream of errors in $SPLUNK_HOME/var/log/splunk/splunkd.log. Same error message:

03-10-2014 04:53:56.015 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'Message'

The solution was to edit $SPLUNK_HOME/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py. I commented out one line, and replaced it with another. Now this appears about 200 lines down in my file:

    #message = json.loads(envelope["Message"])
    message = envelope

Make this change, and in a few minutes, the errors in the splunkd.log disappear, and data begins to populate the dashboards.

Hope this helps.
-Pete

[atanasoffa]

Did you change anything else in the script? I tried your suggestion and it produced same type of error for MessageId..

[nkhetia]

Hi Bruce,

Sure, lets go through the checklist once again in order to verify your setup. Before we do that, can you shoot me an email to nkhetia@splunk.com, so that i can send you some sample screenshots ?

• Remove cloudtrail setup entry which is already there from last week.
• add new configuration using same IAM user credentials
• make sure IAM user is power/admin user who has all grants
• SQS region and queue name should be identical to one which you setup manually
• Also while configuring CloudTrail inputs, specify following things:

Select More Settings checkbox.

Set Source type as Manual and specify "aws-cloudtrail" as Source type.

Under index, select destination index as "aws-cloudtrail".

• In Splunk search bar, try searching for events by index=*, and see if you see any json data.

• You can also try ingesting CloudTrail data using cloudtrail2splunk.py under bin folder. Please refer USAGE.txt to use the same.

• Have you tried setting up aws.conf for Billing data ? if so, do you see any data coming in under Billing & Usage dashboards?

Thanks

Nilesh

[nkhetia]

Hi Bruce, could you send your contact details to nkhetia@splunk.com ? I will try and setup webex to troubleshoot it.

thanks

Nilesh

[bruceav]

Hi Nilesh, took a break last Friday on troubleshooting this issue to concentrate on other issues at work, and to relieve my frustration that this isn't working yet, but hopefully you can help me get this working today.
Still in same situation where my "AWS Cloudtrail Log" seems to be configured correctly but I'm still not getting any of the messages from the SQS to Splunk and the SQS has over 500 messages now. Any suggestions?

[nkhetia]

It uses same port. It could be api call to aws are blocked. Can you try using cloudtrail2splunk.py under bin folder? Its manual way to ingest cloudtrail data in splunk. You can refer to USAGE.txt.

To use billing & usage, aws.conf needs to be configured. Please refer to README.txt. If it is getting data, api call to aws are not blocked.

thx
Nilesh

[bruceav]

I appreciate that but unfortunately we are not allowed to have any type of VTC connections from where I work. The documentation doesn't say anything about port being used by Splunk for AWS, does it use a separate port or is it going out on the same port the Splunk uses? Just curious if my firewall may be blocking Splunk for AWS.

[nkhetia]

Hi Bruce,

• If you are using credentials of IAM user, that IAM user should have enough permissions to access S3 data.
• Do you see messages queued up under SQS in AWS Management Console ?
• Also while configuring CloudTrail inputs, have you specified following things ?

Select More Settings checkbox.

Set Source type as Manual and specify aws-cloudtrail as Source type.
Under index, select destination index as aws-cloudtrail.

thx

Nilesh

[bruceav]

So it was not subscribed to an sns topic but now it is, thanks for that hint, but I am getting messages in the SQS but still nothing in the app.

[nkhetia]

if there are no messages in SQS, make sure it is subscribed to correct sns topic. Please check this link : http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html

Under manual sourcetype, specify "aws-cloudtrail".

thx

Nilesh

[bruceav]

The IAM user has AWSCloudTrailFullAccess under Permissions, as for the SQS there are no messages.
I set Sourcetype to manual but don't know what I should put in the "Source Type" field.

[nkhetia]

There are two portions of this app.

• Billing and Usage

• CloudTrail

aws.conf is used for Billing and usage portion of the app. and AWS CloudTrail inputs under settings-> Data inputs is used for CloudTrail.

Have you configured AWS CloudTrail inputs under settings->Data inputs ?

Thanks

Nilesh

[bruceav]

Hi Nilesh
I upgraded to Splunk 6.0 and I now have the aws-cloudtrail data input, I configured it with the Key ID, Secret Key, SQS Queue Name and region, I then ran the Splunk for AWS app but get "No results found" on all of the panels, I did configure the s3 bucket and when I go to it I can see that it is populated with logs but the Splunk for AWS apparently is not connecting to it. What's worse is that there is nothing in the logs to indicate if there is a problem. Suggestions?

[nkhetia]

yes .. it requires splunk 6.0.

thanks
Nilesh

[bruceav]

I wish I could skype but I'm not allowed to install Skype in the office workstation. I did however go back to check my installation and noticed that the files were owned by root so I chowned them to splunk, that however did not fix the problem. As I restarted Splunk I noticed that there were several errors popping on the screen with the message "Possible typo in stanza [aws-cloudtrail] in $SPLUNK_HOME/etc/apps/SplunkforAWS/default/inpiuts.conf"
I think it may have to do with the version of Splunk I'm running (4.3.1) So I'm going to update my Splunk and try again.

[nkhetia]

If you do not see AWS CloudTrail Log type under Settings -> Data inputs, there could be installation issue with AWS App.

If you are online, skype me on nkhetia@hotmail.com and we can figure it out, real quick.

thanks
Nilesh