All Apps and Add-ons

Splunk For AWS Problem

bruceav
New Member

When I try running the Splunk For AWS app I get the following error:

Splunk cannot find the "AWSCloudTrail-overview" view.

As far as I know the aws.conf is configured correctly and my Cloudtrail bucket is configured correctly. What am I missing?

Tags (1)
0 Karma

zsanaa
New Member

I am having trouble in configuring the AWS Cloudtrail overview in the splunk app for AWS. Not all dashboards are being populated.
I am unable to see Network Configuration actions & Start/stop instances.
I have tried all solutions mentioned above.

Thanks in advance

0 Karma

atanasoffa
Explorer

it appears the reason for these errors was because of a misconfiguration between the data input and the aws-cloudtrail stanza in inputs.conf - no issues with the aws-cloudtrail.py script. Thank you for your time and help!

0 Karma

atanasoffa
Explorer

Thanks for your reply. Here are some of my errors after I applied your suggestion:

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" File "/apps/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py", line 219, in run

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" logging.debug("reading message with id %s at %s",envelope["MessageId"],envelope["Timestamp"])

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'MessageId'

I added in a debug line and I do get similar output as you, just in a different order (the "s3bucket" object and value is before the s3ObjectKey) but then I get the errors above...

0 Karma

grinabms
Explorer

That's the only change that I made. Can you post your error message?

One suggestion is to add a debugging line to see exactly what is in the "envelope"... here is how it should look:
logging.info("envelope: %s",json.dumps(envelope))
#message = json.loads(envelope["Message"])
message = envelope

When you save the edit, then your splunkd.log file should contain log entries like this:

03-25-2014 23:25:54.726 +0000 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" envelope: {"s3ObjectKey": ["AWSLogs/123412341234/CloudTrail/us-east-1/2014/03/24/123412341234_CloudTrail_us-east-1_20140324T1645Z_pUiRsGvGTkwgBOoL.json.gz"], "s3Bucket": "my-log-bucket"}

grinabms
Explorer

I got my Cloudtrail logs into SplunkAppforAWS with a small change in aws-cloudtrail.py.

Background: Cloudtrail data wasn't feeding into my dashboards, and I saw a steady stream of errors in $SPLUNK_HOME/var/log/splunk/splunkd.log. Same error message:

03-10-2014 04:53:56.015 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'Message'

The solution was to edit $SPLUNK_HOME/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py. I commented out one line, and replaced it with another. Now this appears about 200 lines down in my file:

    #message = json.loads(envelope["Message"])
    message = envelope

Make this change, and in a few minutes, the errors in the splunkd.log disappear, and data begins to populate the dashboards.

Hope this helps.
-Pete

0 Karma

atanasoffa
Explorer

Did you change anything else in the script? I tried your suggestion and it produced same type of error for MessageId..

0 Karma

nkhetia
Path Finder

Hi Bruce,

Sure, lets go through the checklist once again in order to verify your setup. Before we do that, can you shoot me an email to nkhetia@splunk.com, so that i can send you some sample screenshots ?

  • Remove cloudtrail setup entry which is already there from last week.
  • add new configuration using same IAM user credentials
  • make sure IAM user is power/admin user who has all grants
  • SQS region and queue name should be identical to one which you setup manually
  • Also while configuring CloudTrail inputs, specify following things:

Select More Settings checkbox.

Set Source type as Manual and specify "aws-cloudtrail" as Source type.

Under index, select destination index as "aws-cloudtrail".

  • In Splunk search bar, try searching for events by index=*, and see if you see any json data.

  • You can also try ingesting CloudTrail data using cloudtrail2splunk.py under bin folder. Please refer USAGE.txt to use the same.

  • Have you tried setting up aws.conf for Billing data ? if so, do you see any data coming in under Billing & Usage dashboards?

Thanks

Nilesh

0 Karma

nkhetia
Path Finder

Hi Bruce, could you send your contact details to nkhetia@splunk.com ? I will try and setup webex to troubleshoot it.

thanks

Nilesh

0 Karma

bruceav
New Member

Hi Nilesh, took a break last Friday on troubleshooting this issue to concentrate on other issues at work, and to relieve my frustration that this isn't working yet, but hopefully you can help me get this working today.
Still in same situation where my "AWS Cloudtrail Log" seems to be configured correctly but I'm still not getting any of the messages from the SQS to Splunk and the SQS has over 500 messages now. Any suggestions?

0 Karma

nkhetia
Path Finder

It uses same port. It could be api call to aws are blocked. Can you try using cloudtrail2splunk.py under bin folder? Its manual way to ingest cloudtrail data in splunk. You can refer to USAGE.txt.

To use billing & usage, aws.conf needs to be configured. Please refer to README.txt. If it is getting data, api call to aws are not blocked.

thx
Nilesh

0 Karma

bruceav
New Member

I appreciate that but unfortunately we are not allowed to have any type of VTC connections from where I work. The documentation doesn't say anything about port being used by Splunk for AWS, does it use a separate port or is it going out on the same port the Splunk uses? Just curious if my firewall may be blocking Splunk for AWS.

0 Karma

nkhetia
Path Finder

Hi Bruce,

  • If you are using credentials of IAM user, that IAM user should have enough permissions to access S3 data.
  • Do you see messages queued up under SQS in AWS Management Console ?
  • Also while configuring CloudTrail inputs, have you specified following things ?

Select More Settings checkbox.

Set Source type as Manual and specify aws-cloudtrail as Source type.
Under index, select destination index as aws-cloudtrail.

thx

Nilesh

bruceav
New Member

So it was not subscribed to an sns topic but now it is, thanks for that hint, but I am getting messages in the SQS but still nothing in the app.

0 Karma

nkhetia
Path Finder

if there are no messages in SQS, make sure it is subscribed to correct sns topic. Please check this link : http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html

Under manual sourcetype, specify "aws-cloudtrail".

thx

Nilesh

0 Karma

bruceav
New Member

The IAM user has AWSCloudTrailFullAccess under Permissions, as for the SQS there are no messages.
I set Sourcetype to manual but don't know what I should put in the "Source Type" field.

0 Karma

nkhetia
Path Finder

There are two portions of this app.

  • Billing and Usage

  • CloudTrail

aws.conf is used for Billing and usage portion of the app. and AWS CloudTrail inputs under settings-> Data inputs is used for CloudTrail.

Have you configured AWS CloudTrail inputs under settings->Data inputs ?

Thanks

Nilesh

bruceav
New Member

Hi Nilesh
I upgraded to Splunk 6.0 and I now have the aws-cloudtrail data input, I configured it with the Key ID, Secret Key, SQS Queue Name and region, I then ran the Splunk for AWS app but get "No results found" on all of the panels, I did configure the s3 bucket and when I go to it I can see that it is populated with logs but the Splunk for AWS apparently is not connecting to it. What's worse is that there is nothing in the logs to indicate if there is a problem. Suggestions?

0 Karma

nkhetia
Path Finder

yes .. it requires splunk 6.0.

thanks
Nilesh

0 Karma

bruceav
New Member

I wish I could skype but I'm not allowed to install Skype in the office workstation. I did however go back to check my installation and noticed that the files were owned by root so I chowned them to splunk, that however did not fix the problem. As I restarted Splunk I noticed that there were several errors popping on the screen with the message "Possible typo in stanza [aws-cloudtrail] in $SPLUNK_HOME/etc/apps/SplunkforAWS/default/inpiuts.conf"
I think it may have to do with the version of Splunk I'm running (4.3.1) So I'm going to update my Splunk and try again.

0 Karma

nkhetia
Path Finder

If you do not see AWS CloudTrail Log type under Settings -> Data inputs, there could be installation issue with AWS App.

If you are online, skype me on nkhetia@hotmail.com and we can figure it out, real quick.

thanks
Nilesh

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...