topic Re: Splunk For AWS Problem in All Apps and Add-ons

Splunk For AWS Problem

bruceav — Tue, 03 Dec 2013 17:59:45 GMT

When I try running the Splunk For AWS app I get the following error:

Splunk cannot find the "AWSCloudTrail-overview" view.

As far as I know the aws.conf is configured correctly and my Cloudtrail bucket is configured correctly. What am I missing?

Re: Splunk For AWS Problem

nkhetia — Tue, 03 Dec 2013 18:39:19 GMT

There are two portions of this app.

Billing and Usage
CloudTrail

aws.conf is used for Billing and usage portion of the app. and AWS CloudTrail inputs under settings-> Data inputs is used for CloudTrail.

Have you configured AWS CloudTrail inputs under settings->Data inputs ?

Thanks

Nilesh

Re: Splunk For AWS Problem

bruceav — Tue, 03 Dec 2013 19:57:57 GMT

Nilesh, Thanks for your response. I tried setting the Data Inputs section but nothing there shows up by the name of CloudTrail and the instructions are not specific as to whether I should create one by that name and if so what kind of input (TCP, UDP, file,etc...)

Re: Splunk For AWS Problem

nkhetia — Tue, 03 Dec 2013 20:12:05 GMT

If you do not see AWS CloudTrail Log type under Settings -> Data inputs, there could be installation issue with AWS App.

If you are online, skype me on nkhetia@hotmail.com and we can figure it out, real quick.

thanks
Nilesh

Re: Splunk For AWS Problem

bruceav — Tue, 03 Dec 2013 20:55:03 GMT

I wish I could skype but I'm not allowed to install Skype in the office workstation. I did however go back to check my installation and noticed that the files were owned by root so I chowned them to splunk, that however did not fix the problem. As I restarted Splunk I noticed that there were several errors popping on the screen with the message "Possible typo in stanza [aws-cloudtrail] in $SPLUNK_HOME/etc/apps/SplunkforAWS/default/inpiuts.conf"
I think it may have to do with the version of Splunk I'm running (4.3.1) So I'm going to update my Splunk and try again.

Re: Splunk For AWS Problem

nkhetia — Tue, 03 Dec 2013 21:34:17 GMT

yes .. it requires splunk 6.0.

thanks
Nilesh

Re: Splunk For AWS Problem

bruceav — Thu, 05 Dec 2013 18:23:31 GMT

Hi Nilesh
I upgraded to Splunk 6.0 and I now have the aws-cloudtrail data input, I configured it with the Key ID, Secret Key, SQS Queue Name and region, I then ran the Splunk for AWS app but get "No results found" on all of the panels, I did configure the s3 bucket and when I go to it I can see that it is populated with logs but the Splunk for AWS apparently is not connecting to it. What's worse is that there is nothing in the logs to indicate if there is a problem. Suggestions?

Re: Splunk For AWS Problem

nkhetia — Thu, 05 Dec 2013 18:35:10 GMT

Hi Bruce,

If you are using credentials of IAM user, that IAM user should have enough permissions to access S3 data.
Do you see messages queued up under SQS in AWS Management Console ?
Also while configuring CloudTrail inputs, have you specified following things ?

Select More Settings checkbox.

Set Source type as Manual and specify aws-cloudtrail as Source type.
Under index, select destination index as aws-cloudtrail.

thx

Nilesh

Re: Splunk For AWS Problem

bruceav — Thu, 05 Dec 2013 18:49:55 GMT

The IAM user has AWSCloudTrailFullAccess under Permissions, as for the SQS there are no messages.
I set Sourcetype to manual but don't know what I should put in the "Source Type" field.

Re: Splunk For AWS Problem

nkhetia — Thu, 05 Dec 2013 19:02:01 GMT

if there are no messages in SQS, make sure it is subscribed to correct sns topic. Please check this link : http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html

Under manual sourcetype, specify "aws-cloudtrail".

thx

Nilesh

Re: Splunk For AWS Problem

bruceav — Thu, 05 Dec 2013 19:58:10 GMT

So it was not subscribed to an sns topic but now it is, thanks for that hint, but I am getting messages in the SQS but still nothing in the app.

Re: Splunk For AWS Problem

nkhetia — Thu, 05 Dec 2013 20:14:26 GMT

Hi Bruce, could you send your contact details to nkhetia@splunk.com ? I will try and setup webex to troubleshoot it.

thanks

Nilesh

Re: Splunk For AWS Problem

bruceav — Thu, 05 Dec 2013 20:23:47 GMT

I appreciate that but unfortunately we are not allowed to have any type of VTC connections from where I work. The documentation doesn't say anything about port being used by Splunk for AWS, does it use a separate port or is it going out on the same port the Splunk uses? Just curious if my firewall may be blocking Splunk for AWS.

Re: Splunk For AWS Problem

nkhetia — Thu, 05 Dec 2013 20:32:47 GMT

It uses same port. It could be api call to aws are blocked. Can you try using cloudtrail2splunk.py under bin folder? Its manual way to ingest cloudtrail data in splunk. You can refer to USAGE.txt.

To use billing & usage, aws.conf needs to be configured. Please refer to README.txt. If it is getting data, api call to aws are not blocked.

thx
Nilesh

Re: Splunk For AWS Problem

bruceav — Mon, 09 Dec 2013 14:27:35 GMT

Hi Nilesh, took a break last Friday on troubleshooting this issue to concentrate on other issues at work, and to relieve my frustration that this isn't working yet, but hopefully you can help me get this working today.
Still in same situation where my "AWS Cloudtrail Log" seems to be configured correctly but I'm still not getting any of the messages from the SQS to Splunk and the SQS has over 500 messages now. Any suggestions?

Re: Splunk For AWS Problem

nkhetia — Mon, 09 Dec 2013 19:03:21 GMT

Hi Bruce,

Sure, lets go through the checklist once again in order to verify your setup. Before we do that, can you shoot me an email to nkhetia@splunk.com, so that i can send you some sample screenshots ?

Remove cloudtrail setup entry which is already there from last week.
add new configuration using same IAM user credentials
make sure IAM user is power/admin user who has all grants
SQS region and queue name should be identical to one which you setup manually
Also while configuring CloudTrail inputs, specify following things:

Select More Settings checkbox.

Set Source type as Manual and specify "aws-cloudtrail" as Source type.

Under index, select destination index as "aws-cloudtrail".

In Splunk search bar, try searching for events by index=*, and see if you see any json data.
You can also try ingesting CloudTrail data using cloudtrail2splunk.py under bin folder. Please refer USAGE.txt to use the same.
Have you tried setting up aws.conf for Billing data ? if so, do you see any data coming in under Billing & Usage dashboards?

Thanks

Nilesh

Re: Splunk For AWS Problem

grinabms — Fri, 14 Mar 2014 15:04:27 GMT

I got my Cloudtrail logs into SplunkAppforAWS with a small change in aws-cloudtrail.py.

Background: Cloudtrail data wasn't feeding into my dashboards, and I saw a steady stream of errors in $SPLUNK_HOME/var/log/splunk/splunkd.log. Same error message:

03-10-2014 04:53:56.015 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'Message'

The solution was to edit $SPLUNK_HOME/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py. I commented out one line, and replaced it with another. Now this appears about 200 lines down in my file:

    #message = json.loads(envelope["Message"])
    message = envelope

Make this change, and in a few minutes, the errors in the splunkd.log disappear, and data begins to populate the dashboards.

Hope this helps.
-Pete

Re: Splunk For AWS Problem

atanasoffa — Wed, 30 Apr 2014 22:41:20 GMT

Did you change anything else in the script? I tried your suggestion and it produced same type of error for MessageId..

Re: Splunk For AWS Problem

grinabms — Mon, 28 Sep 2020 16:31:59 GMT

That's the only change that I made. Can you post your error message?

One suggestion is to add a debugging line to see exactly what is in the "envelope"... here is how it should look:
logging.info("envelope: %s",json.dumps(envelope))
#message = json.loads(envelope["Message"])
message = envelope

When you save the edit, then your splunkd.log file should contain log entries like this:

03-25-2014 23:25:54.726 +0000 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" envelope: {"s3ObjectKey": ["AWSLogs/123412341234/CloudTrail/us-east-1/2014/03/24/123412341234_CloudTrail_us-east-1_20140324T1645Z_pUiRsGvGTkwgBOoL.json.gz"], "s3Bucket": "my-log-bucket"}

Re: Splunk For AWS Problem

atanasoffa — Mon, 05 May 2014 18:23:33 GMT

Thanks for your reply. Here are some of my errors after I applied your suggestion:

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" File "/apps/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py", line 219, in run

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" logging.debug("reading message with id %s at %s",envelope["MessageId"],envelope["Timestamp"])

05-05-2014 18:10:00.495 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkAppforAWS/bin/aws-cloudtrail.py" KeyError: 'MessageId'

I added in a debug line and I do get similar output as you, just in a different order (the "s3bucket" object and value is before the s3ObjectKey) but then I get the errors above...