topic Re: Can anyone please help me with Splunk App for AWS? in All Apps and Add-ons

Can anyone please help me with Splunk App for AWS?

ashishebansal — Wed, 23 Jan 2019 09:45:59 GMT

I am facing issue as per which S3 dashboard option in Splunk App for AWS is not loading any data. The Splunk documentation says that S3 data should be on-boarded using "sourcetype=aws:s3:accesslogs", however the query in S3 traffic tab has "aws:cloudtrai"l in it. I tried changing it to "aws:s3:accesslogs" but query is still not working out. I tried almost everything but it seems like a complicated issue as per which I will have to rebuilt the entire dashboard.

So, is it something that is a miss out from Splunk or myself?

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Wed, 23 Jan 2019 14:58:41 GMT

First, make sure that you are sending your AWS data to the main index, if you created your own 'aws' index, then make sure to update the macros so that the summary indexes are properly populated. Here is a troubleshooting guide to help you.

Once you have set that up, then the queries should work and the source type 'aws:cloudtrail' will show the S3 access since the query is using the cloud trail API calls to S3 to show the access of the events.

That specific dashboard is using data models which are populated with the saved searches and macros mentioned in the troubleshooting guide.

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Tue, 29 Sep 2020 22:56:19 GMT

I have tried most of the things to make that S3 dashboard work but it isn't.

Just to explain it little more, I am sending the data to main index with sourcetype="aws:s3:accesslogs" but it is still not working out. I am trying to work on some POC that will need this app as well, if possible then can we connect regarding this on call or email?

Query one the view in this dashboard is using.

(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail"
| lookup all_eventName eventName OUTPUTNEW function
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName"
| spath output=objectName path=requestParameters.key
| spath output=userName path=userIdentity.userName
| eval error=if(errorCode=="success",0, 1)
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0)
| stats count by region, aws_account_id, bucketName, objectName, userName, eventName, userAgent, sourceIPAddress,Unauthorized, error, readOnly,_time
| search (aws_account_id="") (region="")
| stats sum(count) as count sum(error) as errors
| nadefault errors count
| eval _total_name="Data Events", _total_field=count

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Wed, 23 Jan 2019 16:40:07 GMT

There seems to be an issue with the field extraction for the userName. Try setting the 6th line to :
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName

The search should complete with the count and errors tables populating. Here is the complete search, I took out the last 'search' since I'm not running it within the context of the search filters on the dashboard:

(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail" 
| lookup all_eventName eventName OUTPUTNEW function 
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName" 
| spath output=objectName path=requestParameters.key 
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName
| eval error=if(errorCode=="success",0, 1) 
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0) 
| stats count by region, aws_account_id, bucketName, objectName, userName, userAgent, sourceIPAddress,Unauthorized, error, readOnly, _time
| stats sum(count) as count sum(error) as errors 
| nadefault errors count 
| eval _total_name="Data Events", _total_field=count

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Tue, 29 Sep 2020 22:56:31 GMT

Hi Amiracle,

Thank you so much for providing that solution.

I followed your approach and was able to fix most of the issues but "Most Recent Modifications - Latest 10" is still not working out.

Below is the query:

(index="main" OR (index="main" OR index="default") OR index="default") sourcetype="aws:cloudtrail"
| lookup all_eventName eventName OUTPUTNEW function
| search function="S3 Data Event"
| spath output=bucketName path="requestParameters.bucketName"
| spath output=objectName path=requestParameters.key
| spath output=userName path=userIdentity.sessionContext.sessionIssuer.userName
| eval error=if(errorCode=="success",0, 1)
| lookup unauthorized_errorCode errorCode OUTPUT Unauthorized
| eval Unauthorized=if(Unauthorized=="true", 1, 0)
| stats count by region, aws_account_id, bucketName, objectName, userName, eventName, userAgent, sourceIPAddress,Unauthorized, error, readOnly,_time
| search (aws_account_id="") (region="")
| search readOnly=false (eventName="DeleteObject" OR eventName="PutObject" OR eventName="PostObject")
| sort - _time limit=10
| eval _time=strftime(_time, "%m/%d/%Y %H:%M:%S")
| table bucketName, objectName, userName, eventName, sourceIPAddress, userAgent, _time

| rename bucketName as BucketName, objectName as ObjectName, userName as UserName, eventName as EventName, sourceIPAddress as "Source IP Address", userAgent as UserAgent, _time as Time

The other tab "S3 - Traffic Analysis" is not working at all.

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Tue, 29 Sep 2020 22:55:20 GMT

S3 Traffic Analysis is based on the data model: S3_Access_Log (Settings -> Data Models) and see if the data model was built. If not, hit the rebuild button or update it.

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Wed, 23 Jan 2019 18:33:59 GMT

Yes, sure thanks a lot for help. Is there anything that you can suggest me to try before the model rebuilt ?

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Wed, 23 Jan 2019 18:36:46 GMT

No, all the data is contained in the model so changing any of the searches would just lead to missing fields etc. Are you sending the data into Splunk using the SQS Based S3 input or are you using another method?

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Tue, 29 Sep 2020 22:56:33 GMT

I am using the SQS based S3 only with below configurations.

[aws_sqs_based_s3://S3_Logs]
aws_account = RoleName
aws_iam_role = RoleName
interval = 300
s3_file_decoder = S3AccessLogs
sourcetype = aws:s3:accesslogs
sqs_batch_size = 10
sqs_queue_region = us-east-1
sqs_queue_url = queueName

disabled = 0

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Wed, 23 Jan 2019 18:44:38 GMT

Any solution for "Most Recent Modifications - Latest 10" from S3 - Data events that is also not working out.

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Wed, 23 Jan 2019 18:50:16 GMT

Yes, check the saved search "CloudTrail S3 Data Event Search" and see if it has completed with results. (Settings -> Search, reports and Alerts -> Find "CloudTrail S3 Data Event Search" and then click on Results.) If it's disabled, enable the search.

Re: Can anyone please help me with Splunk App for AWS?

amiracle — Wed, 23 Jan 2019 18:50:46 GMT

That is the base search that is being used by this dashboard to help populate the different reports.

Re: Can anyone please help me with Splunk App for AWS?

ashishebansal — Wed, 23 Jan 2019 19:00:34 GMT

Yes, it is enabled and has completed the results.