topic Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS in All Apps and Add-ons

Cisco Umbrella Log Management with App: Splunk Add-on for AWS

kennymann — Thu, 11 Jan 2018 20:26:59 GMT

Has anyone been able to successfully setup Cisco Umbrella(OpenDNS, Cloud) and the AWS Splunk Add-on?

I have setup Cisco Umbrella using the Amazon S3. The bucket is syncing data according to Cisco Umbrella.
https://support.umbrella.com/hc/en-us/articles/115004685266

My problems start with the configuration of Splunk Addon Inputs error:

S3ResponseError: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

kennymann — Fri, 12 Jan 2018 14:58:15 GMT

After opening a ticket with Cisco Umbrella. This is the answer I received.

"Hello Kenny,
Currently it's not possible to retrieve Cisco managed S3 log files with Splunk. This is due to the restrictive permissions on the bucket itself. The Splunk AWS module expects to be able to list all buckets, which it cannot - as the Cisco Managed buckets are restricted to that singular bucket. However, you can access the logs with the Amazon cli tool. And other third party tools like S3 Browser."

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

nickhills — Fri, 12 Jan 2018 17:25:09 GMT

That’s rubbish!
You may need to change the access policy for your Splunk user so that it has list* and get* permissions on the bucket containing the logs, but there is no reason a bucket in your s3 account can not be retrieved by Splunk with the correct policy settings.

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

rajeevlalla — Wed, 30 May 2018 03:40:54 GMT

I think he's referring to the Cisco provided S3 instance i.e. managed/owned by Cisco Umbrella service.

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

BonMot — Tue, 04 Sep 2018 20:30:31 GMT

Cisco doesn't allow listing on the root of the bucket, and the Splunk AWS app doesn't allow you to type in a bucket_id. Luckily you can manually enter the bucket id in local/inputs.conf. Here is an example that sets a sourcetype compatible with TA-cisco-umbrella. This assumes your S3 bucket is

s3://YOUR-CISCO-S3-BUCKET/YOUR-INCREDIBLY-LONG-FOLDER-NAME

Everything in ALL-CAPS should be replaced. Don't forget the trailing slash (/) after the key_name

[aws_s3://ANY-NAME-YOU-WANT]
start_by_shell = false
aws_account = YOUR-ACCOUNT-NAME-CREATE-IN-UI
sourcetype = opendns:s3
initial_scan_datetime = default
max_items = 100000
max_retries = 3
polling_interval = 30
interval = 30
recursion_depth = -1
character_set = auto
is_secure = True
host_name = s3.amazonaws.com
ct_blacklist = ^$
ct_excluded_events_index =
key_name = YOUR-INCREDIBLY-LONG-FOLDER-NAME/
bucket_name = YOUR-CISCO-S3-BUCKET

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hrithiktej — Fri, 12 Oct 2018 20:26:30 GMT

Hi BonMot,

Thanks for your post but i tried exactly this and it is still not working? Anything else you can suggest?

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hrithiktej — Wed, 17 Oct 2018 18:07:48 GMT

Umbrella suport replied to me the same that due to permission issues it is not possible to ingest the logs using this app. So i followed the work around mentioned in this article

https://support.umbrella.com/hc/en-us/articles/360001388406-Configuring-Splunk-with-a-Cisco-managed-S3-Bucket

and now i have it all working

his article covers the basics of getting Splunk up and running so it is able to consume the logs from your Cisco-managed S3 bucket. You will:

1) Set up your Cisco-managed S3 bucket in your dashboard.

2) Create a cron job to retrieve files from the bucket and store them locally on your server.

3) Configure Splunk to read from a local directory.

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

david_rose — Thu, 17 Jan 2019 22:17:25 GMT

I also tried this with no luck. Logs always show 403 Forbidden. settings are verified by working with s3browser.

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hrithiktej — Fri, 18 Jan 2019 09:58:42 GMT

@ David rose

TRY THIS workaround it works
The work around mentioned in this article

https://support.umbrella.com/hc/en-us/articles/360001388406-Configuring-Splunk-with-a-Cisco-managed-S3-Bucket

and now i have it all working

I am downloading logs onto my syslog servers with a cron job to run after every 10mins and from there i have the splunk reading them.

If u need help with cron job let me know

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

david_rose — Fri, 18 Jan 2019 14:34:28 GMT

Thanks. If I can't use the existing aws add on, i might as well create a modular input. Just trying not to invent the wheel if I don't have to.

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hmorales1 — Fri, 15 Mar 2019 14:58:06 GMT

When you are running this on your Splunk instance is it downloading the csv.gz files?

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hrithiktej — Fri, 15 Mar 2019 15:01:24 GMT

Yes below are the examples of files getting downloaded, and i am downloading it to my syslog server and then from there the splunk UF reads and forwards to my splunk indexer

-rw-r--r--. 1 root root 628 Mar 15 14:50 2019-03-15-14-40-2593.csv.gz
-rw-r--r--. 1 root root 683 Mar 15 14:50 2019-03-15-14-40-32fa.csv.gz
-rw-r--r--. 1 root root 844 Mar 15 14:50 2019-03-15-14-40-7f7d.csv.gz
-rw-r--r--. 1 root root 930 Mar 15 14:50 2019-03-15-14-40-b3ab.csv.gz
-rw-r--r--. 1 root root 798 Mar 15 14:50 2019-03-15-14-40-dcd8.csv.gz

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hmorales1 — Tue, 26 Mar 2019 15:47:10 GMT

Awesome, I'm getting the same thing!

Question how are you cleaning these folders up? are you running a cron job?

Re: Cisco Umbrella Log Management with App: Splunk Add-on for AWS

hrithiktej — Tue, 26 Mar 2019 16:11:06 GMT

Nope i manually clean up after every 30 days. You can setup a cron job for 30 days cleanup. Because in umbrella bucket we have set the log retention for 30 days, so if clean up anything it downloads complete 30 days of log folders again, so once anything gets older than 30 days i delete them.