All Apps and Add-ons

Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

a263534
New Member

I'm configuring the Splunk Add-on for Amazon Web Services and want to forward CloudWatch logs into Splunk. I can do this if I know the exact loggroup name in CloudWatch logs however if the lambda function is created using CloudFormation it creates a dynamic name with an ID in the loggroup. How can I tell Splunk to look for CloudWatch log groupnames using regex?

I'm configuring using this file: aws_cloudwatch_logs_tasks.conf:

[direct data wildcard]
account = splunk-aws-lob-npd
delay = 1800
groups = /aws/lambda/directdata-dev.*
index = default
interval = 60
only_after = 1970-01-01T00:00:00
region = us-east-1
sourcetype = aws:cloudwatchlogs:directdatawildcard
stream_matcher = .*
0 Karma

rarsan_splunk
Splunk Employee
Splunk Employee

To avoid CloudWatch Logs API throttling issues due to polling, you may want to consider the near real-time streaming of CloudWatch Logs into Splunk via Lambda (i.e. CloudWatch Logs --> Lambda --> Splunk) as explained in this blog post:
http://blogs.splunk.com/2017/02/03/how-to-easily-stream-aws-cloudwatch-logs-to-splunk/

To help with automation, these Lambda functions, acting as logs forwarders, could even be created along with your original logs-producing Lambda functions (or other AWS services) within the same CloudFormation template.

0 Karma

mwiora
Explorer

Hi rarsan,

this seems to be your blogpost.
Could you please have a look on the image-issues?

Cheers and thanks in advance,
µatthias

alt text

rarsan_splunk
Splunk Employee
Splunk Employee

@mwiora - it's possible that the images are not loading on your end due to their size. Here's a direct link to one of the images for example.

Best place to investigate is usually your browser console. Otherwise, try clearing your cache and refresh.

0 Karma

mwiora
Explorer

@rarsan yeah - thanks for the fast reply!
It turned out that blogs.spunk.com has been provided with a SHA1 signed Certificate and you included the pictures by using HTTPS (probably by default).

As of Google Chrome 56.0.2924.87 does not recognize SHA1 signed Certificates as secure, the images are not displayed - yay 😄

https://security.googleblog.com/2016/11/sha-1-certificates-in-chrome.html

Splunk should take action and update their certificates - especially, since this is the main wildcard-certificate 😉

0 Karma

rarsan_splunk
Splunk Employee
Splunk Employee

Thanks @mwiora. I've reported this to our web dev team.
You're right, the explicit https was default behavior. I've also updated the images sources to follow page protocol.

lcasey001
Explorer

I don't think you can. I tried almost every combination as well and wasn't able to do it myself. I ended resorting to this aws logs describe-log-groups --output text --query 'logGroups[*].[logGroupName]' |tr '\n' ',' . This however leads to other issues where a large amount of log_groups can cause ThrottlingExceptions

0 Karma

a263534
New Member

Thank you - yeah I ended up opening a case w\ splunk and they are aware of this issue and it will be added in a future release. I also created a script that used aws CLI but I'm pulling directly from the list of lambda functions to only get the most current CW log groups which is helping with throttling.

0 Karma

mwiora
Explorer

Hi a263534,

can you share your script? It would be a great help!
Cheers,
µatthias

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...