topic Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx? in All Apps and Add-ons

Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

a263534 — Tue, 29 Sep 2020 11:09:31 GMT

I'm configuring the Splunk Add-on for Amazon Web Services and want to forward CloudWatch logs into Splunk. I can do this if I know the exact loggroup name in CloudWatch logs however if the lambda function is created using CloudFormation it creates a dynamic name with an ID in the loggroup. How can I tell Splunk to look for CloudWatch log groupnames using regex?

I'm configuring using this file: aws_cloudwatch_logs_tasks.conf:

[direct data wildcard]
account = splunk-aws-lob-npd
delay = 1800
groups = /aws/lambda/directdata-dev.*
index = default
interval = 60
only_after = 1970-01-01T00:00:00
region = us-east-1
sourcetype = aws:cloudwatchlogs:directdatawildcard
stream_matcher = .*

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

lcasey001 — Tue, 04 Oct 2016 20:31:41 GMT

I don't think you can. I tried almost every combination as well and wasn't able to do it myself. I ended resorting to this aws logs describe-log-groups --output text --query 'logGroups[*].[logGroupName]' |tr '\n' ',' . This however leads to other issues where a large amount of log_groups can cause ThrottlingExceptions

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

a263534 — Thu, 06 Oct 2016 16:41:24 GMT

Thank you - yeah I ended up opening a case w\ splunk and they are aware of this issue and it will be added in a future release. I also created a script that used aws CLI but I'm pulling directly from the list of lambda functions to only get the most current CW log groups which is helping with throttling.

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

rarsan_splunk — Mon, 06 Feb 2017 04:33:50 GMT

To avoid CloudWatch Logs API throttling issues due to polling, you may want to consider the near real-time streaming of CloudWatch Logs into Splunk via Lambda (i.e. CloudWatch Logs --> Lambda --> Splunk) as explained in this blog post:
http://blogs.splunk.com/2017/02/03/how-to-easily-stream-aws-cloudwatch-logs-to-splunk/

To help with automation, these Lambda functions, acting as logs forwarders, could even be created along with your original logs-producing Lambda functions (or other AWS services) within the same CloudFormation template.

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

mwiora — Tue, 07 Feb 2017 16:43:30 GMT

Hi rarsan,

this seems to be your blogpost.
Could you please have a look on the image-issues?

Cheers and thanks in advance,
µatthias

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

mwiora — Tue, 07 Feb 2017 16:52:40 GMT

Hi a263534,

can you share your script? It would be a great help!
Cheers,
µatthias

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

rarsan_splunk — Tue, 07 Feb 2017 16:58:49 GMT

@mwiora - it's possible that the images are not loading on your end due to their size. Here's a direct link to one of the images for example.

Best place to investigate is usually your browser console. Otherwise, try clearing your cache and refresh.

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

mwiora — Tue, 07 Feb 2017 17:20:06 GMT

@rarsan yeah - thanks for the fast reply!
It turned out that blogs.spunk.com has been provided with a SHA1 signed Certificate and you included the pictures by using HTTPS (probably by default).

As of Google Chrome 56.0.2924.87 does not recognize SHA1 signed Certificates as secure, the images are not displayed - yay 😄

https://security.googleblog.com/2016/11/sha-1-certificates-in-chrome.html

Splunk should take action and update their certificates - especially, since this is the main wildcard-certificate 😉

Re: Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

rarsan_splunk — Tue, 07 Feb 2017 23:12:22 GMT

Thanks @mwiora. I've reported this to our web dev team.
You're right, the explicit https was default behavior. I've also updated the images sources to follow page protocol.