Splunk Add-on for Amazon Web Services: How can I configure Splunk to look for CloudWatch logs group names using RegEx?

[a263534]

I'm configuring the Splunk Add-on for Amazon Web Services and want to forward CloudWatch logs into Splunk. I can do this if I know the exact loggroup name in CloudWatch logs however if the lambda function is created using CloudFormation it creates a dynamic name with an ID in the loggroup. How can I tell Splunk to look for CloudWatch log groupnames using regex?

I'm configuring using this file: aws_cloudwatch_logs_tasks.conf:

[direct data wildcard]
account = splunk-aws-lob-npd
delay = 1800
groups = /aws/lambda/directdata-dev.*
index = default
interval = 60
only_after = 1970-01-01T00:00:00
region = us-east-1
sourcetype = aws:cloudwatchlogs:directdatawildcard
stream_matcher = .*

[rarsan_splunk]

To avoid CloudWatch Logs API throttling issues due to polling, you may want to consider the near real-time streaming of CloudWatch Logs into Splunk via Lambda (i.e. CloudWatch Logs --> Lambda --> Splunk) as explained in this blog post:
http://blogs.splunk.com/2017/02/03/how-to-easily-stream-aws-cloudwatch-logs-to-splunk/

To help with automation, these Lambda functions, acting as logs forwarders, could even be created along with your original logs-producing Lambda functions (or other AWS services) within the same CloudFormation template.

[mwiora]

Hi rarsan,

this seems to be your blogpost.
Could you please have a look on the image-issues?

Cheers and thanks in advance,
µatthias

[rarsan_splunk]

@mwiora - it's possible that the images are not loading on your end due to their size. Here's a direct link to one of the images for example.

Best place to investigate is usually your browser console. Otherwise, try clearing your cache and refresh.

[mwiora]

@rarsan yeah - thanks for the fast reply!
It turned out that blogs.spunk.com has been provided with a SHA1 signed Certificate and you included the pictures by using HTTPS (probably by default).

As of Google Chrome 56.0.2924.87 does not recognize SHA1 signed Certificates as secure, the images are not displayed - yay 😄

https://security.googleblog.com/2016/11/sha-1-certificates-in-chrome.html

Splunk should take action and update their certificates - especially, since this is the main wildcard-certificate 😉

[rarsan_splunk]

Thanks @mwiora. I've reported this to our web dev team.
You're right, the explicit https was default behavior. I've also updated the images sources to follow page protocol.

[lcasey001]

I don't think you can. I tried almost every combination as well and wasn't able to do it myself. I ended resorting to this aws logs describe-log-groups --output text --query 'logGroups[*].[logGroupName]' |tr '\n' ',' . This however leads to other issues where a large amount of log_groups can cause ThrottlingExceptions

[a263534]

Thank you - yeah I ended up opening a case w\ splunk and they are aware of this issue and it will be added in a future release. I also created a script that used aws CLI but I'm pulling directly from the list of lambda functions to only get the most current CW log groups which is helping with throttling.

[mwiora]

Hi a263534,

can you share your script? It would be a great help!
Cheers,
µatthias