topic Re: To filter data from cloudwatch logs to splunk in Getting Data In

To filter data from cloudwatch logs to splunk

niddhi — Thu, 16 May 2019 19:05:41 GMT

Hi,

I am getting cloudwatch logs data into Splunk. Right now, i am getting all the log data but i want only specific data(for eg, only the json stream being populated in logs once in a while).

How can i filter the data before Splunk ingest all of it from Cloudwatch Logs.

Thanks,
Niddhi

Re: To filter data from cloudwatch logs to splunk

koshyk — Thu, 16 May 2019 20:40:03 GMT

multiple options are there for you
1. If you are collecting using HF, you can do props/transforms to filter messages
2. If you are using standalone indexer, you can do props/transforms at indexer tier to filter it

Plenty of examples are present in this forum. But until unless you provide sample data and sample configuration you have, we don't really know which field to exclude etc.

Re: To filter data from cloudwatch logs to splunk

niddhi — Thu, 16 May 2019 21:21:07 GMT

I have done it by installing and configuring splunk add On for AWS. I already had the log groups defined. So just pointing to the log groups in the cloudwatch log config did the trick to get the data.

There are props.conf and transform.conf specific to this AddOn for AWS(different location on the splunk under apps.

I am very nee to Splunk and have tried configuring props.conf and transform.conf without any luck.
Can you give me an example, as in what to give for source, sourcetype and where to define filter/pattern for cloudwatch logs.

Thanks

Re: To filter data from cloudwatch logs to splunk

niddhi — Thu, 16 May 2019 21:22:03 GMT

this is the link for configuring the cloudwatch logs: https://docs.splunk.com/Documentation/AddOns/released/AWS/CloudWatchLogs

Re: To filter data from cloudwatch logs to splunk

deepashri_123 — Fri, 17 May 2019 11:08:06 GMT

Hey niddhi,

Refer this link:
https://docs.splunk.com/Documentation/Splunk/7.2.6/Forwarding/Routeandfilterdatad
You need to add the filters in props and transforms.

You need to add the parameters in the sourcetype stanza(sourcetype = aws:cloudwatch) in the props and transforms.
Create this stanza in /opt/splunk/etc/apps/aws/local and not in default.
Thou, you will have to verify the path as i am not sure how the app name will reflect.

Let me know if this helps!!