To filter data from cloudwatch logs to splunk

[niddhi]

Hi,

I am getting cloudwatch logs data into Splunk. Right now, i am getting all the log data but i want only specific data(for eg, only the json stream being populated in logs once in a while).

How can i filter the data before Splunk ingest all of it from Cloudwatch Logs.

Thanks,
Niddhi

[deepashri_123]

Hey niddhi,

Refer this link:
https://docs.splunk.com/Documentation/Splunk/7.2.6/Forwarding/Routeandfilterdatad
You need to add the filters in props and transforms.

You need to add the parameters in the sourcetype stanza(sourcetype = aws:cloudwatch) in the props and transforms.
Create this stanza in /opt/splunk/etc/apps/aws/local and not in default.
Thou, you will have to verify the path as i am not sure how the app name will reflect.

Let me know if this helps!!

[koshyk]

multiple options are there for you
1. If you are collecting using HF, you can do props/transforms to filter messages
2. If you are using standalone indexer, you can do props/transforms at indexer tier to filter it

Plenty of examples are present in this forum. But until unless you provide sample data and sample configuration you have, we don't really know which field to exclude etc.

[niddhi]

I have done it by installing and configuring splunk add On for AWS. I already had the log groups defined. So just pointing to the log groups in the cloudwatch log config did the trick to get the data.

There are props.conf and transform.conf specific to this AddOn for AWS(different location on the splunk under apps.

I am very nee to Splunk and have tried configuring props.conf and transform.conf without any luck.
Can you give me an example, as in what to give for source, sourcetype and where to define filter/pattern for cloudwatch logs.

Thanks

[niddhi]

this is the link for configuring the cloudwatch logs: https://docs.splunk.com/Documentation/AddOns/released/AWS/CloudWatchLogs