topic Re: Transforming error logs in Splunk Cloud Platform

Transforming error logs

vickyingle — Fri, 15 Nov 2024 15:17:07 GMT

I'm trying to transform a error log

Below is a sample log (nginx_error)

2024/11/15 13:10:11 [error] 4080#4080: *260309 connect() failed (111: Connection refused) while connecting to upstream, client: 210.54.88.72, server: mpos.mintpayments.com, request: "GET /payment-mint/cnpPayments/v1/publicKeys?callback=jQuery360014295356911736334_1731369073329&X-Signature=plkb810sFSSSIbASLb818BMXxgtUM76QNvhI%252FBA%253D&X-Timestamp=1731368881376&X-ApiKey=CSSSAPXXXXXXPxmO7kjMi&X-CompanyToken=d1111e8lV1mpvljiCD2zRgEEU121p&_=1731369073330 HTTP/1.1", upstream: "https://10.20.3.59:28076//cnpPayments/v1/publicKeys?callback=jQuery360014295356911736334_1731369073329&X-Signature=plkb810sFY3jmET4IbASLb818BMXxgtUM76QNvhI%252FBA%253D&X-Timestamp=1731368881376&X-ApiKey=CNPAPIIk7elIMDTunrIGMuXPxmO7kjMi&X-CompanyToken=dX6E3yDe8lV1mpvljiCD2zRgEEU121p&_=173123073330", host: "test.mintpayments.com", referrer: "https://vicky9.mintpayments.com/testing??asd

We are trying to
1) GET query parameters must not be logged
2) Referrer must not contain the query string

I have updated my config as below

[04:59 PM] [root@dev-web01 splunkforwarder]# cat ./etc/system/local/props.conf [source::///var/log/devops/nginx_error.log] TRANSFORMS-sanitize_referer = remove_get_query_params, remove_referer_query [04:59 PM] [root@dev-web01 splunkforwarder]# cat ./etc/system/local/transforms.conf [remove_get_query_params] REGEX = (GET|POST|HEAD) ([^? ]+)\?.* FORMAT = $1 $2 DEST_KEY = _raw REPEAT_MATCH = true [remove_referer_query] REGEX = referrer: "(.*?)\?.*" FORMAT = referrer: "$1" DEST_KEY = _raw REPEAT_MATCH = true

Verified that the regex is correct and when I run below to list the changes, its present
/opt/splunkforwarder/bin/splunk btool transforms list --debug
/opt/splunkforwarder/bin/splunk btool props list --debug

Still I can see no transformation in the logs, what could be the issue here ?
We are using custom splunkforwarder in our env.

Re: Transforming error logs

isoutamo — Fri, 15 Nov 2024 15:13:37 GMT

Is this custom forwarder a Heavy Forwarder instead of Universal Forwarder?
You can use transforms.conf only in HF.

Your sample didn't contain end " which you are expecting on REGEX.

Should those regex are like https://regex101.com/r/iDjLlJ/1 and https://regex101.com/r/kuIxoI/1 as you are basically replacing _raw on both case with your matching groups?

(.*)(GET|POST|HEAD) ([^? ]+)\?([^\"]+)(\".*) => $1$2 $3$5

(.*referrer: ")([^\?]+\?)\?([^"]+)(") => $1$2$4

Re: Transforming error logs

richgalloway — Fri, 15 Nov 2024 16:10:06 GMT

FWIW, REPEAT_MATCH is ignored when DEST_KEY=_raw. I believe DEST_KEY is not needed here since FORMAT says where the capture groups go.

Re: Transforming error logs

vickyingle — Mon, 18 Nov 2024 00:34:16 GMT

I'm using a universal forwarder, hence the transforms are not working, appreciate your response.

Re: Transforming error logs

vickyingle — Mon, 18 Nov 2024 00:39:11 GMT

Is there any way I can transform these logs once I receive them in Splunk (cloud) ?
These are nginx error logs which contains sensitive data, and in nginx we can not sanitize the error_logs.
Any suggestions will be highly appreciated.

Re: Transforming error logs

isoutamo — Mon, 18 Nov 2024 16:52:15 GMT

The easiest way is setup HF on your own site to do it.