I'm trying to transform a error log
Below is a sample log (nginx_error)
2024/11/15 13:10:11 [error] 4080#4080: *260309 connect() failed (111: Connection refused) while connecting to upstream, client: 210.54.88.72, server: mpos.mintpayments.com, request: "GET /payment-mint/cnpPayments/v1/publicKeys?callback=jQuery360014295356911736334_1731369073329&X-Signature=plkb810sFSSSIbASLb818BMXxgtUM76QNvhI%252FBA%253D&X-Timestamp=1731368881376&X-ApiKey=CSSSAPXXXXXXPxmO7kjMi&X-CompanyToken=d1111e8lV1mpvljiCD2zRgEEU121p&_=1731369073330 HTTP/1.1", upstream: "https://10.20.3.59:28076//cnpPayments/v1/publicKeys?callback=jQuery360014295356911736334_1731369073329&X-Signature=plkb810sFY3jmET4IbASLb818BMXxgtUM76QNvhI%252FBA%253D&X-Timestamp=1731368881376&X-ApiKey=CNPAPIIk7elIMDTunrIGMuXPxmO7kjMi&X-CompanyToken=dX6E3yDe8lV1mpvljiCD2zRgEEU121p&_=173123073330", host: "test.mintpayments.com", referrer: "https://vicky9.mintpayments.com/testing??asd
We are trying to 1) GET query parameters must not be logged 2) Referrer must not contain the query string
I have updated my config as below
[04:59 PM] [root@dev-web01 splunkforwarder]# cat ./etc/system/local/props.conf
[source::///var/log/devops/nginx_error.log]
TRANSFORMS-sanitize_referer = remove_get_query_params, remove_referer_query
[04:59 PM] [root@dev-web01 splunkforwarder]# cat ./etc/system/local/transforms.conf
[remove_get_query_params]
REGEX = (GET|POST|HEAD) ([^? ]+)\?.*
FORMAT = $1 $2
DEST_KEY = _raw
REPEAT_MATCH = true
[remove_referer_query]
REGEX = referrer: "(.*?)\?.*"
FORMAT = referrer: "$1"
DEST_KEY = _raw
REPEAT_MATCH = true
Verified that the regex is correct and when I run below to list the changes, its present /opt/splunkforwarder/bin/splunk btool transforms list --debug /opt/splunkforwarder/bin/splunk btool props list --debug
Still I can see no transformation in the logs, what could be the issue here ? We are using custom splunkforwarder in our env.
... View more