topic Re: Filtering syslog HaProxy 200 status Logs in Getting Data In

How to filter syslog HaProxy 200 status Logs?

jkostovich — Mon, 25 Jul 2022 17:33:26 GMT

Hello

I currently need to filter out some logs from our HaProxy Server. Have a UniversalForwarder installed on the haproxy box and it is currently logging everything locally through syslog to /var/log/haproxy.log

It is logging all 200 logs which generate thousands of logs an hour which we dont need. I have tried creating a props and transform to do this as follows. I am possibly not placing them in the right location?

props.conf 

[sourcetype::/var/log/haproxy.log]
TRANSFORM-null = setnull

transform.conf

[setnull]
REGEX =  200 #Looking for the string 200 in the log
DEST_KEY = queue
FORMAT = nullQueue

I thought putting this on the indexers would then filter out all logs with "200" in them, but it did nothing.

Next I tried editing the haproxy syslog config itself.

When I added the following code it completely killed all log flow, not just 200.

if ( \
      $programname contains 'haproxy' and \
      not ($msg contains ' 200 ' ) \
   )
then -/var/log/haproxy.log

These are the two solutions, editing the rsyslog config and creating transforms and props edits, that are most advertised in my searches as solutions but I cannot seem to get them to work.

Any help would be greatly appreciated!

Re: Filtering syslog HaProxy 200 status Logs

hgrow — Thu, 07 Sep 2017 11:03:21 GMT

"I am possibly not placing them in the right location?"

From http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Routeandfilterdatad

"You can configure routing only on a heavy forwarder. "

If you placed the filtering-configuration on your Universal Forwarder that is the wrong place since the UF does not parse the data. In other words it 's not looking into your data and applying your transformation.

You have to place the configuration on the first splunk-system thats acutaly parsing, which is most likely your indexer.

Sincerely

Re: Filtering syslog HaProxy 200 status Logs

jkat54 — Thu, 07 Sep 2017 11:47:26 GMT

The props and transforms can go on the indexer or a heavy forwarder because both support sending to nullQueue.

See this: http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

However, the major issue i see is that it appears you have an incorrect props.conf, try these settings instead:

props.conf

 [source::/var/log/haproxy.log]
 TRANSFORMS-null = setnull

Note there are two changes... sourcetype becomes source, and TRANSFORM becomes TRANSFORMS.

transforms.conf

 [setnull]
 REGEX =  200 
 DEST_KEY = queue
 FORMAT = nullQueue

No difference from what you had.

Re: Filtering syslog HaProxy 200 status Logs

jkat54 — Thu, 07 Sep 2017 11:57:24 GMT

Note according to this: https://wiki.splunk.com/Community:HowIndexingWorks

UFs support nullQueue as well, but most of the other documentation says Queue routing can only be done on a HF or IDX.... your mileage may vary

Re: Filtering syslog HaProxy 200 status Logs

jkostovich — Thu, 07 Sep 2017 13:41:50 GMT

Regarding placing of the props and transform. Do they go into the haproxy add-on app folder or into the indexers etc/system/local folder?

Re: Filtering syslog HaProxy 200 status Logs

alemarzu — Thu, 07 Sep 2017 13:48:42 GMT

Beware of the spelling, configuration file name is transforms.conf not transform. You could place it inside the add-on created and deploy/install into the Indexer.

Re: Filtering syslog HaProxy 200 status Logs

jkostovich — Thu, 07 Sep 2017 14:24:11 GMT

I attempted the props.conf and tranforms.conf fix on my indexers to no avail. All logs are still coming through, unless my REGEX of 200 inst actually finding the ' 200 ' string that is in the haproxy log.

My rsyslog haproxy config looks like this.

$ModLoad imudp

#Opens Port 514 to listen for haproxy messages
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
$template Haproxy, "msg%n"

if $programname startswith 'haproxy' then /var/log/haproxy.log

#Defines the http log will be saved in haproxy.log
#Logs Everything
local0.=info -/var/log/haproxy.log;HaProxy
#Keeps logs in local host
local0.* ~

I have attempted to filter by $msg contains ' 200 ' in the RainerScript but it seems to filter out everything when I do so.

Re: Filtering syslog HaProxy 200 status Logs

jkostovich — Thu, 07 Sep 2017 15:43:12 GMT

We have figured out the solution to this. The props and transforms was not working for us so we went to the syslog config and try to filter at the source.

$ModLoad imudp

#Opens Port 514 to listen for haproxy messages
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
#$template Haproxy, "msg%n"

:msg, regex, " 200 " ~
local0.=info -/var/log/haproxy.log

The regex was the answer to this for us. It is searching for a 200 status code and filtering them out then allowing whats left through to the log file that splunk is monitoring.

Re: Filtering syslog HaProxy 200 status Logs

woodcock — Sun, 10 Sep 2017 05:02:58 GMT

The problem was your inlined comment here: REGEX = 200 #Looking for the string 200 in the log
You must never do that; it was being used as part of your REGEX.

Re: Filtering syslog HaProxy 200 status Logs

ethan1el — Mon, 25 Jul 2022 17:08:18 GMT

Here is how we removed all 200 entries in the haproxy.log. Simply added this line to /etc/rsyslog.conf:

if ($programname == "haproxy" and not ($msg contains " 200 ")) then /var/log/haproxy.log