topic Re: Splunk blacklist regex using inputs.conf of universal forwarder? in Getting Data In

Splunk blacklist regex using inputs.conf of universal forwarder?

bapun18 — Tue, 28 Mar 2023 01:05:48 GMT

Needs to blacklist certain syslogs messages from the forwarder level. We have raw syslogs as below:

2023-03-27T00:00:00+00:00 10.10.33.15 Mar 27 2023 00:00:00.028 UTC : %UC_Test-4-DeviceTransientConnection: %[ConnectingPort=2000][DeviceName=AN004A1328478011][IPAddress=10.152.157.107][DeviceType=30027][Reason=3][Protocol=SCCP][IPAddrAttributes=2][UNKNOWN_PARAMNAME:LastSignalReceived=StationRegister][UNKNOWN_PARAMNAME:StationState=wait_register][AppID=Siso CallManager][ClusterID=c6801ccm][NodeID=c6801011ccm007]: A device attempted to register but did not complete registration 0.0.3.1 0 0

2023-03-27T00:00:00+00:00 10.10.33.15 Mar 27 2023 00:00:00.144 UTC : %UC_Test-4-DeviceTransientConnection: %[ConnectingPort=2000][DeviceName=ANF000673BC20003][IPAddress=10.70.56.248][DeviceType=30027][Reason=3][Protocol=SCCP][IPAddrAttributes=2][UNKNOWN_PARAMNAME:LastSignalReceived=StationRegister][UNKNOWN_PARAMNAME:StationState=wait_register][AppID=Siso CallManager][ClusterID=c6801ccm][NodeID=c6801011ccm007]: A device attempted to register but did not complete registration 0.0.3.1 0 0

2023-03-27T00:00:00+00:00 10.10.33.15 Mar 27 2023 00:00:00.147 UTC : %UC_Test-4-DeviceTransientConnection: %[ConnectingPort=2000][DeviceName=AN00A13274B800D][IPAddress=10.108.2.248][DeviceType=30027][Reason=3][Protocol=SCCP][IPAddrAttributes=2][UNKNOWN_PARAMNAME:LastSignalReceived=StationRegister][UNKNOWN_PARAMNAME:StationState=wait_register][AppID=Siso CallManager][ClusterID=c6801ccm][NodeID=c6801011ccm007]: A device attempted to register but did not complete registration 0.0.3.1 0

I need to filter the data before pushing it to the Splunk indexer, with respect to UC_Test-4-DeviceTransientConnection and Reason=3

which means I don't want to push the data which have UC_Test-4-DeviceTransientConnection and Reason=3.

I have tried blacklisting it in inputs.conf
blacklist = ^.*UC_Test-4-DeviceTransientConnection.*\[Reason=3\].*$

above isn't working then I have tried with props.conf and transforms.conf like below

[testsys]
TRUNCATE = 0
TRANSFORMS-NULL = setnull

[setnull]
REGEX = ^.*UC_Test-4-DeviceTransientConnection.*\[Reason=3\].*$
DEST_KEY = queue
FORMAT = nullQueue

But unfortunately, it's still not filtering.

Re: Splunk blacklist regex using inputs.conf of universal forwrder

yeahnah — Mon, 27 Mar 2023 21:07:03 GMT

Hi @bapun18

Using the props and transforms method is the correct method. This configuration is not meant to not work at source using a Splunk universal forwarder and needs to live on the Splunk parsing layer (heavy forwarders/indexers, depending on your env).

If it must be filtered at source, then you may need to look at a separate job/script to pre-filter the log and have Splunk UF forward the new output file.

Hope this helps

Re: Splunk blacklist regex using inputs.conf of universal forwarder?

gcusello — Tue, 28 Mar 2023 06:23:02 GMT

Hi @bapun18,

blacklisting events usually runs only on windows logs.

the only way is described at https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

You must put the props.conf and transforms.conf on the first Heavy Forwarder (if present) between the Universal Forwarder and the Indexers, if not present, on Indexers.

you could try with following:

pros.conf:

[your_sourcetype] TRANSFORMS-null= setnull

transforms.conf:

[setnull] REGEX = UC_Test-4-DeviceTransientConnection and Reason\=3 DEST_KEY = queue FORMAT = nullQueue

Ciao.

Giuseppe

Re: Splunk blacklist regex using inputs.conf of universal forwarder?

yeahnah — Wed, 29 Mar 2023 00:44:17 GMT

I just discovered there is a UF setting (props.conf > force_local_processing) - documented since v7 - that makes the UF parse the data, i.e. make props.conf and transforms.conf work at source.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Possibly, the UF will not be as configurable to the same level as a heavy forwarder would.

See: https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774

This may also cook the data at source, so the HF will not do any further processing.

Understandably, there will be a cost to local UF CPU and mem usage, depending on what it's configured to do and amount of data being processed, so should be used with caution, i.e. test it.

https://community.splunk.com/t5/Getting-Data-In/What-are-the-capabilities-of-the-quot-force-local-processing/m-p/336932