Splunk Search

Why alert triggers false positive, but testing from a new Search produces correct results?

RNB
Path Finder

I am having an issue where I have created a search string that returns the correct results, but when used as an alert, it returns different results. I am attempting to identify all Cisco messages that have a severity level of -1-.

( -1- NOT (search OR "CALLMANAGER-6"))

This is the expected result from Search and from the Alert.

Jan 19 14:19:49 192.203.1.20 Wireless: *apfReceiveTask: Jan 19 19:19:46.819: #APF-1-ADD_TO_BLACKLIST_FAILED: apf_ms.c:5636 Unable to create exclusion-list entry for mobile a4:ee:57:22:e9:a3 

However, the Alert also triggers on the following types of events where -1- appears anywhere in the log entry (near end of line 3), despite excluding CALLMANAGER-6 which is the typical false positive.

Jan 19 14:06:27 192.203.1.21 456037: Jan 19 19:06:27.664 UTC : %CCM_CALLMANAGER-CALLMANAGER-6-DeviceRegistered: Device registered. Device name.:SEP44ADD9BC819C Device IP address [Optional].:10.42.31.22 Protocol.:SCCP Device type. [Optional]:495 Performance monitor object type:2 Device description [Optional].:OR-Res2-101 Load ID. [Optional]:SCCP69xx.9-4-1-3SR1 Associated directory numbers. [Optional].:3672 Device MAC address [Optional].:44ADD9BC819C IPAddressAttributes [Optional].:3 ActiveLoadId [Optional].:SCCP69xx.9-3-3-2 App ID:Cisco CallManager Cluster ID:StandAloneCluster Node ID:BAUCCMS01

I also tried a regex with no success, where the regex matches any string starting with % or #, includes -1-, ends with a colon.

* | regex_raw="[%,#]*.-1-*.:"

Any ideas how to get the Alert working the way I need it to?

Thank you
Randy

Tags (2)
0 Karma

MuS
Legend

Hi RNB,

I cannot solve this problem, but i can help you to find a different way to achieve this. If I get it correct you want to match the -1- in #APF-1-ADD_TO_BLACKLIST_FAILED? If this is correct, you can use field extraction like you tried and simply search for the values you're interested in. Try something like this:

your base search to get the events here | rex "\:\s[\#\%](?<myField>.*?)\:" | search myField="*-1-*" 

This will create a new field called myField (name can be changed to what ever) containing APF-1-ADD_TO_BLACKLIST_FAILED and CCM_CALLMANAGER-CALLMANAGER-6-DeviceRegistered based on the provided examples. The last | search will only return all myField values which contains a -1- .

After you got all working, setup automatic field extraction http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/Createandmaintainsearch-timefieldextract... and you no longer need to use rex.

hope this helps to get you started ...

cheers, MuS

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

New This Month - Observability Updates Give Extended Visibility and Improve User ...

This month is a collection of special news! From Magic Quadrant updates to AppDynamics integrations to ...

Intro to Splunk Synthetic Monitoring

In our last post, we mentioned that the 3 key pieces of observability – metrics, logs, and traces – provide ...