Splunk Search

Will someone help me with my REGEX in this Transforms.conf?

Path Finder
8/30/18 9:38:51.000 AM **rec_type=71** dns_query=s3.amazonaws.com dns_record_name=A src_tos=0 ssl_expected_action=Unknown ...
8/30/18 9:14:19.000 AM **rec_type=109** id=185 rec_type_desc="Web Application" name="Google Translate" rec_type_simple=PAYLOAD
8/30/18 9:13:55.000 AM **rec_type=520** id=600 rec_type_desc="Geolocation Data" name=paraguay rec_type_simple=GEOLOCATION
8/30/18 9:13:55.000 AM **rec_type=63** id=1033 rec_type_desc="Server Metadata" name=Youku rec_type_simple=SERVICE

props.conf
[cisco:estreamer:data]
TRANSFORMS-send-data-to-null-queue = setnull_2

transforms.conf
[setnull_2]
REGEX = (rec_type=(?<!71))
DEST_KEY = queu
FORMAT = null Queue

I'd like all rec_type not equal "71" send to nullQueue. But seems my REGEX doesn't work. Can anyone help? Thanks!

0 Karma

SplunkTrust
SplunkTrust

Hi,

props.conf

 [cisco:estreamer:data]
 TRANSFORMS-cisco = throw_away, index_this

transforms.conf

[throw_away]
 REGEX = rec_type=(?!71)
 DEST_KEY = queue
 FORMAT = nullQueue

 [index_this]
 REGEX = rec_type=(?=71)
 DEST_KEY = queue
 FORMAT = indexQueue
0 Karma

Path Finder
[cisco:estreamer:data]
TRANSFORMS-send-data-to-null-queue = setnull_2, setnull_1

[cisco:estreamer:log]
TRANSFORMS-drop-data = setdrop

[cisco:estreamer:status]
TRANSFORMS-drop-data = setdrop

[setnull_1]
REGEX = ^fw_rule_reason\=N\/A$
DEST_KEY = queue
FORMAT = nullQueue

[setnull_2]
REGEX = ^((?!rec_type\=71).)*$
DEST_KEY = queu
FORMAT = null Queue

[setdrop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Seems the issue is not about the REGEX, it's about how to make multi-REGEX to work. Every time when I restart the service to test it, I only can have one correct logs. rec_type not equal "71" works or fw_rule_reason=N/A works.

0 Karma

Motivator

Hi there,

Try this and see if it works. Below combo routes all events to nullQueue except events containing rec_type=71

props.conf:

[cisco:estreamer:data]
TRANSFORMS-cisco = setnull, setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = rec\_type\=71
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

Path Finder

Thanks for replying. But this splunk is just a forwarder, don't have any license in it. Can I still using the "indexQueue"?

0 Karma

Motivator

If you installed the Universal Forwarder, and told it to get the data and forward it to Splunk Indexer, your configuration should go to Indexers (or heavy forwarder if you have one).

The incoming data passes through a couple of phases; Input, Parsing, Indexing and Search. These type of nullQueue routing operations take place during the Parsing phase. Universal Forwarder cannot do parsing, and will just ignore any such settings

The below link is really old but explains basics well.
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

0 Karma

Path Finder

the regex pass the "https://regex101.com/" online test, but still doesn't work. Does anyone know why?

0 Karma

Path Finder
[setnull_2]
REGEX = ^((?!rec_type\=71).)*$
DEST_KEY = queu
FORMAT = null Queue

This website is really helpful "https://regex101.com/"

0 Karma