topic Re: MultiLine Event- How to Ignore/Drop Specific Events in Getting Data In

MultiLine Event- How to Ignore/Drop Specific Events

chrisboy68 — Mon, 09 Feb 2015 18:03:28 GMT

Hi,

I have a multi line flat file where I want to ignore/drop specifc events. I'm using the Universial Forwarder, so as I understand it, the indexer needs to drop/ignore the event. Below is my props.conf on the indexer. I want to drop/ignore any event that is not an ERROR. I tried the PREAMBLE_REGEX property setting a NOT regex and one explicitly looking for DEBUG. Neither are working.

Sample event to ignore:
02/09/2015 11:37:54,807 - DEBUG - https://Blah

[Alerts]
BREAK_ONLY_BEFORE_DATE = TRUE
SHOULD_LINEMERGE = TRUE
TIME_FORMAT=%m/%d/%Y %T
TRUNCATE = 0
MAX_DAYS_AGO = 2
PREAMBLE_REGEX =^((?!\d\d.\d\d.\d\d\d\d\s\d\d:\d\d:\d\d\,\d+\s\-\sERROR).)*
PREAMBLE_REGEX=^\d\d.\d\d.\d\d\d\d\s\d\d:\d\d:\d\d\,\d+\s\-\sDEBUG

Any suggestions?

Thank you!

Chris

Re: MultiLine Event- How to Ignore/Drop Specific Events

chrisboy68 — Mon, 09 Feb 2015 20:39:19 GMT

Ok, found examples of using transforms.conf, but its still not working. Below is my transforms.conf. I want to drop DEBUG statements and INFOs and other stuff. Only keep ERRORS

Sample Error:
02/09/2015 16:25:54,220 - ERROR - ECommerceBlah
in transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue


[setparsing]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

In props.conf

[sourcetype::mySourceType]
TRANSFORMS-trash = setnull,setparsing

Any ideas?

Thank you,

Chris

Re: MultiLine Event- How to Ignore/Drop Specific Events

somesoni2 — Mon, 09 Feb 2015 21:59:22 GMT

Give this a try

transforms.conf (on INDEXER)

 [eliminate-debug]
REGEX=(?m)-\s*DEBUG\s*-
DEST_KEY=queue
FORMAT=nullQueue

props.conf (on INDEXER)
[sourcetype::mySourceType]
TRANSFORMS-trash = eliminate-debug

Restart/reload the Indexer after change.

Re: MultiLine Event- How to Ignore/Drop Specific Events

chrisboy68 — Tue, 10 Feb 2015 15:02:37 GMT

Ugh, this is baffling. Nothing is working. I placed splunkd in debug mod and didnt see anything in the errors. Is there a way to debug if the "transforms" are being hit?

Chris

Re: MultiLine Event- How to Ignore/Drop Specific Events

chrisboy68 — Mon, 28 Sep 2020 18:52:59 GMT

Output of ./splunk cmd btool transforms list below. So it looks like the transforms are loading. Just cant figure out why they it is not working..

[eliminate-debug]
CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEST_KEY = queue
FORMAT = nullQueue
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MV_ADD = False
REGEX = (?m)-\s*DEBUG\s*-
SOURCE_KEY = _raw
WRITE_META = False

Re: MultiLine Event- How to Ignore/Drop Specific Events

chrisboy68 — Thu, 12 Feb 2015 15:25:36 GMT

Ah, so [sourcetype::mySourceType] is not correct. I found I just needed [mySourceType] in the props.config, now its working.

Chris