topic Unable to nullqueue unnecessary lines without a date in Splunk Search

Unable to nullqueue unnecessary lines without a date

babcolee — Thu, 20 Jun 2019 20:51:10 GMT

I have two lines of events that are unnecessary because there is no date and would like to null queue these out. I have tried several REGEX statements ( I have tried ^\s+(\<\/ns2:Subject.*\>), <\/ns2:.*\>, and to name a few) to delete these events. The Field Extraction GUI and REGEX online tester show a match. However, these events still show up in the search.
The data which needs to be deleted are as follows:
(Please note there are spaces showing before the events and they are on separate lines)

 </ns2:Subject>
       </ns2:SubjectConfirmation>

transforms.conf

[TrashEmptySubject]
REGEX = ^\s+(\<\/ns2:Subject.*\>)
DEST_KEY = queue

props.conf

[smtrace]
TRANSFORMS-null = TrashEmptySubject
FORMAT = nullQueue

Re: Unable to nullqueue unnecessary lines without a date

pruthvikrishnap — Thu, 20 Jun 2019 21:52:55 GMT

add FORMAT = nullQueue to transforms.conf, instead of props.conf

Re: Unable to nullqueue unnecessary lines without a date

woodcock — Thu, 20 Jun 2019 23:11:44 GMT

Try this:

props.conf

[smtrace]
TRANSFORMS-null = TrashEmptySubject

transforms.conf

[TrashEmptySubject]
REGEX = [\r\n\s]+<\/ns2:Subject[^>]+>
DEST_KEY = queue
FORMAT = nullQueue

Re: Unable to nullqueue unnecessary lines without a date

babcolee — Fri, 21 Jun 2019 13:17:44 GMT

Thank you for your responses. Unfortunately, the events are still showing when I do a search after applying this REGEX

Re: Unable to nullqueue unnecessary lines without a date

woodcock — Sat, 22 Jun 2019 19:59:49 GMT

If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value NOT the new value, then you must deploy this to the first full instance(s) of Splunk that handles the events (usually either the HF-tier, if you use this, or your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Re: Unable to nullqueue unnecessary lines without a date

babcolee — Mon, 24 Jun 2019 16:24:13 GMT

Thank you for the help. We are not doing any sourcetype overide. We are getting these events directly from a forwarder, passing through one of two HF's (not being indexed), ultimately landing on a Splunk instance which is both search head / indexer. I have edited both the HF's and search head / indexer with these stanzas and we are still seeing these type of events.

Re: Unable to nullqueue unnecessary lines without a date

woodcock — Mon, 24 Jun 2019 17:00:25 GMT

My answer definitely should work. Did you notice that you had a line in props.conf that should be in transforms.conf?

Re: Unable to nullqueue unnecessary lines without a date

babcolee — Mon, 24 Jun 2019 17:45:51 GMT

Here are my configurations:

props.conf
[smtrace]
TRANSFORMS-null = TrashEmptySubject

transforms.conf
[TrashEmptySubject]
REGEX = [\r\n\s]+<\/ns2:Subject[^>]+>
DEST_KEY = queue
FORMAT = nullQueue