topic Re: Help filtering data to nullQueue in Getting Data In

Help filtering data to nullQueue

johnward4 — Thu, 05 Dec 2019 21:15:59 GMT

I'm trying to filter out unwanted data but it's not working using my current stanzas in props & transforms. However, I was able to filter using the regex and reset the sourcetype so that should rule out an issue with the regex I'm attempting to use..

sample_log for applicationone :

2019-12-03 00:59:57,812  stdout INFO [ajp-/0.0.0.0:8009-16]: Hibernate: select sample.SAMPLE_ID as SAMPLE_ID1_5_, SAMPLE0_.sample_DESCRIPTION as sample_DESCRIPTI2_5_ from sample_SAMPLE functional0_

props.conf

[applicationone:log]
TRANSFORMS-sendtonull = removeDBqueries

transforms.conf

[removeDBqueries]
REGEX = select\s+.*)
DEST_KEY = queue
FORMAT = nullQueue

Re: Help filtering data to nullQueue

harsmarvania57 — Thu, 05 Dec 2019 21:31:43 GMT

Can you please remove bracket from REGEX and check ? Like REGEX = select\s+.*

Re: Help filtering data to nullQueue

johnward4 — Thu, 05 Dec 2019 21:46:05 GMT

@harsmarvania57 I tried that and it still isn't working. Could it be a problem with the sourcetype I using, does it need to be applied to _raw log data?

Re: Help filtering data to nullQueue

harsmarvania57 — Thu, 05 Dec 2019 21:59:09 GMT

Sourcetype should work, and that REGEX will apply to _raw. Have you restarted splunk after changing config? Additionally only new data will go to nullQueue based on REGEX match, old data will stay.

Re: Help filtering data to nullQueue

harsmarvania57 — Thu, 05 Dec 2019 22:07:40 GMT

Can you please confirm on which instance you have applied above configuration ? It must be on Indexer or Heavy Forwarder, whichever comes first from Universal Forwarder.

Re: Help filtering data to nullQueue

johnward4 — Thu, 05 Dec 2019 22:34:33 GMT

I'm trying this on a single test instance. After I make a change to my configs, I delete the data from the index and restart the instance. I then upload the data again to apply my updated configs against it.

Re: Help filtering data to nullQueue

gcusello — Fri, 06 Dec 2019 07:52:25 GMT

Hi @johnward4,
two questions:

where are you executing this filter? you can do it only on Indexers or (when present) on Heavy Forwarders;
what's "applicationone:log" that you use in the stanza's title in props.conf? usually it's used sourcetype (better) or host or source.

Ciao.
Giuseppe

Re: Help filtering data to nullQueue

johnward4 — Fri, 06 Dec 2019 15:58:33 GMT

Right now, I'm building the add-on in my single instance test environment.

"applicationone:log" is the name I picked for the data sourcetype.

Re: Help filtering data to nullQueue

woodcock — Fri, 06 Dec 2019 21:20:57 GMT

Fix this:

 REGEX = select\s+.*\)

Re: Help filtering data to nullQueue

johnward4 — Sun, 08 Dec 2019 21:29:47 GMT

there was an issue with my REGEX. This did the trick:

REGEX = (SELECT|Select|select)\s+
DEST_KEY = queue
FORMAT = nullQueue