Help filtering data to nullQueue

[johnward4]

I'm trying to filter out unwanted data but it's not working using my current stanzas in props & transforms. However, I was able to filter using the regex and reset the sourcetype so that should rule out an issue with the regex I'm attempting to use..

sample_log for applicationone :

2019-12-03 00:59:57,812  stdout INFO [ajp-/0.0.0.0:8009-16]: Hibernate: select sample.SAMPLE_ID as SAMPLE_ID1_5_, SAMPLE0_.sample_DESCRIPTION as sample_DESCRIPTI2_5_ from sample_SAMPLE functional0_ 

props.conf

[applicationone:log]
TRANSFORMS-sendtonull = removeDBqueries

transforms.conf

[removeDBqueries]
REGEX = select\s+.*)
DEST_KEY = queue
FORMAT = nullQueue

[harsmarvania57]

Can you please remove bracket from REGEX and check ? Like REGEX = select\s+.*

[johnward4]

@harsmarvania57 I tried that and it still isn't working. Could it be a problem with the sourcetype I using, does it need to be applied to _raw log data?

[harsmarvania57]

Sourcetype should work, and that REGEX will apply to _raw. Have you restarted splunk after changing config? Additionally only new data will go to nullQueue based on REGEX match, old data will stay.

[harsmarvania57]

Can you please confirm on which instance you have applied above configuration ? It must be on Indexer or Heavy Forwarder, whichever comes first from Universal Forwarder.

[johnward4]

I'm trying this on a single test instance. After I make a change to my configs, I delete the data from the index and restart the instance. I then upload the data again to apply my updated configs against it.

[gcusello]

Hi @johnward4,
two questions:

• where are you executing this filter? you can do it only on Indexers or (when present) on Heavy Forwarders;
• what's "applicationone:log" that you use in the stanza's title in props.conf? usually it's used sourcetype (better) or host or source.

Ciao.
Giuseppe

[johnward4]

Right now, I'm building the add-on in my single instance test environment.

"applicationone:log" is the name I picked for the data sourcetype.

[woodcock · Accepted solution]

Fix this:

 REGEX = select\s+.*\)

[johnward4]

there was an issue with my REGEX. This did the trick:

REGEX = (SELECT|Select|select)\s+
DEST_KEY = queue
FORMAT = nullQueue