Why is nullQueue : log discard not working?

[craymore]

Hi,

I recently experimented with Splunk transformations in order to discard some log entries ( and that worked well on my lab setup )

I am now trying to implement such solution on our main Splunk Enterprise server in pre-preproduction ( UAT ), but, for some reason, Splunk seems to ignore completly the transformation statements

NB : in order to troubleshoot this behaviour, I already tried lots of things, including, among other things

• running splunkd in debug mode
• setting global regex(es) which would include everything
• checking the configurations using btool
• restarting the splunk service ( on the indexer ), countless times

... sadly, without any results so far 😐

Setup : summary

• 1 x main server / indexer ( Splunk Enterprise 6.4.1 w/ licence )
• 117 x universal forwarders ( same version as the indexer )
• logs which are sent to the indexer : JSON applicative logs

*Setup : forwarder(s) *

/opt/splunkforwarder/etc/system/local/inputs.conf 

[...]
[monitor:///var/log/vplatform/latest/app.log.json]
index = vplatform_uat_logs
crcSalt = <SOURCE>
sourcetype = VPLATFORM_JSON
[...]

*Setup : indexer *

/opt/splunkforwarder/etc/system/local/indexes.conf

[vplatform_uat_logs]

homePath   = $SPLUNK_DB/vplatform_UAT_logs/db
coldPath   = $SPLUNK_DB/vplatform_UAT_logs/colddb
thawedPath = $SPLUNK_DB/vplatform_UAT_logs/thaweddb

/opt/splunk/etc/apps/vplatform_UAT/local/props.conf

[VPLATFORM_JSON]
KV_MODE = none
INDEXED_EXTRACTIONS = JSON
TRUNCATE = 0
TRANSFORMS-null = drop_info_jsons

/opt/splunk/etc/apps/vplatform_UAT/local/transforms.conf

[drop_info_jsons]
REGEX = .*
DEST_KEY = queue
FORMAT = nullQueue

Behaviour

According to the setup above, all JSON logs sent from the forwarders, with the sourcetype VPLATFORM_JSON, should be discarded

But unfortunately, this isn't the case.

If I issue the a query in the Splunk search app, it still gives results

Query

index="vplatform_uat_logs" sourcetype="VPLATFORM_JSON"

Result sample

 { [-]
   @level: INFO
   @level_value: 20000
   @service_name: vplatform-datamanager
   hostname: t1vbbservice.*****
   processId: 25568
   version: 2.5.0-9243
} 

host = t1vbbservice.******
source = /var/log/vplatform/latest/app.log.json
sourcetype = VPLATFORM_JSON

Solution ?

If someone could try to give me a clue about the reason Splunk is apparently not taking the transformation statements present in props & transforms, I would really appreciate it ( I have already spent two days on this issue and still don't understand why it is behaving like this )

Thanks beforehand

[p_gurav]

Hi,

try using setnull in props.conf :
TRANSFORMS-null = setnull

And in transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[craymore]

Hello !

This is exactly what I did ( except for the * in the regex which isn't necessary ), as indicated in the main post, but it isn't working

[gcusello]

Hi craymore,
at first two quick suggestions:

• indexed extractions are the only situation where you need to deploy props.conf and transforms.conf also in Universal Forwarder, and not only on Indexers;

• regex in transforms.conf doesn't need asterisk

  [drop_info_jsons]
  REGEX = .
  DEST_KEY = queue
  FORMAT = nullQueue

There's a thing that I don't understand in your question: reading your props.conf and transform.conf, it seems that you want to discard all the events with sourcetype VPLATFORM_JSON, in this case why don't you change your inputs.conf?

Anyway, to solve the main problem (discarding not running), I think that you followed Splunk documentation at http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad .

To discard only specific events, I usually use two stanzas, something like this:
in props.conf

[source::/var/log/messages]
TRANSFORMS-set= setparsing, setnull

in transforms.conf

[setnull]
REGEX = my_regex
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

[craymore]

Hi Giuseppe,

Thanks for your relply.

Regarding your answer specifically :

Indexed extractions

"indexed extractions are the only situation where you need to deploy props.conf and transforms.conf also in Universal Forwarder, and not only on Indexers"

I was really surprised to read this, as nearly all the posts I have read so far mention that these files are not even processed by the universal forwarder.

If this is wrong, then it would explain why my transform is not working

Discard

Regarding the discard rule specifically, I understand you were surprised to see that I want to discard all the vents, but this is just to emphasize the fact that the discard isn't working as intended ( the goal is indeed to discard only a few entries in the end, but first I need to get it working properly )

[gcusello]

Hi craymore,
a clarification:
discard filters are always enabled on Indexers,
instead sourcetype specifications are defined on Universal Forwarders when you have indexed extractions.
This means that the discard problem isn't related to the position of discard filters.
I think that the first thing is to test the filtering regex.
To do this you can use Splunk itself (rex command).
When you'll sure of your regex, choose is you have to discard some events and take all the otheres ot to take some events and discard the others.
Thisis importanto in the props.conf and transforms.conf definition:
if you have to discard some events and take all the others, you can use my above configurations;
if instead your have to take some events and discard all the others use the following configurations:
props.conf

[source::/var/log/messages]
TRANSFORMS-set= setnull, setparsing

transforms.conf.

 [setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 [setparsing]
 REGEX = my_regex
 DEST_KEY = queue
 FORMAT = indexQueue

In few words:
in props.conf, the order of stanzas is:

• before the all events stanza,
• then the filtering stanza. If you want, share the regex for filtering and an example to test it.

Bye.
Giuseppe