I have started the conditional logging on Splunk but still i'm getting the logs?

[ayushchoudhary]

I have configured transforms.conf and props.conf on below path

/opt/splunk/etc/apps/search/local

transforms.conf

[setnull]
REGEX = INFO
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[source::/opt/assays/log/assays.stdout.log]
TRANSFORMS-null= setnull

But still i'm getting logs from source = /opt/assays/log/assays.stdout.log.
Please Help.

[ayushchoudhary]

Everything working fine now, thanks all.
All the logs having INFO are now stopped to be indexed.

Just out of curiosity can any one help on what parameters we can apply conditional logging i.e. we can apply using hostname and source but we apply conditional logging using index ??

[koshyk]

in your transforms , you are missing
REGEX=.

[gcusello]

Hi ayushchoudhary,
probably I'm doing a stupid question: do you inserted your .conf files in your indexers or in your forwarders? they must be on Indexers.

In addition I usually not use nullQueue alone and I never use source or host in stanzas but always sourcetype because sometimes fails, try in this way:

• on props.conf:

  [your_sourcetype]
  TRANSFORMS-filter_sourcetype1=set_sourcetype1,set_nullqueue

• on transforms.conf:

  [set_nullqueue]
  REGEX=INFO
  DEST_KEY=queue
  FORMAT=nullQueue
  [set_sourcetype1]
  REGEX=.
  DEST_KEY = queue
  FORMAT = indexQueue
  Note that it's important sets order in props.conf TRANSFORMS command, it isn't important in transforms.conf stanzas.

This run if you want to take all but not INFO; if you need to take only something and discard other, you have to invert order in props.conf sets and change regexes.

Bye.
Giuseppe

[bshuler_splunk]

I put this app together as an example. Please only use it in a non-production environment.

SA-null_queue.spl

I noticed when building it, that getting the props.conf:source:: stanza was tricky, as it did not accept $SPLUNK_HOME

I looked at the entries that match source:: for examples:

./splunk cmd btool props list source:: | grep \\[
[source::...((.(bak|old))|,v|~|#)]
[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
[source::....(?<!tar.)gz(.\d+)?]
[source::....(cache|class|cxx|dylib|jar|lo|xslt|md5|rpm|deb|iso|vim)]
[source::....(css|htm|html|sgml|shtml|template)]
[source::....(jar)(.\d+)?]
[source::....(tar.gz|tgz)(.\d+)?]
[source::....(tbz|tbz2)(.\d+)?]
[source::....Z(.\d+)?]
[source::....bz2?(.\d+)?]
[source::....crash.log(.\d+)?]
[source::....csv]
[source::....tar(.\d+)?]
[source::.../(apache|httpd).../error*]
[source::.../(readme|README)...]
[source::.../(u_|)ex(tend|\d{4,8})*?.log]
 ... 

I hope this example helps. Good Luck!

[bshuler_splunk · Accepted solution]

You seem to be following good examples. The docs you want to follow for this are here: http://docs.splunk.com/Documentation/Splunk/6.5.0/Forwarding/Routeandfilterdatad#Discard_specific_ev...

You didn't mention where you put these files. They need to go on the splunk system that does the parsing. This is most likely the indexers, but could be a heavy weight forwarder.

Use btool to validate your configurations: https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

If you have the files on the correct splunk system, and they validate with btool, the only other variable is, does your source in props match, and does your REGEX match?

Hope this helps. Good luck.

[ayushchoudhary]

I have added these files in the indexer as the forwarder is a Universal forwarder. Yes i do have INFO strings in the events coming from the source.
After applying the conditional logging i can see the INFO events.

[Masa]

Configuration seems okay. Probably missing open [ is just this Slunk answer issue.
Have you restarted the indexer(s)? If you deployed this to indexers using Cluster Master, you might need to wait for a while due to how Splunk reload works for regex-replacement(transforms.conf).
Or, you restart CPs and see if you still have the issue.

As @bshulter said, btool to check configuration is important. Also checking configuration file permissions or configuration name would be worth.