topic Re: I have started the conditional logging on Splunk but still i'm getting the logs? in Getting Data In

I have started the conditional logging on Splunk but still i'm getting the logs?

ayushchoudhary — Wed, 02 Nov 2016 07:42:51 GMT

I have configured transforms.conf and props.conf on below path

/opt/splunk/etc/apps/search/local

transforms.conf

[setnull]
REGEX = INFO
DEST_KEY = queue
FORMAT = nullQueue

props.conf

[source::/opt/assays/log/assays.stdout.log]
TRANSFORMS-null= setnull

But still i'm getting logs from source = /opt/assays/log/assays.stdout.log.
Please Help.

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

bshuler_splunk — Wed, 02 Nov 2016 08:13:05 GMT

You seem to be following good examples. The docs you want to follow for this are here: http://docs.splunk.com/Documentation/Splunk/6.5.0/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

You didn't mention where you put these files. They need to go on the splunk system that does the parsing. This is most likely the indexers, but could be a heavy weight forwarder.

Use btool to validate your configurations: https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurations

If you have the files on the correct splunk system, and they validate with btool, the only other variable is, does your source in props match, and does your REGEX match?

Hope this helps. Good luck.

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

ayushchoudhary — Wed, 02 Nov 2016 10:49:49 GMT

I have added these files in the indexer as the forwarder is a Universal forwarder. Yes i do have INFO strings in the events coming from the source.
After applying the conditional logging i can see the INFO events.

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

bshuler_splunk — Wed, 02 Nov 2016 14:03:44 GMT

I put this app together as an example. Please only use it in a non-production environment.

SA-null_queue.spl

I noticed when building it, that getting the props.conf:source:: stanza was tricky, as it did not accept $SPLUNK_HOME

I looked at the entries that match source:: for examples:

./splunk cmd btool props list source:: | grep \\[
[source::...((.(bak|old))|,v|~|#)]
[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
[source::....(?<!tar.)gz(.\d+)?]
[source::....(cache|class|cxx|dylib|jar|lo|xslt|md5|rpm|deb|iso|vim)]
[source::....(css|htm|html|sgml|shtml|template)]
[source::....(jar)(.\d+)?]
[source::....(tar.gz|tgz)(.\d+)?]
[source::....(tbz|tbz2)(.\d+)?]
[source::....Z(.\d+)?]
[source::....bz2?(.\d+)?]
[source::....crash.log(.\d+)?]
[source::....csv]
[source::....tar(.\d+)?]
[source::.../(apache|httpd).../error*]
[source::.../(readme|README)...]
[source::.../(u_|)ex(tend|\d{4,8})*?.log]
 ...

I hope this example helps. Good Luck!

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

Masa — Wed, 02 Nov 2016 22:38:52 GMT

Configuration seems okay. Probably missing open [ is just this Slunk answer issue.
Have you restarted the indexer(s)? If you deployed this to indexers using Cluster Master, you might need to wait for a while due to how Splunk reload works for regex-replacement(transforms.conf).
Or, you restart CPs and see if you still have the issue.

As @bshulter said, btool to check configuration is important. Also checking configuration file permissions or configuration name would be worth.

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

gcusello — Wed, 30 Sep 2020 02:54:00 GMT

Hi ayushchoudhary,
probably I'm doing a stupid question: do you inserted your .conf files in your indexers or in your forwarders? they must be on Indexers.

In addition I usually not use nullQueue alone and I never use source or host in stanzas but always sourcetype because sometimes fails, try in this way:

on props.conf:

[your_sourcetype]
TRANSFORMS-filter_sourcetype1=set_sourcetype1,set_nullqueue
on transforms.conf:

[set_nullqueue]
REGEX=INFO
DEST_KEY=queue
FORMAT=nullQueue
[set_sourcetype1]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue
Note that it's important sets order in props.conf TRANSFORMS command, it isn't important in transforms.conf stanzas.

This run if you want to take all but not INFO; if you need to take only something and discard other, you have to invert order in props.conf sets and change regexes.

Bye.
Giuseppe

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

koshyk — Sat, 05 Nov 2016 08:59:30 GMT

in your transforms , you are missing
REGEX=.

Re: I have started the conditional logging on Splunk but still i'm getting the logs?

ayushchoudhary — Sat, 05 Nov 2016 10:37:29 GMT

Everything working fine now, thanks all.
All the logs having INFO are now stopped to be indexed.

Just out of curiosity can any one help on what parameters we can apply conditional logging i.e. we can apply using hostname and source but we apply conditional logging using index ??