topic Re: Trouble ignoring events in Getting Data In

Trouble ignoring events

_smp_ — Tue, 05 Jul 2016 20:53:51 GMT

I'm having some difficulty forcing Splunk to ignore events which start with a '#' character. The file is compressed, but the events appear to be indexing OK. Here are my props and transforms - is there anything obviously wrong here?

[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
NO_BINARY_CHECK = true
invalid_cause = archive
unarchive_cmd = gunzip -c -f -S .processed
TRANSFORMS-comments = setNull
TRUNCATE = 20000
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19

-
[setNull]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

Re: Trouble ignoring events

maciep — Wed, 06 Jul 2016 01:40:26 GMT

At a glance, I think everything looks ok. Do you have the props and transforms on the parsing layer of your environment, typically an indexer?

Re: Trouble ignoring events

MuS — Wed, 06 Jul 2016 04:00:59 GMT

or a Heavy weight forwarder - don't forget to restart the Splunk instance after you added the config files.

Re: Trouble ignoring events

jkat54 — Wed, 06 Jul 2016 12:06:17 GMT

You might also try using sedcmd in props negating the need for transforms.conf

SEDCMD-removeHashLines = s/^#.*//g

SEDCMD only happens at index time, so you'd have to reindex the data to see the changes.

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 12:18:47 GMT

I should have mentioned this in my original post - I have the props and transforms on a universal forwarder. I will try moving the conf files to the indexer and post the results.

Re: Trouble ignoring events

maciep — Wed, 06 Jul 2016 12:22:52 GMT

Just to be clear, I believe the no binary check, invalid clause and unarchive cmd settings will need to remain on your forwarder. Those happen at input time. The rest happens at parse time and should be on your indexers.

Wasn't sure if you were literally going to move both files or just copy them, so wanted to mention that 🙂

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 12:26:51 GMT

Actually I am a new Splunk admin and I struggle quite a bit understanding which parameters go where. I found that comment to really helpful - thanks. I'm working on the config files now...

Re: Trouble ignoring events

maciep — Wed, 06 Jul 2016 12:42:37 GMT

in case you haven't come across it yet, this article may help unmuddy the waters a bit.

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 12:43:20 GMT

maciep was right - I had the correct stanzas, but in the wrong place. Here is the corrected versions. Thank you very much!!!

Universal Forwarder: props.conf
[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
NO_BINARY_CHECK = true
invalid_cause = archive
unarchive_cmd = gunzip -c -f -S .processed

Indexer: props.conf
[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
TRANSFORMS-comments = setNull
TRUNCATE = 20000
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19

Indexer: transforms.conf
[setNull]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 12:47:56 GMT

Again, great reference - thank you.

You were right, I had the stanzas in the wrong place. I don't see a way to convert your comment to an answer, so I wasn't sure what to do. If you know how, and care about the credit, let me know and I'll be happy to do it.

Thanks again for a really helpful answer.

Re: Trouble ignoring events

woodcock — Wed, 06 Jul 2016 13:45:27 GMT

You should use ^\s*# instead.

Re: Trouble ignoring events

woodcock — Wed, 06 Jul 2016 13:46:10 GMT

Click Accept on this answer.

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 14:18:28 GMT

The first character in the line I want to ignore is a #, so that regex would not match.

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 14:19:10 GMT

This seems odd to me because maciep's answer was right, not mine.

Re: Trouble ignoring events

woodcock — Wed, 06 Jul 2016 15:43:20 GMT

You are missing the point; there might be leading whitespace in front of the pound sign.

Re: Trouble ignoring events

maciep — Wed, 06 Jul 2016 16:16:26 GMT

no worries...karma is cool and all, but just glad it's working for you now 🙂

Re: Trouble ignoring events

_smp_ — Wed, 06 Jul 2016 16:42:42 GMT

That wasn't what you posted initially, which is why I asked the follow up. But I understand your revised regex, and you're right, that's a good idea. Thanks for bringing it up for consideration.

Re: Trouble ignoring events

woodcock — Wed, 06 Jul 2016 19:17:06 GMT

You can transfer the points that you got for answering to him.

Re: Trouble ignoring events

jkat54 — Thu, 07 Jul 2016 13:48:07 GMT

All fixed now 😉

Re: Trouble ignoring events

adepasquale — Fri, 30 Aug 2019 19:23:30 GMT

I have the exact opposite issue, my sql logs contain useful information after the # sign but they are omitted (as comments i suppose)

How can i fix this?