Trouble ignoring events

[_smp_]

I'm having some difficulty forcing Splunk to ignore events which start with a '#' character. The file is compressed, but the events appear to be indexing OK. Here are my props and transforms - is there anything obviously wrong here?

[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
NO_BINARY_CHECK = true
invalid_cause = archive
unarchive_cmd = gunzip -c -f -S .processed
TRANSFORMS-comments = setNull
TRUNCATE = 20000
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19

-
[setNull]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

[woodcock]

You should use ^\s*# instead.

[_smp_]

The first character in the line I want to ignore is a #, so that regex would not match.

[woodcock]

You are missing the point; there might be leading whitespace in front of the pound sign.

[_smp_]

That wasn't what you posted initially, which is why I asked the follow up. But I understand your revised regex, and you're right, that's a good idea. Thanks for bringing it up for consideration.

[jkat54]

You might also try using sedcmd in props negating the need for transforms.conf

SEDCMD-removeHashLines = s/^#.*//g

SEDCMD only happens at index time, so you'd have to reindex the data to see the changes.

[maciep · Accepted solution]

At a glance, I think everything looks ok. Do you have the props and transforms on the parsing layer of your environment, typically an indexer?

[adepasquale]

I have the exact opposite issue, my sql logs contain useful information after the # sign but they are omitted (as comments i suppose)

How can i fix this?

[_smp_]

That's a weird one. So Splunk indexes the entire output until # and then nothing else to the end of the line? What type of input - monitor?

[adepasquale]

It's a multiline input from a MYSQL slow log. I had created my own source type for this with the add monitor command.

# sample
# output
text here

would result in just

text here

Anyways, i changed the sourcetype to the predefined mysql_slow source type and it's working now.

[_smp_]

maciep was right - I had the correct stanzas, but in the wrong place. Here is the corrected versions. Thank you very much!!!

Universal Forwarder: props.conf
[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
NO_BINARY_CHECK = true
invalid_cause = archive
unarchive_cmd = gunzip -c -f -S .processed

Indexer: props.conf
[source::/logs/proxy/SG_proxyna_SIEM__1920629171951.log.gz.processed]
TRANSFORMS-comments = setNull
TRUNCATE = 20000
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19

Indexer: transforms.conf
[setNull]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

[woodcock]

Click Accept on this answer.

[_smp_]

This seems odd to me because maciep's answer was right, not mine.

[woodcock]

You can transfer the points that you got for answering to him.

[maciep]

no worries...karma is cool and all, but just glad it's working for you now 🙂

[jkat54]

All fixed now 😉

[_smp_]

I should have mentioned this in my original post - I have the props and transforms on a universal forwarder. I will try moving the conf files to the indexer and post the results.

[maciep]

Just to be clear, I believe the no binary check, invalid clause and unarchive cmd settings will need to remain on your forwarder. Those happen at input time. The rest happens at parse time and should be on your indexers.

Wasn't sure if you were literally going to move both files or just copy them, so wanted to mention that 🙂

[_smp_]

Actually I am a new Splunk admin and I struggle quite a bit understanding which parameters go where. I found that comment to really helpful - thanks. I'm working on the config files now...

[maciep]

in case you haven't come across it yet, this article may help unmuddy the waters a bit.

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

[_smp_]

Again, great reference - thank you.

You were right, I had the stanzas in the wrong place. I don't see a way to convert your comment to an answer, so I wasn't sure what to do. If you know how, and care about the credit, let me know and I'll be happy to do it.

Thanks again for a really helpful answer.