How to move unwanted logs into nullQueue

[uagraw01]

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060

I want to remove ####< from my events, so i used props.conf along with transforms.conf with this below setting. But still ####< is not removed from the events.

My props.conf

[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

My transforms.conf
[eliminate-hash-item]
DELIMS = ####<
DEST_KEY=queue
FORMAT=nullQueue

Please help me to solve this issue.

[gcusello]

Hi @uagraw01,
there are two problems in your transforms.conf:
the first parameter is wrong, you have to use REGEX = ####<, not DELIMS, as you can see at https://docs.splunk.com/Documentation/SplunkCloud/8.0.2004/Forwarding/Routeandfilterdatad#Filter_eve...

Then # and < are special chars for regexes, so you have to escape them, try:

REGEX = \#\#\#\#\<

Ciao.
Giuseppe

[uagraw01]

@gcusello As i tried the setting as you suggest but still ####< is not remoed from the logs.

props.conf
[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

I am unable to paste screenshot.

[gcusello]

Hi @uagraw01,
about the regex, you can test your regex using the regex command on your logs:

your-search
| regex "\#\#\#\#\<"

and see if Splunk correctly find the logs to remove, so you can adjust your regex inside Splunk and find the correct one (sometimes there are differences between Splunk and regex101)

Then I see a difference in you conf files:
the name in TRANSFORMS-remove-hash in props.conf, must be the same in transforms.conf stanza:
props.conf:

TRANSFORMS-remove-hash = eliminate-hash-item

transforms.conf:

[eliminate-hash-item]
REGEX = \#\#\#\#\<
DEST_KEY=queue
FORMAT=nullQueue

Ciao.
Giuseppe

[uagraw01]

@gcusello Thanks but i tried everything logs ###< is not removed from the events, by correcting everything from my side .

[gcusello]

Hi @uagraw01,
did you tested your regex in Splunk using the regex command?

your_search
| regex "\#\#\#\#\<"

what's the result?

ciao.
Giuseppe

[uagraw01]

@gcusello
When i perform a search index=main | regex "####<"
The result is same 12/05/202014:00:09.000 ####

It is capturing full events which have timestamps with #.

I want only this May 12, 2020 2:00:09 PM CD

[gcusello]

Hi @uagraw01,
let me understand:
using the regex command do you find the events to filter or not?
if yes the regex is correct if not, you have to modify the regex.

Could you share two or three events to discard and two or three events to take? so I can help you with the regex.

Ciao.
giuseppe

[uagraw01]

@gcusello Please see my full log path:

[ ####

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

in which i want [May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

and ignore [####<

<

<

<]

in these call we can use nullQueue or indexQueue in transforms.conf but nothing works.

props.conf
[hast_sourcetype]
CHARSET = UTF-8
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item,include-date-item
description = hash_sourcetype

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

[include-date-item]
REGEX = [A-Za-z]{3}\s[0-9]{2},\s\d+\s\d:\d+:\d+\s\w{2}.*
DEST_KEY=queue
FORMAT=indexQueue

Please provide your inputs.

[gcusello]

Hi @uagraw01 ,

I thought that you solved!

Anyway, I see that you have logs like this:

[ ####
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

you want to discard the fist row and take the other three, is it correct?

If this is your need, try this props.conf:

[hastsourcetype]
CHARSET = UTF-8
LINEBREAKER = ([\r\n]+)
MAXTIMESTAMPLOOKAHEAD = 29
SHOULDLINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item
description = hashsourcetype

and this transforms.conf:

[eliminate-hash-item]
REGEX=\[\s+\#\#\#\#
DEST_KEY=queue
FORMAT=nullQueue

Ciao.

Giuseppe

[gcusello · Accepted solution]

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

[gcusello]

Hi @uagraw01,

Ok good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

[uagraw01]

@gcusello But not getting the resolution.

[uagraw01]

Thanks for answering .Please let me know if i use REGEX = (.*#<) would it work? Because it is correctly matched on regex101 engine. But when i used this on transforms.conf it is not remove any ####< char from events while indexing. Please suggest

[uagraw01]

In place of DELIMS = ####< i used REGEX= ####< also but event showing the same.