topic Re: I want to move my unwanted logs into nullQueue.But no luck in Getting Data In

How to move unwanted logs into nullQueue

uagraw01 — Wed, 30 Sep 2020 05:37:15 GMT

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060

I want to remove ####< from my events, so i used props.conf along with transforms.conf with this below setting. But still ####< is not removed from the events.

My props.conf

[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

My transforms.conf
[eliminate-hash-item]
DELIMS = ####<
DEST_KEY=queue
FORMAT=nullQueue

Please help me to solve this issue.

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Wed, 03 Jun 2020 15:56:40 GMT

In place of DELIMS = ####< i used REGEX= ####< also but event showing the same.

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Wed, 03 Jun 2020 16:01:54 GMT

Hi @uagraw01,
there are two problems in your transforms.conf:
the first parameter is wrong, you have to use REGEX = ####<, not DELIMS, as you can see at https://docs.splunk.com/Documentation/SplunkCloud/8.0.2004/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

Then # and < are special chars for regexes, so you have to escape them, try:

REGEX = \#\#\#\#\<

Ciao.
Giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Wed, 03 Jun 2020 17:56:27 GMT

Thanks for answering .Please let me know if i use REGEX = (.*#<) would it work? Because it is correctly matched on regex101 engine. But when i used this on transforms.conf it is not remove any ####< char from events while indexing. Please suggest

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Wed, 30 Sep 2020 05:36:58 GMT

@gcusello As i tried the setting as you suggest but still ####< is not remoed from the logs.

props.conf
[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

I am unable to paste screenshot.

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Thu, 04 Jun 2020 06:34:40 GMT

Hi @uagraw01,
about the regex, you can test your regex using the regex command on your logs:

your-search
| regex "\#\#\#\#\<"

and see if Splunk correctly find the logs to remove, so you can adjust your regex inside Splunk and find the correct one (sometimes there are differences between Splunk and regex101)

Then I see a difference in you conf files:
the name in TRANSFORMS-remove-hash in props.conf, must be the same in transforms.conf stanza:
props.conf:

TRANSFORMS-remove-hash = eliminate-hash-item

transforms.conf:

[eliminate-hash-item]
REGEX = \#\#\#\#\<
DEST_KEY=queue
FORMAT=nullQueue

Ciao.
Giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Thu, 04 Jun 2020 12:35:12 GMT

@gcusello Thanks but i tried everything logs ###< is not removed from the events, by correcting everything from my side .

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Thu, 04 Jun 2020 12:50:04 GMT

Hi @uagraw01,
did you tested your regex in Splunk using the regex command?

your_search
| regex "\#\#\#\#\<"

what's the result?

ciao.
Giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Thu, 04 Jun 2020 13:06:50 GMT

@gcusello
When i perform a search index=main | regex "####<"
The result is same 12/05/202014:00:09.000 ####

It is capturing full events which have timestamps with #.

I want only this May 12, 2020 2:00:09 PM CD

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Thu, 04 Jun 2020 14:10:01 GMT

Hi @uagraw01,
let me understand:
using the regex command do you find the events to filter or not?
if yes the regex is correct if not, you have to modify the regex.

Could you share two or three events to discard and two or three events to take? so I can help you with the regex.

Ciao.
giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Wed, 30 Sep 2020 05:37:05 GMT

@gcusello Please see my full log path:

[ ####

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

in which i want [May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

and ignore [####<

<

<]

in these call we can use nullQueue or indexQueue in transforms.conf but nothing works.

props.conf
[hast_sourcetype]
CHARSET = UTF-8
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item,include-date-item
description = hash_sourcetype

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

[include-date-item]
REGEX = [A-Za-z]{3}\s[0-9]{2},\s\d+\s\d:\d+:\d+\s\w{2}.*
DEST_KEY=queue
FORMAT=indexQueue

Please provide your inputs.

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Tue, 16 Jun 2020 15:58:55 GMT

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

uagraw01 — Tue, 16 Jun 2020 19:04:26 GMT

@gcusello But not getting the resolution.

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Wed, 17 Jun 2020 10:54:40 GMT

Hi @uagraw01 ,

I thought that you solved!

Anyway, I see that you have logs like this:

[ #### 2020-05-12 14:34:52,060 2020-05-12 14:34:52,060 2020-05-12 14:34:52,060]

you want to discard the fist row and take the other three, is it correct?

If this is your need, try this props.conf:

[hastsourcetype] CHARSET = UTF-8 LINEBREAKER = ([\r\n]+) MAXTIMESTAMPLOOKAHEAD = 29 SHOULDLINEMERGE = false TRANSFORMS-remove-hash = eliminate-hash-item description = hashsourcetype

and this transforms.conf:

[eliminate-hash-item] REGEX=\[\s+\#\#\#\# DEST_KEY=queue FORMAT=nullQueue

Ciao.

Giuseppe

Re: I want to move my unwanted logs into nullQueue.But no luck

gcusello — Wed, 30 Sep 2020 06:40:23 GMT

Hi @uagraw01,

Ok good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉