Getting Data In

How to move unwanted logs into nullQueue

uagraw01
Contributor

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060

I want to remove ####< from my events, so i used props.conf along with transforms.conf with this below setting. But still ####< is not removed from the events.

My props.conf

[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

My transforms.conf
[eliminate-hash-item]
DELIMS = ####<
DEST_KEY=queue
FORMAT=nullQueue

Please help me to solve this issue.

Labels (2)
Tags (1)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01,
there are two problems in your transforms.conf:
the first parameter is wrong, you have to use REGEX = ####<, not DELIMS, as you can see at https://docs.splunk.com/Documentation/SplunkCloud/8.0.2004/Forwarding/Routeandfilterdatad#Filter_eve...

Then # and < are special chars for regexes, so you have to escape them, try:

REGEX = \#\#\#\#\<

Ciao.
Giuseppe

0 Karma

uagraw01
Contributor

@gcusello As i tried the setting as you suggest but still ####< is not remoed from the logs.

props.conf
[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

I am unable to paste screenshot.

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01,
about the regex, you can test your regex using the regex command on your logs:

your-search
| regex "\#\#\#\#\<"

and see if Splunk correctly find the logs to remove, so you can adjust your regex inside Splunk and find the correct one (sometimes there are differences between Splunk and regex101)

Then I see a difference in you conf files:
the name in TRANSFORMS-remove-hash in props.conf, must be the same in transforms.conf stanza:
props.conf:

TRANSFORMS-remove-hash = eliminate-hash-item

transforms.conf:

[eliminate-hash-item]
REGEX = \#\#\#\#\<
DEST_KEY=queue
FORMAT=nullQueue

Ciao.
Giuseppe

0 Karma

uagraw01
Contributor

@gcusello Thanks but i tried everything logs ###< is not removed from the events, by correcting everything from my side .

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01,
did you tested your regex in Splunk using the regex command?

your_search
| regex "\#\#\#\#\<"

what's the result?

ciao.
Giuseppe

0 Karma

uagraw01
Contributor

@gcusello
When i perform a search index=main | regex "####<"
The result is same 12/05/202014:00:09.000 ####

It is capturing full events which have timestamps with #.

I want only this May 12, 2020 2:00:09 PM CD

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01,
let me understand:
using the regex command do you find the events to filter or not?
if yes the regex is correct if not, you have to modify the regex.

Could you share two or three events to discard and two or three events to take? so I can help you with the regex.

Ciao.
giuseppe

0 Karma

uagraw01
Contributor

@gcusello Please see my full log path:

[ ####

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

in which i want [May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

and ignore [####<

<

<

<]

in these call we can use nullQueue or indexQueue in transforms.conf but nothing works.

props.conf
[hast_sourcetype]
CHARSET = UTF-8
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item,include-date-item
description = hash_sourcetype

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

[include-date-item]
REGEX = [A-Za-z]{3}\s[0-9]{2},\s\d+\s\d:\d+:\d+\s\w{2}.*
DEST_KEY=queue
FORMAT=indexQueue

Please provide your inputs.

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01 ,

I thought that you solved!

Anyway, I see that you have logs like this:

[ ####
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

you want to discard the fist row and take the other three, is it correct?

If this is your need, try this props.conf:

[hastsourcetype]
CHARSET = UTF-8
LINEBREAKER = ([\r\n]+)
MAXTIMESTAMPLOOKAHEAD = 29
SHOULDLINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item
description = hashsourcetype

and this transforms.conf:

[eliminate-hash-item]
REGEX=\[\s+\#\#\#\#
DEST_KEY=queue
FORMAT=nullQueue

Ciao.

Giuseppe

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

0 Karma

gcusello
Esteemed Legend

Hi @uagraw01,

Ok good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

uagraw01
Contributor

@gcusello But not getting the resolution.

 

 

0 Karma

uagraw01
Contributor

Thanks for answering .Please let me know if i use REGEX = (.*#<) would it work? Because it is correctly matched on regex101 engine. But when i used this on transforms.conf it is not remove any ####< char from events while indexing. Please suggest

0 Karma

uagraw01
Contributor

In place of DELIMS = ####< i used REGEX= ####< also but event showing the same.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...