Getting Data In

How to move unwanted logs into nullQueue

uagraw01
Builder

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060

I want to remove ####< from my events, so i used props.conf along with transforms.conf with this below setting. But still ####< is not removed from the events.

My props.conf

[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

My transforms.conf
[eliminate-hash-item]
DELIMS = ####<
DEST_KEY=queue
FORMAT=nullQueue

Please help me to solve this issue.

Labels (2)
Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01,
there are two problems in your transforms.conf:
the first parameter is wrong, you have to use REGEX = ####<, not DELIMS, as you can see at https://docs.splunk.com/Documentation/SplunkCloud/8.0.2004/Forwarding/Routeandfilterdatad#Filter_eve...

Then # and < are special chars for regexes, so you have to escape them, try:

REGEX = \#\#\#\#\<

Ciao.
Giuseppe

0 Karma

uagraw01
Builder

@gcusello As i tried the setting as you suggest but still ####< is not remoed from the logs.

props.conf
[hast_sourcetype]
BREAK_ONLY_BEFORE_DATE =
CHARSET = UTF-8
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = include-date-item
category = Custom
description = hash_sourcetype
pulldown_type = true

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

I am unable to paste screenshot.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01,
about the regex, you can test your regex using the regex command on your logs:

your-search
| regex "\#\#\#\#\<"

and see if Splunk correctly find the logs to remove, so you can adjust your regex inside Splunk and find the correct one (sometimes there are differences between Splunk and regex101)

Then I see a difference in you conf files:
the name in TRANSFORMS-remove-hash in props.conf, must be the same in transforms.conf stanza:
props.conf:

TRANSFORMS-remove-hash = eliminate-hash-item

transforms.conf:

[eliminate-hash-item]
REGEX = \#\#\#\#\<
DEST_KEY=queue
FORMAT=nullQueue

Ciao.
Giuseppe

0 Karma

uagraw01
Builder

@gcusello Thanks but i tried everything logs ###< is not removed from the events, by correcting everything from my side .

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01,
did you tested your regex in Splunk using the regex command?

your_search
| regex "\#\#\#\#\<"

what's the result?

ciao.
Giuseppe

0 Karma

uagraw01
Builder

@gcusello
When i perform a search index=main | regex "####<"
The result is same 12/05/202014:00:09.000 ####

It is capturing full events which have timestamps with #.

I want only this May 12, 2020 2:00:09 PM CD

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01,
let me understand:
using the regex command do you find the events to filter or not?
if yes the regex is correct if not, you have to modify the regex.

Could you share two or three events to discard and two or three events to take? so I can help you with the regex.

Ciao.
giuseppe

0 Karma

uagraw01
Builder

@gcusello Please see my full log path:

[ ####

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

in which i want [May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

May 12, 2020 2:00:09 PM CDT>

2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

and ignore [####<

<

<

<]

in these call we can use nullQueue or indexQueue in transforms.conf but nothing works.

props.conf
[hast_sourcetype]
CHARSET = UTF-8
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
SHOULD_LINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item,include-date-item
description = hash_sourcetype

transforms.conf
[eliminate-hash-item]
REGEX = ####<
DEST_KEY=queue
FORMAT=nullQueue

[include-date-item]
REGEX = [A-Za-z]{3}\s[0-9]{2},\s\d+\s\d:\d+:\d+\s\w{2}.*
DEST_KEY=queue
FORMAT=indexQueue

Please provide your inputs.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01 ,

I thought that you solved!

Anyway, I see that you have logs like this:

[ ####
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060
2020-05-12 14:34:52,060]

you want to discard the fist row and take the other three, is it correct?

If this is your need, try this props.conf:

[hastsourcetype]
CHARSET = UTF-8
LINEBREAKER = ([\r\n]+)
MAXTIMESTAMPLOOKAHEAD = 29
SHOULDLINEMERGE = false
TRANSFORMS-remove-hash = eliminate-hash-item
description = hashsourcetype

and this transforms.conf:

[eliminate-hash-item]
REGEX=\[\s+\#\#\#\#
DEST_KEY=queue
FORMAT=nullQueue

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01 ,

you accepted this answer during the closing period so the acceptance was lost.

Could you accept again this answer?

Thank you.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @uagraw01,

Ok good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

uagraw01
Builder

@gcusello But not getting the resolution.

 

 

0 Karma

uagraw01
Builder

Thanks for answering .Please let me know if i use REGEX = (.*#<) would it work? Because it is correctly matched on regex101 engine. But when i used this on transforms.conf it is not remove any ####< char from events while indexing. Please suggest

0 Karma

uagraw01
Builder

In place of DELIMS = ####< i used REGEX= ####< also but event showing the same.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...