Getting Data In

Exclude events in index time?

aalhabbash1
Path Finder

Hi Splunker;

we need to exclude the below, any event has the below form in index time, this event come to Splunk is has 2 forms.
I used the below configuration but not work, please help me in that:

Configuration:

in transforms.conf:

[setnull]
REGEX = pid
DEST_KEY = queue
FORMAT = nullQueue

In props.conf:

TRANSFORMS-null= setnull

Events:

first form:

{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:

{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

BR;

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

View solution in original post

0 Karma

arjunpkishore5
Motivator

what field identifies these forms ? I would guess pid is available in all your events. If yes, then this will filter everything if REGEX is changed to (pid)

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

H Gcusello;

Yes I added it in indxers.

I used the below;

[my sourcetype]
TRANSFORMS-null= setnull

Thank you;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
obviously you restarted Splunk after configuration update.

So, run a search in Splunk like this

index=your_index
| regex "pid"

This search should find the same events to discard, it's useful to understand if the regex is correct: if results are the events to discard, it's correct, otherwise you have to modify it.

Eventually, try with:

REGEX = (pid)

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Hi Gcusello;

Thank you for reply;

I run the below search and the regex is work:

index=your_index
| regex "pid"

And puted:
REGEX = (pid)

But when I push to indexers servers, the Splunk stop receiving logs from it, and when remove the it splunk return to receiving logs.

What is the problem.

BR;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
this means that in all your events there the "pid" string, so you have to find a different differentiator to filter your events.

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Hi Gcusello;

Ok, can you make 1 regex for the below format logs, to success the exclude these logs?

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,,
the problem is to find a regex to match only events to discard (or only events to take), if you share only one sample, we cannot be sure that the regex match only the events to discard (e.g. the previous regex), so you should share a sample of events to discard and a sample of events to take.

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Ok I will provide you the sample logs from the events which I want and which don't want,

Please help me to make regex.

Format which I need:

2019-11-10 15:22:32.662 +0300 (Default,mydzit.gov.sa\kalsolai,4C8EC4A70F3149988B93370B16CADBEE-0:0,XcgBCNSpBpjM51ezrQYXrQAAAgE) catalina-exec-150 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.domain.session.SessionService - Non-guest user session found.

AND:

2019-11-10 15:22:40.816 +0300 (Default,mydzit.gov.sa\kalsolai,F853E8F10C2742CBB071A090C3047519-0:0,XcgBENSpBpjM51ezrQYXvwAAAZE) catalina-exec-151 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.model.vizql.util.WithSessionAspect - Command not allowed on a shared session F853E8F10C2742CBB071A090C3047519. Cloning to a private session and retrying.

Format which no need:

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
try this regex

(?ms)pid.*?req

that you can test at https://regex101.com/r/oBQOe8/1

Ciao.
Giuseppe

aalhabbash1
Path Finder

Hi @gcusello

Done; good job.

I appreciate your effort 🙂

BR;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @aalhabbash1,
you're welcome!
if this answer solves your problem, please accept and/or upvote it,
Ciao and next time!
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk Life | Splunk is Officially Part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint. Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...