topic Re: Exclude events in index time? in Getting Data In

Exclude events in index time?

aalhabbash1 — Wed, 06 Nov 2019 14:15:35 GMT

Hi Splunker;

we need to exclude the below, any event has the below form in index time, this event come to Splunk is has 2 forms.
I used the below configuration but not work, please help me in that:

Configuration:

in transforms.conf:

[setnull]
REGEX = pid
DEST_KEY = queue
FORMAT = nullQueue

In props.conf:

TRANSFORMS-null= setnull

Events:

first form:

{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:

{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

BR;

Re: Exclude events in index time?

gcusello — Wed, 06 Nov 2019 14:25:06 GMT

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

Re: Exclude events in index time?

aalhabbash1 — Thu, 07 Nov 2019 03:54:50 GMT

H Gcusello;

Yes I added it in indxers.

I used the below;

[my sourcetype]
TRANSFORMS-null= setnull

Thank you;

Re: Exclude events in index time?

gcusello — Thu, 07 Nov 2019 07:40:57 GMT

Hi aalhabbash1,
obviously you restarted Splunk after configuration update.

So, run a search in Splunk like this

index=your_index
| regex "pid"

This search should find the same events to discard, it's useful to understand if the regex is correct: if results are the events to discard, it's correct, otherwise you have to modify it.

Eventually, try with:

REGEX = (pid)

Ciao.
Giuseppe

Re: Exclude events in index time?

aalhabbash1 — Thu, 07 Nov 2019 08:35:06 GMT

Hi Gcusello;

Thank you for reply;

I run the below search and the regex is work:

index=your_index
| regex "pid"

And puted:
REGEX = (pid)

But when I push to indexers servers, the Splunk stop receiving logs from it, and when remove the it splunk return to receiving logs.

What is the problem.

BR;

Re: Exclude events in index time?

gcusello — Thu, 07 Nov 2019 09:02:30 GMT

Hi aalhabbash1,
this means that in all your events there the "pid" string, so you have to find a different differentiator to filter your events.

Ciao.
Giuseppe

Re: Exclude events in index time?

aalhabbash1 — Sun, 10 Nov 2019 05:35:20 GMT

Hi Gcusello;

Ok, can you make 1 regex for the below format logs, to success the exclude these logs?

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you;

Re: Exclude events in index time?

gcusello — Sun, 10 Nov 2019 09:18:40 GMT

Hi aalhabbash1,,
the problem is to find a regex to match only events to discard (or only events to take), if you share only one sample, we cannot be sure that the regex match only the events to discard (e.g. the previous regex), so you should share a sample of events to discard and a sample of events to take.

Ciao.
Giuseppe

Re: Exclude events in index time?

arjunpkishore5 — Sun, 10 Nov 2019 12:08:25 GMT

what field identifies these forms ? I would guess pid is available in all your events. If yes, then this will filter everything if REGEX is changed to (pid)

Re: Exclude events in index time?

aalhabbash1 — Sun, 10 Nov 2019 12:34:45 GMT

Ok I will provide you the sample logs from the events which I want and which don't want,

Please help me to make regex.

Format which I need:

2019-11-10 15:22:32.662 +0300 (Default,mydzit.gov.sa\kalsolai,4C8EC4A70F3149988B93370B16CADBEE-0:0,XcgBCNSpBpjM51ezrQYXrQAAAgE) catalina-exec-150 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.domain.session.SessionService - Non-guest user session found.

AND:

2019-11-10 15:22:40.816 +0300 (Default,mydzit.gov.sa\kalsolai,F853E8F10C2742CBB071A090C3047519-0:0,XcgBENSpBpjM51ezrQYXvwAAAZE) catalina-exec-151 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.model.vizql.util.WithSessionAspect - Command not allowed on a shared session F853E8F10C2742CBB071A090C3047519. Cloning to a private session and retrying.

Format which no need:

Thank you

Re: Exclude events in index time?

gcusello — Mon, 11 Nov 2019 08:23:59 GMT

Hi aalhabbash1,
try this regex

(?ms)pid.*?req

that you can test at https://regex101.com/r/oBQOe8/1

Ciao.
Giuseppe

Re: Exclude events in index time?

aalhabbash1 — Mon, 11 Nov 2019 09:22:19 GMT

Hi @gcusello

Done; good job.

I appreciate your effort 🙂

BR;

Re: Exclude events in index time?

gcusello — Mon, 11 Nov 2019 09:24:25 GMT

Hi @aalhabbash1,
you're welcome!
if this answer solves your problem, please accept and/or upvote it,
Ciao and next time!
Giuseppe