Exclude events in index time?

[aalhabbash1]

Hi Splunker;

we need to exclude the below, any event has the below form in index time, this event come to Splunk is has 2 forms.
I used the below configuration but not work, please help me in that:

Configuration:

in transforms.conf:

[setnull]
REGEX = pid
DEST_KEY = queue
FORMAT = nullQueue

In props.conf:

TRANSFORMS-null= setnull

Events:

first form:

{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:

{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

BR;

[arjunpkishore5]

what field identifies these forms ? I would guess pid is available in all your events. If yes, then this will filter everything if REGEX is changed to (pid)

[gcusello · Accepted solution]

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

[aalhabbash1]

H Gcusello;

Yes I added it in indxers.

I used the below;

[my sourcetype]
TRANSFORMS-null= setnull

Thank you;

[gcusello]

Hi aalhabbash1,
obviously you restarted Splunk after configuration update.

So, run a search in Splunk like this

index=your_index
| regex "pid"

This search should find the same events to discard, it's useful to understand if the regex is correct: if results are the events to discard, it's correct, otherwise you have to modify it.

Eventually, try with:

REGEX = (pid)

Ciao.
Giuseppe

[aalhabbash1]

Hi Gcusello;

Thank you for reply;

I run the below search and the regex is work:

index=your_index
| regex "pid"

And puted:
REGEX = (pid)

But when I push to indexers servers, the Splunk stop receiving logs from it, and when remove the it splunk return to receiving logs.

What is the problem.

BR;

[gcusello]

Hi aalhabbash1,
this means that in all your events there the "pid" string, so you have to find a different differentiator to filter your events.

Ciao.
Giuseppe

[aalhabbash1]

Hi Gcusello;

Ok, can you make 1 regex for the below format logs, to success the exclude these logs?

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you;

[gcusello]

Hi aalhabbash1,,
the problem is to find a regex to match only events to discard (or only events to take), if you share only one sample, we cannot be sure that the regex match only the events to discard (e.g. the previous regex), so you should share a sample of events to discard and a sample of events to take.

Ciao.
Giuseppe

[aalhabbash1]

Ok I will provide you the sample logs from the events which I want and which don't want,

Please help me to make regex.

Format which I need:

2019-11-10 15:22:32.662 +0300 (Default,mydzit.gov.sa\kalsolai,4C8EC4A70F3149988B93370B16CADBEE-0:0,XcgBCNSpBpjM51ezrQYXrQAAAgE) catalina-exec-150 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.domain.session.SessionService - Non-guest user session found.

AND:

2019-11-10 15:22:40.816 +0300 (Default,mydzit.gov.sa\kalsolai,F853E8F10C2742CBB071A090C3047519-0:0,XcgBENSpBpjM51ezrQYXvwAAAZE) catalina-exec-151 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.model.vizql.util.WithSessionAspect - Command not allowed on a shared session F853E8F10C2742CBB071A090C3047519. Cloning to a private session and retrying.

Format which no need:

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you

[gcusello]

Hi aalhabbash1,
try this regex

(?ms)pid.*?req

that you can test at https://regex101.com/r/oBQOe8/1

Ciao.
Giuseppe

[aalhabbash1]

Hi @gcusello

Done; good job.

I appreciate your effort 🙂

BR;

[gcusello]

Hi @aalhabbash1,
you're welcome!
if this answer solves your problem, please accept and/or upvote it,
Ciao and next time!
Giuseppe