Getting Data In

Exclude events in index time?

aalhabbash1
Path Finder

Hi Splunker;

we need to exclude the below, any event has the below form in index time, this event come to Splunk is has 2 forms.
I used the below configuration but not work, please help me in that:

Configuration:

in transforms.conf:

[setnull]
REGEX = pid
DEST_KEY = queue
FORMAT = nullQueue

In props.conf:

TRANSFORMS-null= setnull

Events:

first form:

{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:

{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

BR;

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

View solution in original post

0 Karma

arjunpkishore5
Motivator

what field identifies these forms ? I would guess pid is available in all your events. If yes, then this will filter everything if REGEX is changed to (pid)

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
where do you located these props.conf and transforms.conf files?
they must stay on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, what do you used in props before TRANSFORMS-null= setnull ? sourcetype, host?

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

H Gcusello;

Yes I added it in indxers.

I used the below;

[my sourcetype]
TRANSFORMS-null= setnull

Thank you;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
obviously you restarted Splunk after configuration update.

So, run a search in Splunk like this

index=your_index
| regex "pid"

This search should find the same events to discard, it's useful to understand if the regex is correct: if results are the events to discard, it's correct, otherwise you have to modify it.

Eventually, try with:

REGEX = (pid)

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Hi Gcusello;

Thank you for reply;

I run the below search and the regex is work:

index=your_index
| regex "pid"

And puted:
REGEX = (pid)

But when I push to indexers servers, the Splunk stop receiving logs from it, and when remove the it splunk return to receiving logs.

What is the problem.

BR;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
this means that in all your events there the "pid" string, so you have to find a different differentiator to filter your events.

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Hi Gcusello;

Ok, can you make 1 regex for the below format logs, to success the exclude these logs?

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,,
the problem is to find a regex to match only events to discard (or only events to take), if you share only one sample, we cannot be sure that the regex match only the events to discard (e.g. the previous regex), so you should share a sample of events to discard and a sample of events to take.

Ciao.
Giuseppe

0 Karma

aalhabbash1
Path Finder

Ok I will provide you the sample logs from the events which I want and which don't want,

Please help me to make regex.

Format which I need:

2019-11-10 15:22:32.662 +0300 (Default,mydzit.gov.sa\kalsolai,4C8EC4A70F3149988B93370B16CADBEE-0:0,XcgBCNSpBpjM51ezrQYXrQAAAgE) catalina-exec-150 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.domain.session.SessionService - Non-guest user session found.

AND:

2019-11-10 15:22:40.816 +0300 (Default,mydzit.gov.sa\kalsolai,F853E8F10C2742CBB071A090C3047519-0:0,XcgBENSpBpjM51ezrQYXvwAAAZE) catalina-exec-151 : INFO wgsessionId=rd9xUhIUR92tuPbGQkZrNQ com.tableausoftware.model.vizql.util.WithSessionAspect - Command not allowed on a shared session F853E8F10C2742CBB071A090C3047519. Cloning to a private session and retrying.

Format which no need:

first form:
{ [-]
k: msg
pid: 37552
req: -
sess: -
sev: info
site: -
tid: 4958
ts: 2019-11-06T17:06:06.305
user: -
v: Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)
}

Second Form:
{"ts":"2019-11-06T17:06:06.305","pid":37552,"tid":"4958","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 73,793,536 bytes (current process);52,410,015,744 bytes (Tableau total); 49,979,158,528 bytes (total of all processes); 30 (info count)"}

Thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aalhabbash1,
try this regex

(?ms)pid.*?req

that you can test at https://regex101.com/r/oBQOe8/1

Ciao.
Giuseppe

aalhabbash1
Path Finder

Hi @gcusello

Done; good job.

I appreciate your effort 🙂

BR;

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @aalhabbash1,
you're welcome!
if this answer solves your problem, please accept and/or upvote it,
Ciao and next time!
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...