topic Re: Heavy Forwarder filter in Getting Data In

Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 09:51:57 GMT

Hi,

I have this example of log:

hhhh/mm/dd hh:mm:ss :MGR ,nnnnn:nnn(text):1 [-07728 text.] : Event : Done by > 'text'

I would like to filter by code "07728".

How can do it?

Thanks in advance, Best regards

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 12:52:51 GMT

By filter do you mean discard events containing the value 07728 ?

Then For example you could do:

In props.conf, set the TRANSFORMS-null attribute:

[source::<your_source_path]
TRANSFORMS-null= setnull

In transforms.conf:

[setnull]
REGEX=\[.*07728.*\]
DEST_KEY=queue
FORMAT=nullQueue

*UPDATE*

Keep specific events and discard the rest

In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::<your_source_path]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX=\[.*(07728|07729|07730).*\]
DEST_KEY = queue
FORMAT = indexQueue

Re: Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 13:39:50 GMT

Hi,

Thank you for your answer,
I would like to receive only the logo containing a list of codes for example 07728. The codes are about 80.
Just replace SetNull and nullQueue with setparsing and indexQueue?
How can I specify a list of values to be sent?

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 13:58:12 GMT

I have updated the previous answer and you will have to put the list of all the codes in the regex and between () separated by |

Re: Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 14:32:02 GMT

I tried with SetNull before setparsing but this way does not send events.
I tried without SetNull but this way sends all without filters.

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 14:50:08 GMT

can you post you props and transforms?

Re: Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 15:03:22 GMT

props.conf:

[prova_mgraudit]    
NO_BINARY_CHECK = 1    
pulldown_type = 1    
TRANSFORMS-set = mgraudit,setnull

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[mgraudit]
REGEX = REGEX=\[.*(07728|07895).*\]
DEST_KEY = queue
FORMAT = indexQueue

I have other filters so I used [mgraudit] instead of the file setparsing transforms.conf.
prova_mgraudit is the surcetype that I created via the web interface and corresponds to the log file that I want to filter.

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 15:38:13 GMT

in your props.conf setnull should be first:
TRANSFORMS-set = setnull,mgraudit

and you need to restart splunk

Re: Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 15:43:57 GMT

I tried with SetNull before mgraudit but this way does not send events.
I tried without SetNull but this way sends all without filters.
I tried with SetNull after mgraudit but this way does not send events.

After modifying the file I restarted splunk.
Anyway, now try again.

ps: are you Italian?

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 15:48:35 GMT

that's weird it should work... can you try for test on source instead of sourcetype?

no i am french... 😜 but my mum is sicilian...

Re: Heavy Forwarder filter

benedetto — Mon, 28 Sep 2020 12:31:12 GMT

[source::/home/splunk/mgraudit/mgraudit.log]

NO_BINARY_CHECK = 1

pulldown_type = 1

TRANSFORMS-set = setnull,mgraudit

In this way does not send events

[source::/home/splunk/mgraudit/mgraudit.log]

NO_BINARY_CHECK = 1

pulldown_type = 1

TRANSFORMS-set = mgraudit

In this way sends all without filters.
transforms.conf file is equal to the previous.

You know what it depends?

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 16:15:15 GMT

then it sounds like the regex doesnot match your data.

Can you post sample events?

Re: Heavy Forwarder filter

benedetto — Thu, 27 Sep 2012 16:19:18 GMT

Yes 🙂

2012/09/27 19:07:48 :MGR ,49660:301(entrust):1 [-07728 Entrust subsystem started.] : Event : Done by > 'Master User Master1' : entash subsystem - Entrust Authority (TM) Security Manager, version 7.1 SP3 Patch 154020(189) @ Jan 5 2010 14:29:19 (PID: 49660)

Thank you very much.

Re: Heavy Forwarder filter

MarioM — Thu, 27 Sep 2012 16:34:58 GMT

just seen a typo in your transforms.conf you have twice REGEX and i tried the config on my splunk and it worked.

Re: Heavy Forwarder filter

benedetto — Fri, 28 Sep 2012 07:31:07 GMT

You are Great!!

Thank you very much!!

Re: Heavy Forwarder filter

GreeshmaV — Thu, 19 Mar 2015 07:26:21 GMT

My event is like this "Type=Error, XXXXXXX, xxxxx, CMP=PUR, API=get Att, XXX, XXX" I need to filter it based on the CMP=PUR, I have tried REGEX = [.CMP=PUR.] but its not working can some one help plzzz