topic Re: How to remove all events containing specific values in Splunk? in Getting Data In

How to remove all events containing specific values in Splunk?

Hemnaath — Mon, 24 Apr 2017 17:21:22 GMT

Hi All, Can any one guide me, on how to remove all events containing only the below listed events from rest of the events ? I am sure that we need to configure props.conf and transforms.conf , but not sure what to configure in props and how to right a regex in transforms.conf to remove the events .

Event Timestamp: April 24, 2017 12:54:34

Event Timestamp: April 24, 2017 12:55:30
Event Timestamp: April 24, 2017 12:56:34

Kindly guide me how to configure to remove the above events from rest of the events.

Re: How to remove all events containing specific values in Splunk?

Hemnaath — Mon, 24 Apr 2017 17:50:04 GMT

Hi All, I have configured props.conf and transforms.conf to discard specific events and keep the rest events in splunk. But not sure whether this is the correct regex to do so. Kindly help in correcting the regex and setting it right to remove specific events and keep the rest events indexed.

Discard specific events and keep the rest events

props.conf

[source::/opt/syslogs/network/.../xxx.log] --- > Source of the file from where splunk reads the data
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [Event Timestamp: April 24, 2017 12:54:34] -- > events to be removed from the data
DEST_KEY = queue
FORMAT = nullQueue

Kindly guide me whether the above props and transforms.conf can be used to remove specific events and keep the rest.

Re: How to remove all events containing specific values in Splunk?

Richfez — Mon, 24 Apr 2017 17:55:10 GMT

Firstly, do you actually need to *delete* the events from the indexes? If that's the case, then...

Carefully craft a search that returns those rows and ONLY those rows. There's isn't enough information here to know precisely what it is that search will look like. Maybe index=X sourcetype=Y "Event Timestamp: April 24, 2017 12:56:34" OR "Event Timestamp: April 24, 2017 12:54:34" OR "Event Timestamp: April 24, 2017 12:55:30" Make double sure this works right. Then follow the steps in the documentation on removing data from indexes using YOUR search to prevent those results from showing up again.

If instead you only need to make sure you don't index those when they come in, you'll have to be more specific about why you want them gone. Much like the problem above where I had to say I don't know precisely what it is you are trying to delete, I don't know exactly what it is you are trying to remove. Are these empty lines (basically) with just a timestamp? Where are they coming from? How do they relate to the log lines you WANT to keep? Best would be to remove them from the sending source (or configure your syslog collector or whatever to ignore them). We can go down this route if necessary (and probably someone will!)

Re: How to remove all events containing specific values in Splunk?

Richfez — Mon, 24 Apr 2017 17:57:17 GMT

I just answered - this comment is very helpful and I wish we wouldn't have been both updating at the same time. Still, the answer I gave on deleting them may be useful to others. 🙂

So, do you need to not index these events, or do you need to delete the events already indexed?

The answer I gave below should help with actually deleting them. If you need to not index them then it will require a different answer, much more like your technique above.

Re: How to remove all events containing specific values in Splunk?

Richfez — Mon, 24 Apr 2017 17:58:58 GMT

To recap "how to delete data" from the docs, it's basically
1) Add "delete" capabilities to a role (preferably use a special user)
2) Log in as that role
3) Create a search that returns the events you want gone. Like mysearchhere
4) mysearchhere | delete
5) Confirm.

Re: How to remove all events containing specific values in Splunk?

Hemnaath — Mon, 24 Apr 2017 18:09:30 GMT

Hi rich, actually we are getting the below list of events along with other useful events containing detailed information, so we wanted to remove the below un-wanted events from being indexed in to splunk.

search query : index=x sourcetype = xxx:xxx:xxx

output:
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx

So to discard the above events and keep the rest of the actual events, I had created a props.conf and transform.conf but not sure whether above configuration will help in removing the data.
Kindly guide in doing in this.

Re: How to remove all events containing specific values in Splunk?

Hemnaath — Mon, 24 Apr 2017 19:38:50 GMT

Hi All, can any one guide me on whether the below props and transforms.conf can be used to remove the unwanted data from getting into splunk. i.e is to discard specific events and keep the rest in splunk.

Discard specific events and keep the rest events

props.conf

[source::/opt/syslogs/network/.../xxx.log] --- > Source of the file from where splunk reads the data
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [Event Timestamp: April 24, 2017 12:54:34] -- > events to be removed from the data
DEST_KEY = queue
FORMAT = nullQueue

thanks in advance

Re: How to remove all events containing specific values in Splunk?

somesoni2 — Mon, 24 Apr 2017 20:17:35 GMT

Seems to be correct and inline with what Splunk suggests. (http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest). Make sure it's setup in the Splunk instance which does the parsing (Indexer or heavy forwarder, whichever comes first in the flow) and restart Splunk.

Re: How to remove all events containing specific values in Splunk?

Hemnaath — Tue, 25 Apr 2017 17:27:24 GMT

Hi Somesoni, thanks I have modified and used the above props and transforms.conf in Heavy Forwarder instances to remove the above events and keep the rest of the events to be indexed.

Modified Transforms.conf

[setnull]
REGEX = Event Timestamp
DEST_KEY = queue
FORMAT = nullQueue

After implementing the transforms.conf, the events are removed and could see other events getting indexed now.

Re: How to remove all events containing specific values in Splunk?

bogdan_nicolesc — Thu, 20 Sep 2018 14:46:24 GMT

Hi there!

I've got something similar with this:

source="WinEventLog:Security" | eval EventCode=if(EventCode="4801","NewName4801",EventCode) | eval EventCode=if(EventCode="4800","NameNew4800",EventCode) | eval EventCode=if(EventCode="4625","CertainDescription4625",EventCode)

But now i want to remove some EventCode from Field List in order to remain only 4801, 4800, 4625.

How do i do that? What do i have to type next in serch tab?

Thank you.

Re: How to remove all events containing specific values in Splunk?

bogdan_nicolesc — Mon, 24 Sep 2018 11:06:52 GMT

Found my answer from here: link text

So i had to make a list of codes to be removed from searchresult and enter it in search bar:

source="WinEventLog:Security" Type=* NOT (EventCode=4719 OR EventCode=4624 OR EventCode=4672 OR EventCode=4627 OR EventCode=4634 OR EventCode=4648 OR EventCode=4688 OR EventCode=4616 OR EventCode=4826 OR EventCode=4957 OR EventCode=4776 OR EventCode=1100 OR EventCode=4902 OR EventCode=4647 OR EventCode=1101 OR EventCode=4696 OR EventCode=4905 OR EventCode=4904) | eval EventCode=if(EventCode="4801","Deblocat4801",EventCode) | eval EventCode=if(EventCode="4801","NewName4801",EventCode) | eval EventCode=if(EventCode="4800","NameNew4800",EventCode) | eval EventCode=if(EventCode="4625","CertainDescription4625",EventCode)