topic Re: How to remove a field from data before indexing? in Getting Data In

How to remove a field from data before indexing?

tdiestel — Wed, 02 Dec 2015 19:23:32 GMT

Hi All;

I have an interesting issue. Currently, I have data free flowing into a port on in Splunk, and one of the fields in this data has become corrupt and is not allowing me to search my data correctly. What I want to do is remove this field from the data before it is indexed. Is there any way I can do this in Splunk itself?

Note: I really want to avoid sending the data else where for this change to be made and then sending it to Splunk, and I would want to not be limited the option of changing the field in the source.

Any suggestions are greatly appreciated as always.

Thanks,
Tyler

Re: How to remove a field from data before indexing?

techboyt28 — Wed, 02 Dec 2015 19:40:50 GMT

I came across this document see if it's of any help.

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad

Re: How to remove a field from data before indexing?

techboyt28 — Wed, 02 Dec 2015 19:41:21 GMT

See if this helps,
http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad

Re: How to remove a field from data before indexing?

woodcock — Wed, 02 Dec 2015 21:18:13 GMT

Like this:

http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Anonymizedatausingconfigurationfiles

Re: How to remove a field from data before indexing?

tdiestel — Wed, 02 Dec 2015 23:53:45 GMT

These are good pointers, and I'm still trying to see if I can make this work. To be more specific of the field that is corrupting my data is this field has a timestamp component to it.

Scenario: A single event is sent to splunk that looks like this

2015-12-02T15:34:45-0800
User: Jim
Event_Name: "Click_Event"
Action_Type: "Lower_Menu_Item"
Last_Action: "click_2015-12-01T12:00:00-0800"
Last_Action_Type: "Upper Right Button"

Splunk then indexes this single event as 2 events:

One like this:

_time: 2015-12-02T15:34:45-0800
2015-12-02T15:34:45-0800
User: Jim
Event_Name: "Click_Event"
Action_Type: "Lower_Menu_Item"

The other like this:

_time: 2015-12-01T12:00:00-0800"
Last_Action: "click_2015-12-01T12:00:00-0800"
Last_Action_Type: "Upper Right Button"

End Goal: Stop splunk from splitting up my events.
Would settle for removing the "Last_Action" field if I can do it before splunk splits the event.

Re: How to remove a field from data before indexing?

tdiestel — Tue, 29 Sep 2020 08:03:59 GMT

Tried this in our props.conf file just to remove the field entirely but still no success. Is there something I'm doing wrong?

[mobile]
SEDCMD-nonrequiredtimestamps = s/[Last_Action =].*/Last_Action =/g

Re: How to remove a field from data before indexing?

jeffland — Thu, 03 Dec 2015 08:31:35 GMT

I would advise against deleting the time information after Last_Action. What if you wanted to use it in a search?
You could simply tell splunk where to break events and where to look for the timestamp of the event itself explicitly, like so:

[mobile]
LINE_BREAKER=([\r\n]+)\d{4}-
SHOULD_LINEMERGE=false
TIME_PREFIX=^

This should break your events properly and still retain all data.

Re: How to remove a field from data before indexing?

woodcock — Thu, 03 Dec 2015 17:45:50 GMT

Then you asked the wrong question. See what @jeffland said.