Getting Data In

How to remove a field from data before indexing?

tdiestel
Path Finder

Hi All;

I have an interesting issue. Currently, I have data free flowing into a port on in Splunk, and one of the fields in this data has become corrupt and is not allowing me to search my data correctly. What I want to do is remove this field from the data before it is indexed. Is there any way I can do this in Splunk itself?

Note: I really want to avoid sending the data else where for this change to be made and then sending it to Splunk, and I would want to not be limited the option of changing the field in the source.

Any suggestions are greatly appreciated as always.

Thanks,
Tyler

0 Karma

jeffland
Champion

I would advise against deleting the time information after Last_Action. What if you wanted to use it in a search?
You could simply tell splunk where to break events and where to look for the timestamp of the event itself explicitly, like so:

[mobile]
LINE_BREAKER=([\r\n]+)\d{4}-
SHOULD_LINEMERGE=false
TIME_PREFIX=^

This should break your events properly and still retain all data.

tdiestel
Path Finder

These are good pointers, and I'm still trying to see if I can make this work. To be more specific of the field that is corrupting my data is this field has a timestamp component to it.

Scenario: A single event is sent to splunk that looks like this

2015-12-02T15:34:45-0800
User: Jim
Event_Name: "Click_Event"
Action_Type: "Lower_Menu_Item"
Last_Action: "click_2015-12-01T12:00:00-0800"
Last_Action_Type: "Upper Right Button"

Splunk then indexes this single event as 2 events:

One like this:

_time: 2015-12-02T15:34:45-0800
2015-12-02T15:34:45-0800
User: Jim
Event_Name: "Click_Event"
Action_Type: "Lower_Menu_Item"

The other like this:

_time: 2015-12-01T12:00:00-0800"
Last_Action: "click_2015-12-01T12:00:00-0800"
Last_Action_Type: "Upper Right Button"

End Goal: Stop splunk from splitting up my events.
Would settle for removing the "Last_Action" field if I can do it before splunk splits the event.

0 Karma

woodcock
Esteemed Legend

Then you asked the wrong question. See what @jeffland said.

0 Karma

tdiestel
Path Finder

Tried this in our props.conf file just to remove the field entirely but still no success. Is there something I'm doing wrong?

[mobile]
SEDCMD-nonrequiredtimestamps = s/[Last_Action =].*/Last_Action =/g

0 Karma

woodcock
Esteemed Legend
0 Karma

techboyt28
Engager

techboyt28
Engager

I came across this document see if it's of any help.

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...