Getting Data In

Splunk event sourcetype overide and send event back to parsing queue

ankithreddy777
Contributor

I have a situation where I have to parse the data, especially timestamp extraction based on the keyword in the message.

like if event contains keyword "hello" I need to assigntimestamp from one field , if event contains "hi" I need to assign timestamp from other field during index time.

I tried to override source type based on matching keyword. But how to send the event back to parsing queue to re-assign timestamp based on new source type.

Or is there any other method achieve to assign timestamp based on keyword present in event.

0 Karma

tiagofbmm
Influencer

remember sourcetype overwrite is done in the end of the pipeline , just as cosmetic change for the latter search time field extractions and calculations.

I would still overwrite the sourcetype like you are doing but considering your use case of timestamp, I would create an EVAL in each sourcetype definition stating that _time=fieldA in one sourcetype and the respective thing on _time=fieldB on the other one.

Any time you later search for any of the sourcetype, your time will be showing modified.

tiagofbmm
Influencer

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that

0 Karma

imgarytan
Path Finder

Yes, similar post :
https://answers.splunk.com/answers/447812/per-event-sourcetype-overrides-not-actually-a-lot.html?utm...

Then anyone tried custom datetime.xml as below:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configuredatetimexml#Create_or_modify_a_cust...

Is this something complex but feasible? or better not even try it since the words "In nearly all cases, you do not need to modify datetime.xml" ?

Thanks

0 Karma

jluo_splunk
Splunk Employee
Splunk Employee

Could you post a sample of your data? That would be helpful 🙂

0 Karma

tiagofbmm
Influencer

Splunk data goes only once through the parsing pipeline so there is no way to do exactly what you are asking for

0 Karma

imgarytan
Path Finder

Looks someone tried Chaining Universal Forwarders to allow forwarding more than once.

Then, for technical discussion, can we try add an intermediate heavy forwarder (HF1) to do the event sourcetype overiding, then forward that to indexer or second heavy forwarder (HF2) to do the timestamp parsing again as per new sourcetypes? 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...