Splunk Search

SEDCMD not working on Heavy forwarder and Http Event Collect?

clorne
Communicator

Hello,

I have data collected through a Splunk HEC on a Heavy Forwarder.

The data has this structure:

2023-03-16T16:59:01+01:00 serverIP event_info [data1][datat2] {json_data}.

I want to get the json_data indexed as raw data. I have tried several regex with SEDCMD. I have tried several regex that are all working on a standalone Splunk but they have no effect with the configuration Splunk HF->Splunk IDX

Here is my latest SEDCMD: SEDCMD-json=s/^[^{]+//g

Currently there is no TA on the Splunk indexer and I am wondering if this is the cause of the issue. Is SEDCMD compatible with HEC ? 

Regards

Labels (1)
Tags (1)
0 Karma

clorne
Communicator

This only difference is that this is not a json data and the SEDCMD is successful and it is not done on the indexer.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The document I linked to is not canonical so it may have errors.  If you found instances where SEDCMD works then that's the better answer.

---
If this reply helps you, Karma would be appreciated.

clorne
Communicator

I am still checking on other TA. But I have seen one where the SEDCMD is working with the HEC collection. Therefore I do not know what to think 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

HEC events pass through a different pipeline that doesn't include SEDCMD.  See https://www.aplura.com/assets/pdf/hec_pipelines.pdf

---
If this reply helps you, Karma would be appreciated.

PickleRick
SplunkTrust
SplunkTrust

Actually, this document shows that hec-ingested events go through filtering/ routing . They just might skip timestamp parsing and line breaking/ line merging. Generally speaking, events pushed to hec should get through transforms normally (including sedcmd).

Question is whether the props calling those transforms are properly configured (right sourcetype/source/host), are the transforms called at all and so on.

0 Karma

clorne
Communicator

Yes, the configuration is correct. At the beginning we had a complicated transform to remove the header. It was working partially 75%, therefore we decided to use the SEDCMD.

0 Karma

clorne
Communicator

Hello,

Thanks for your reply Richgalloway.

So, do you think that put the TA (at least the sedcmd part) on the indexer would solve my problem ?

I just checked in the other TAs used by  my company and it seems that they have put the TA on the HF AND on the indexer. Therefore SEDCMD should work on it ?

Regards

Céline

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Events are parsed on the first "heavy" component they go through (unless there is some very strange configuration in place which skips some queues) so your HEC-ingested events _should_ be processed on the HF and pushed as parsed to indexers where they would not be touched anymore.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...