SEDCMD not working on Heavy forwarder and Http Event Collect?

[clorne]

Hello,

I have data collected through a Splunk HEC on a Heavy Forwarder.

The data has this structure:

2023-03-16T16:59:01+01:00 serverIP event_info [data1][datat2] {json_data}.

I want to get the json_data indexed as raw data. I have tried several regex with SEDCMD. I have tried several regex that are all working on a standalone Splunk but they have no effect with the configuration Splunk HF->Splunk IDX

Here is my latest SEDCMD: SEDCMD-json=s/^[^{]+//g

Currently there is no TA on the Splunk indexer and I am wondering if this is the cause of the issue. Is SEDCMD compatible with HEC ? 

Regards

[clorne]

This only difference is that this is not a json data and the SEDCMD is successful and it is not done on the indexer.

[richgalloway]

The document I linked to is not canonical so it may have errors.  If you found instances where SEDCMD works then that's the better answer.

[clorne]

I am still checking on other TA. But I have seen one where the SEDCMD is working with the HEC collection. Therefore I do not know what to think 

[richgalloway]

HEC events pass through a different pipeline that doesn't include SEDCMD.  See https://www.aplura.com/assets/pdf/hec_pipelines.pdf

[PickleRick]

Actually, this document shows that hec-ingested events go through filtering/ routing . They just might skip timestamp parsing and line breaking/ line merging. Generally speaking, events pushed to hec should get through transforms normally (including sedcmd).

Question is whether the props calling those transforms are properly configured (right sourcetype/source/host), are the transforms called at all and so on.

[clorne]

Yes, the configuration is correct. At the beginning we had a complicated transform to remove the header. It was working partially 75%, therefore we decided to use the SEDCMD.

[clorne]

Hello,

Thanks for your reply Richgalloway.

So, do you think that put the TA (at least the sedcmd part) on the indexer would solve my problem ?

I just checked in the other TAs used by  my company and it seems that they have put the TA on the HF AND on the indexer. Therefore SEDCMD should work on it ?

Regards

Céline

[PickleRick]

Events are parsed on the first "heavy" component they go through (unless there is some very strange configuration in place which skips some queues) so your HEC-ingested events _should_ be processed on the HF and pushed as parsed to indexers where they would not be touched anymore.