Duplicate field values in Splunk events from Cribl

[Raffaele53]

Hello,

I’m using Cribl Cloud to pull JSON events from an Azure Event Hub and forward them to Splunk via HEC.

Each incoming event (on Cribl) contains a nested array field called records, for example:

{
  "records": [
    {
      "FileName": "xx",
      "FileType": "xx",
      "NetworkMessageId": "xx",
      "RecipientEmailAddress": "xx",
      "RecipientObjectId": "xx",
      "ReportId": "xx",
      "SHA256": "xx",
      "SenderDisplayName": "xx",
      "SenderObjectId": "x",
      "SenderFromAddress": "x",
      "FileSize": x,
      "Timestamp": "xx",
      "TimeGenerated": "xx",
      "_ItemId": "xx",
      "TenantId": "xx",
      "_TimeReceived": "xx",
      "_Internal_WorkspaceResourceId": "xx",
      "Type": "xx"
    },
    {
            "FileName": "xx",
      "FileType": "xx",
      "NetworkMessageId": "xx",
      "RecipientEmailAddress": "xx",
      "RecipientObjectId": "xx",
      "ReportId": "xx",
      "SHA256": "xx",
      "SenderDisplayName": "xx",
      "SenderObjectId": "x",
      "SenderFromAddress": "x",
      "FileSize": x,
      "Timestamp": "xx",
      "TimeGenerated": "xx",
      "_ItemId": "xx",
      "TenantId": "xx",
      "_TimeReceived": "xx",
      "_Internal_WorkspaceResourceId": "xx",
      "Type": "xx"
    },
    {
            "FileName": "xx",
      "FileType": "xx",
      "NetworkMessageId": "xx",
      "RecipientEmailAddress": "xx",
      "RecipientObjectId": "xx",
      "ReportId": "xx",
      "SHA256": "xx",
      "SenderDisplayName": "xx",
      "SenderObjectId": "x",
      "SenderFromAddress": "x",
      "FileSize": x,
      "Timestamp": "xx",
      "TimeGenerated": "xx",
      "_ItemId": "xx",
      "TenantId": "xx",
      "_TimeReceived": "xx",
      "_Internal_WorkspaceResourceId": "xx",
      "Type": "xx"
    }
  ],
  "_time": 1756902850.057,
  "cribl": "yes",
  "security_event_hub": "yes"
}

My goal is to split each element of the records array into a separate, flat event. Here’s what I’ve tried:

• Unroll function (Cribl) on records to produce individual events

• Flatten function (Cribl) to promote nested fields and delete records array

In Splunk, each field’s values are duplicated (and sometimes triplicated), as shown here: (censored values are equals between them)

I’ve identified that extracting nested values is causing this anomaly in Splunk.

I’ve tried numerous approaches to resolve it:

• Replaced the Flatten function with an Eval expression like that (Cribl):
  Object.assign(__e, Object.assign({}, __e, __e.rec || {})); delete __e.rec; delete __e.records;

• Tested various JavaScript snippets in Code functions (Cribl)

• Used JSON Unroll and JSON Decode functions (Cribl)

• Toggled KV_MODE, AUTO_KV_JSON, and INDEXED_EXTRACTIONS on Heavy Forwarders and Search Heads

None of these solutions work consistently; in some cases values were even triplicated.
Do you have any suggestions to resolve this issue?

Thank you in advance for any insights or working examples.

[m_pham]

I think your question may be better answered by the Cribl community first as Cribl is modifying the JSON events before it hits Splunk. 

I recommend reviewing this doc to properly format JSON event for Splunk HEC destination: https://help.splunk.com/en/splunk-cloud-platform/get-data-in/get-started-with-getting-data-in/10.6/g...

Here are some possibilities to your duplicate field values:

Cribl's Splunk HEC destination turns top-level fields into indexed fields. If an event has a _raw, Cribl sends _raw as the HEC event body. Every other top-level field (except _time, host, source, sourcetype, index) goes in the HEC fields object, so Splunk stores it as an index-time field. After Flatten, FileName, SHA256 and the rest are all top-level, so they all become indexed fields.

_raw still holds the JSON. Unroll copies the whole event, including _raw, into each new event. Flatten and their Object.assign eval only change event fields; neither touches _raw. So _raw is still JSON, possibly the original Event Hub payload with the whole records array. Search-time JSON extraction (KV_MODE=json, or AUTO_KV_JSON under KV_MODE=auto) pulls the same keys out of _raw again. 

Just some questions:

- What is the splunk hec endpoint you're using? raw? event?

- Can you double check which are indexed fields?

| tstats count where index=<idx> sourcetype=<st> by FileName
- If this returns results, FileName is an indexed field, which points at the Cribl HEC fields path.

| walklex index=<idx> type=field
- This lists every indexed field name.

- Can you show us what the sourcetype configuration is set to on your heavy forwarder vs what is on the search head? You should have the big 6 props configuration for your sourcetype if you haven't already done so.

-  Not sure what you can or can't do in your environment but you're already running an HF, have you tried using Splunk Add-on for Microsoft Cloud Services TA to ingest the data?

[nunoaragao]

I've had this experience, not with Azure Event Hub, but OpenAI Connector.
My assumption is that Cribl Connector automatically adds top-level fields from JSON object to metadata, and we can't do anything about it. Messing with KV MODE on Splunk side only adds entropy

My fix was to add an Eval function on Cribl's Pipeline to remove all fields except _raw (I already had such a function in order to hardcode the index/source)

[PickleRick]

1. You're showing us only the resulting fields without raw message contents. It's impossible to say what your _raw looks like. But typically duplication of values occurs when you have both fields extracted index time by indexed extraction as well as search-time extracted ones by kv_mode.

2. As @livehybrid said - first thing to do is check what's going on in your Cribl and what it does to your data and how it sends it to Splunk. And most probably fix it there. But that's beyond the scope of this forum.

[Raffaele53]

Sorry, I forgot to include the raw event going out from Cribl and coming into Splunk.

Raw event extracted from Splunk search:

{"cribl":"yes","security_event_hub":"yes","NetworkMessageId":"xxx","ReportId":"xxx","Timestamp":"2025-09-04T07:04:14.0000000Z","Url":"xxx","UrlDomain":"xxx","UrlLocation":"Body","TimeGenerated":"2025-09-04T07:04:14.0000000Z","_ItemId":"xxx","TenantId":"xxx","_TimeReceived":"2025-09-04T07:07:00.1639029Z","_Internal_WorkspaceResourceId":"xxx","Type":"EmailUrlInfo"}

Each field and its value appear only once.

On Cribl, the event looks the same:

{
  "_time": 1756969933.064,
  "cribl": "yes",
  "security_event_hub": "yes",
  "NetworkMessageId": "xxx",
  "ReportId": "xxx",
  "Timestamp": "2025-09-04T07:08:47.0000000Z",
  "Url": "xxx",
  "UrlDomain": "xxx",
  "UrlLocation": "Body",
  "TimeGenerated": "2025-09-04T07:08:47.0000000Z",
  "_ItemId": "xxx",
  "TenantId": "xxx",
  "_TimeReceived": "2025-09-04T07:12:07.2054241Z",
  "_Internal_WorkspaceResourceId": "xxx",
  "Type": "EmailUrlInfo",
  "cribl_pipe": "Azure_Event_Hub_processing"
}

I tried different combinations of KV_MODE, AUTO_KV_JSON, and INDEXED_EXTRACTIONS:

• On the Heavy Forwarder:

  • KV_MODE=JSON, then KV_MODE=none

  • INDEXED_EXTRACTIONS=JSON

  • AUTO_KV_JSON=none

  • and also all of them combined

• Afterwards, I tried:

  • On the Search Head: KV_MODE=none and AUTO_KV_JSON=none

  • On the Heavy Forwarder: INDEXED_EXTRACTIONS=JSON

But nothing changed.

On Cribl, I also tried several functions to delete and recreate fields/values, but that didn’t work either.

Do you have any suggestions?

Thanks a lot!

[PickleRick]

If you have a json event, don't use indexed extractions.

Having said that - raw event is one thing but cribl _might_ (I have no idea if it does in your case) be adding indexed fields anyway.

[Raffaele53]

I tried using Indexed Extractions to see if anything would change, but it didn’t work.

On the Cribl side, I only see the event going out and nothing else.

Thanks for the help—hopefully someone’s run into this before and can help me!

[livehybrid]

Hi @Raffaele53 

Im not too familiar with breaking up events in Cribl but if these are being sent as parsed events to Splunk then this should be done before it reaches Splunk. 

When you preview the output from Cribl do you see the raw json, output, or do you also see the fields? 

I'd start off by making sure you are happy with the output from Cribl using the preview option and then checking the KV_MODE etc is correect on the Splunk side to match as required.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Raffaele53]

Hi,

Thanks for your reply.
I shared the raw events going out from Cribl and coming into Splunk in my other message.

On the Cribl side, I can only see the raw events being sent to Splunk, while field extraction can only be checked on the Splunk side.

It’s strange because if I don’t extract fields from nested "records" field, everything works correctly.
It’s as if Cribl leaves some old metadata in the events, referring to fields that were previously modified.