topic Re: Deduplicate events in Getting Data In

Deduplicate events- How can we override?

cyberhaven — Tue, 30 May 2023 03:38:03 GMT

We have a script as a data source, and sometimes events could be duplicated (same ID). Using | dedup id in the search helps, but we want to override events with the same ID if possible. We have tried some solutions from the internet and documentation, but they haven't helped.

props.conf
[incidents_script]

TZ = UTC

category = Splunk App Add-on Builder

pulldown_type = 1

python.version = python3

TRUNCATE = 1000000

INDEXED_EXTRACTIONS = json

TIMESTAMP_FIELDS = trigger_time

SHOULD_LINEMERGE = false

AUTO_KV_JSON = false

KV_MODE = none

TRANSFORMS-index = replace_existing deduplicate

REPORT-id = extract_id

TRANSFORMS-debug = debug_deduplicate

EXTRACT-id = "id"\s*:\s*"([^"]+)"

transforms.conf

[replace_existing]

REGEX = .

DEST_KEY = _SYS_CHECKSUM

FORMAT = index-replace

[deduplicate]

REGEX = .

MV_ADD = true

[debug_deduplicate]

REGEX = .

MV_ADD = true

[extract_id]

REGEX = "id"\s*:\s*"([^"]+)"

FORMAT = id::$1

Re: Deduplicate events

PickleRick — Sun, 28 May 2023 09:42:51 GMT

If I understand you correctly, you'd like to deduplicate events on ingest - either not ingest an event if there is already one with the same value of a field called ID or overwrite previous values of such field.

Well, that's not possible with native splunk functionalities.

1. Splunk ingestion process works one event at a time.

2. Splunk ingestion process works "one-way" - you can't "check what's already in the index". Remember that parsing can be performed way, way before the event even reaches the indexers (and different events from the same source can be processed on different components). Also, you don't have access to search-time extracted values during the ingestion process.

3. There is no "overwriting" in Splunk.

So if it's really essential for you that you don't ingest duplicated ID's, you need to design your own ingestion process that will keep your events deduplicated (but for that you'd need some buffer window which will increase latency).