topic Re: Splunk duplicates events from a custom-made JSON event file in Getting Data In

Splunk duplicates events from a custom-made JSON event file

Vyber90 — Sat, 31 Jul 2021 17:32:31 GMT

What I'm doing is: I am doing stuff by my own an then parsing all the information as a JSON in order to append it to the end of an Splunk indexed file.

The problem is: Splunk, instead of just adding the information that wasn't there when the file was last updated, adds all the information again as if it was completely new, and thus giving me duplicate information.

I tried to append the information without altering what was already there, but this doesn't seem to solve anything. What would you do to only add the new info?

Re: Splunk duplicates events from a custom-made JSON event file

scelikok — Sun, 01 Aug 2021 04:55:14 GMT

Hi @Vyber90,

If you are parsing JSON by yourself, you should disable automatic KV extraction of Splunk.

You can disable it by adding KV_MODE = none into your sourcetype.

[your_sourcetype] KV_MODE = none

Re: Splunk duplicates events from a custom-made JSON event file

Vyber90 — Fri, 06 Aug 2021 19:58:54 GMT

I'm sorry to tell you that it doesn't work for me (I put that line in a new sourcetype in the props.conf file, where I think it should go).

As I said, I put that line in and it was today that I added information after the latest event of the JSON. As I said in the question, now instead of 7 events (6 + the new event), I have 13 (6 events + their 6 duplicate events + the new event).

By the way, I don't know if this is obvious or not, so I'll tell you this piece of information.

Duplicate events would look like this:

The duplicated events are carbon copies. They have the same information, the same timestamps, the same quantity of lines, and they do even filter out when using dedup.

Re: Splunk duplicates events from a custom-made JSON event file

jhanvidattani — Sun, 08 Aug 2021 13:16:43 GMT

@Vyber90

Can you add the below config in the inputs.conf file for your stanza?

followTail = 1

ReF: Splunk Ans

Check props and conf of followTail here: Inputsconf

If you find my solution/debugging steps fruitful, an upvote would be appreciated.

Re: Splunk duplicates events from a custom-made JSON event file

scelikok — Mon, 09 Aug 2021 06:59:51 GMT

Hi @Vyber90,

I thought you were seeing duplicate fields on Interesting Fields, but you see duplicate raw events. Could you please share your inputs.conf for that json file?

Re: Splunk duplicates events from a custom-made JSON event file

Vyber90 — Mon, 09 Aug 2021 07:24:11 GMT

Here you go:

In order to not change JSON's original stuff I created this new sourcetype. The lines I've manually added are TRUNCATE and followTail (the last one was added as a solution proposed in this very thread).