Getting Data In

Splunk duplicates events from a custom-made JSON event file

Vyber90
Explorer

What I'm doing is: I am doing stuff by my own an then parsing all the information as a JSON in order to append it to the end of an Splunk indexed file.

The problem is: Splunk, instead of just adding the information that wasn't there when the file was last updated, adds all the information again as if it was completely new, and thus giving me duplicate information.

I tried to append the information without altering what was already there, but this doesn't seem to solve anything. What would you do to only add the new info?

Labels (1)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @Vyber90,

I thought you were seeing duplicate fields on Interesting Fields, but you see duplicate raw events. Could you please share your inputs.conf for that json file?

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

Vyber90
Explorer

Here you go:

Vyber90_0-1628493736339.png

In order to not change JSON's original stuff I created this new sourcetype. The lines I've manually added are TRUNCATE and followTail (the last one was added as a solution proposed in this very thread).

0 Karma

jhanvidattani
Path Finder

@Vyber90 

Can you add the below config in the inputs.conf file for your stanza?

followTail = 1

 ReF: Splunk Ans 

Check props and conf of followTail here: Inputsconf 

 

If you find my solution/debugging steps fruitful, an upvote would be appreciated.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @Vyber90,

If you are parsing JSON by yourself, you should disable automatic KV extraction of Splunk.

You can disable it by adding KV_MODE = none into your sourcetype.

[your_sourcetype]
KV_MODE = none
If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

Vyber90
Explorer

I'm sorry to tell you that it doesn't work for me (I put that line in a new sourcetype in the props.conf file, where I think it should go).

As I said, I put that line in and it was today that I added information after the latest event of the JSON. As I said in the question, now instead of 7 events (6 + the new event), I have 13 (6 events + their 6 duplicate events + the new event).

By the way, I don't know if this is obvious or not, so I'll tell you this piece of information.

Duplicate events would look like this:

Vyber90_0-1628279664492.png

The duplicated events are carbon copies. They have the same information, the same timestamps, the same quantity of lines, and they do even filter out when using dedup.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...