topic How to skip reindexing events of manually edited JSON in Getting Data In

How to skip reindexing events of manually edited JSON

evelenke — Mon, 10 Apr 2017 21:17:49 GMT

Hi Splunkers,

I need to test folder monitoring inputs with source in JSON format.
Since I'm not able to test it with an application right now, I'm adding JSON events to the file manually.. After this change all previous events are getting indexed again.
What configuration will help to avoid this behavior? I've tried

followTail = 1

but this did not help.

Re: How to skip reindexing events of manually edited JSON

somesoni2 — Mon, 10 Apr 2017 21:36:51 GMT

By default, Splunk does followTail only for monitor data inputs. Does the update that you make is within first 256 characters of the file?

Re: How to skip reindexing events of manually edited JSON

evelenke — Mon, 10 Apr 2017 21:59:29 GMT

No, it's more - each event has approx. 400 chars

Re: How to skip reindexing events of manually edited JSON

mattymo — Mon, 10 Apr 2017 23:06:00 GMT

Can you share the entire config for this input?

./splunk btool inputs list --debug

look for the stanza for this particular input and share here!

Re: How to skip reindexing events of manually edited JSON

evelenke — Tue, 29 Sep 2020 13:37:57 GMT

sure:

[monitor://mypath*.json]
disabled = 0
host = myhost
sourcetype = my_json_test1
source = my_json_test1
index = my_json_test1

alwaysOpenFile = 1

followTail=1

Re: How to skip reindexing events of manually edited JSON

woodcock — Mon, 10 Apr 2017 23:11:29 GMT

It is the followTail = 1 that is doing it; delete this or set to 0 instead.

Re: How to skip reindexing events of manually edited JSON

evelenke — Mon, 10 Apr 2017 23:38:28 GMT

The same, actually I've added followTail=1 with hope this will resolve it.

Re: How to skip reindexing events of manually edited JSON

mattymo — Mon, 10 Apr 2017 23:44:50 GMT

windows or nix?

Re: How to skip reindexing events of manually edited JSON

evelenke — Mon, 10 Apr 2017 23:53:51 GMT

windows 10

Re: How to skip reindexing events of manually edited JSON

evelenke — Tue, 11 Apr 2017 12:27:42 GMT

The same on Linux.

Re: How to skip reindexing events of manually edited JSON

evelenke — Tue, 11 Apr 2017 14:45:43 GMT

So the problem was caused with square brackets [] in the beginning and the end of file.
Without these Splunk do not reindex all events included.

Thanks to everyone for assistance.

Re: How to skip reindexing events of manually edited JSON

mattymo — Tue, 11 Apr 2017 20:01:27 GMT

ah nice! Glad to see you got it sorted!

Re: How to skip reindexing events of manually edited JSON

evelenke — Tue, 29 Sep 2020 13:38:39 GMT

I'll try to continue here as my troubles didn't finish yet.

So I need to remove those square brackets and let Splunk recognize this type like a usual JSON ({.....}). I've applied transforms like this:

[jsontest_null]
REGEX = ^([|)([^]$]+)(]|$)
FORMAT = $2
DEST_KEY = _raw

But even with this the events still get reindexed with adding new records to the file.
Is there any way to escape this?
My props.conf:

LINE_BREAKER = (,)({"action")
AUTO_KV_JSON = true
SHOULD_LINEMERGE = false

Sample fake events (always separated by comma):
[{"action":{"date":"2017-03-27T07:56:14","name":"open","host":"host1","user":{"id":"27","name":"Doe, Joe","class":"5"},"obj":{"file":"3","name":"3","version":"1","size":"2963","type":"1","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}},{"action":{"date":"2017-03-27T07:56:15","name":"open","host":"host2","user":{"id":"27","name":"Doe, Joe","class":"6"},"obj":{"file":"3","name":"3","version":"1","size":"2433","type":"2","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}}]

Re: How to skip reindexing events of manually edited JSON

evelenke — Tue, 11 Apr 2017 23:14:11 GMT

Thanks, but I think it's kind of bug, although it may be rare output type with '[]' but it happens...
Please also check my comment below , don't know if I need to create new topic for this.

Re: How to skip reindexing events of manually edited JSON

mattymo — Wed, 12 Apr 2017 10:38:33 GMT

i don't believe its a bug, i thiink its more to do with how splunk reads a json array. If u are using json extractions, an array will be one event, so if you are hand changing an event, it will read all of it because it is a "new" event. at least thats what I think is happening

Re: How to skip reindexing events of manually edited JSON

evelenke — Wed, 12 Apr 2017 11:58:10 GMT

Yes, I also have similar thoughts.
Looking for a workaround, the transformation I've provided below doesn't work, unfortunately.

Re: How to skip reindexing events of manually edited JSON

mattymo — Wed, 12 Apr 2017 13:30:03 GMT

you just need to get the linebreaker right. regex101.com is my best friend for that.

Check out this answer to see how other users did a similar thing

https://answers.splunk.com/answers/511589/how-to-configure-line-breaking-for-my-sample-json.html#answer-511700

Re: How to skip reindexing events of manually edited JSON

evelenke — Sat, 22 Apr 2017 13:10:42 GMT

Thank you, that was resolved. Always use regex101 but sometimes Splunk needs something else)