How to skip reindexing events of manually edited JSON

[evelenke]

Hi Splunkers,

I need to test folder monitoring inputs with source in JSON format.
Since I'm not able to test it with an application right now, I'm adding JSON events to the file manually.. After this change all previous events are getting indexed again.
What configuration will help to avoid this behavior? I've tried

followTail = 1 

but this did not help.

[evelenke · Accepted solution]

So the problem was caused with square brackets [] in the beginning and the end of file.
Without these Splunk do not reindex all events included.

Thanks to everyone for assistance.

[evelenke]

I'll try to continue here as my troubles didn't finish yet.

So I need to remove those square brackets and let Splunk recognize this type like a usual JSON ({.....}). I've applied transforms like this:

[jsontest_null]
REGEX = ^([|)([^]$]+)(]|$)
FORMAT = $2
DEST_KEY = _raw

But even with this the events still get reindexed with adding new records to the file.
Is there any way to escape this?
My props.conf:

LINE_BREAKER = (,)({"action")
AUTO_KV_JSON = true
SHOULD_LINEMERGE = false

Sample fake events (always separated by comma):
[{"action":{"date":"2017-03-27T07:56:14","name":"open","host":"host1","user":{"id":"27","name":"Doe, Joe","class":"5"},"obj":{"file":"3","name":"3","version":"1","size":"2963","type":"1","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}},{"action":{"date":"2017-03-27T07:56:15","name":"open","host":"host2","user":{"id":"27","name":"Doe, Joe","class":"6"},"obj":{"file":"3","name":"3","version":"1","size":"2433","type":"2","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}}]

[mattymo]

ah nice! Glad to see you got it sorted!

[evelenke]

Thanks, but I think it's kind of bug, although it may be rare output type with '[]' but it happens...
Please also check my comment below , don't know if I need to create new topic for this.

[mattymo]

i don't believe its a bug, i thiink its more to do with how splunk reads a json array. If u are using json extractions, an array will be one event, so if you are hand changing an event, it will read all of it because it is a "new" event. at least thats what I think is happening

[evelenke]

Yes, I also have similar thoughts.
Looking for a workaround, the transformation I've provided below doesn't work, unfortunately.

[mattymo]

you just need to get the linebreaker right. regex101.com is my best friend for that.

Check out this answer to see how other users did a similar thing

https://answers.splunk.com/answers/511589/how-to-configure-line-breaking-for-my-sample-json.html#ans...

[evelenke]

Thank you, that was resolved. Always use regex101 but sometimes Splunk needs something else)

[woodcock]

It is the followTail = 1 that is doing it; delete this or set to 0 instead.

[evelenke]

The same, actually I've added followTail=1 with hope this will resolve it.

[mattymo]

Can you share the entire config for this input?

./splunk btool inputs list --debug

look for the stanza for this particular input and share here!

[evelenke]

sure:

[monitor://mypath*.json]
disabled = 0
host = myhost
sourcetype = my_json_test1
source = my_json_test1
index = my_json_test1

alwaysOpenFile = 1

followTail=1

[mattymo]

windows or nix?

[evelenke]

windows 10

[evelenke]

The same on Linux.

[somesoni2]

By default, Splunk does followTail only for monitor data inputs. Does the update that you make is within first 256 characters of the file?

[evelenke]

No, it's more - each event has approx. 400 chars