Hi Splunkers,
I need to test folder monitoring inputs with source in JSON format.
Since I'm not able to test it with an application right now, I'm adding JSON events to the file manually.. After this change all previous events are getting indexed again.
What configuration will help to avoid this behavior? I've tried
followTail = 1
but this did not help.
So the problem was caused with square brackets []
in the beginning and the end of file.
Without these Splunk do not reindex all events included.
Thanks to everyone for assistance.
So the problem was caused with square brackets []
in the beginning and the end of file.
Without these Splunk do not reindex all events included.
Thanks to everyone for assistance.
I'll try to continue here as my troubles didn't finish yet.
So I need to remove those square brackets and let Splunk recognize this type like a usual JSON ({.....}). I've applied transforms like this:
[jsontest_null]
REGEX = ^([|)([^]$]+)(]|$)
FORMAT = $2
DEST_KEY = _raw
But even with this the events still get reindexed with adding new records to the file.
Is there any way to escape this?
My props.conf:
LINE_BREAKER = (,)({"action")
AUTO_KV_JSON = true
SHOULD_LINEMERGE = false
Sample fake events (always separated by comma):
[{"action":{"date":"2017-03-27T07:56:14","name":"open","host":"host1","user":{"id":"27","name":"Doe, Joe","class":"5"},"obj":{"file":"3","name":"3","version":"1","size":"2963","type":"1","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}},{"action":{"date":"2017-03-27T07:56:15","name":"open","host":"host2","user":{"id":"27","name":"Doe, Joe","class":"6"},"obj":{"file":"3","name":"3","version":"1","size":"2433","type":"2","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}}]
ah nice! Glad to see you got it sorted!
Thanks, but I think it's kind of bug, although it may be rare output type with '[]' but it happens...
Please also check my comment below , don't know if I need to create new topic for this.
i don't believe its a bug, i thiink its more to do with how splunk reads a json array. If u are using json extractions, an array will be one event, so if you are hand changing an event, it will read all of it because it is a "new" event. at least thats what I think is happening
Yes, I also have similar thoughts.
Looking for a workaround, the transformation I've provided below doesn't work, unfortunately.
you just need to get the linebreaker right. regex101.com is my best friend for that.
Check out this answer to see how other users did a similar thing
Thank you, that was resolved. Always use regex101 but sometimes Splunk needs something else)
It is the followTail = 1
that is doing it; delete this or set to 0
instead.
The same, actually I've added followTail=1 with hope this will resolve it.
Can you share the entire config for this input?
./splunk btool inputs list --debug
look for the stanza for this particular input and share here!
sure:
[monitor://mypath*.json]
disabled = 0
host = myhost
sourcetype = my_json_test1
source = my_json_test1
index = my_json_test1
followTail=1
windows or nix?
windows 10
The same on Linux.
By default, Splunk does followTail only for monitor data inputs. Does the update that you make is within first 256 characters of the file?
No, it's more - each event has approx. 400 chars