Getting Data In

How to skip reindexing events of manually edited JSON

evelenke
Contributor

Hi Splunkers,

I need to test folder monitoring inputs with source in JSON format.
Since I'm not able to test it with an application right now, I'm adding JSON events to the file manually.. After this change all previous events are getting indexed again.
What configuration will help to avoid this behavior? I've tried

followTail = 1 

but this did not help.

0 Karma
1 Solution

evelenke
Contributor

So the problem was caused with square brackets [] in the beginning and the end of file.
Without these Splunk do not reindex all events included.

Thanks to everyone for assistance.

View solution in original post

evelenke
Contributor

So the problem was caused with square brackets [] in the beginning and the end of file.
Without these Splunk do not reindex all events included.

Thanks to everyone for assistance.

evelenke
Contributor

I'll try to continue here as my troubles didn't finish yet.

So I need to remove those square brackets and let Splunk recognize this type like a usual JSON ({.....}). I've applied transforms like this:

[jsontest_null]
REGEX = ^([|)([^]$]+)(]|$)
FORMAT = $2
DEST_KEY = _raw

But even with this the events still get reindexed with adding new records to the file.
Is there any way to escape this?
My props.conf:

LINE_BREAKER = (,)({"action")
AUTO_KV_JSON = true
SHOULD_LINEMERGE = false

Sample fake events (always separated by comma):
[{"action":{"date":"2017-03-27T07:56:14","name":"open","host":"host1","user":{"id":"27","name":"Doe, Joe","class":"5"},"obj":{"file":"3","name":"3","version":"1","size":"2963","type":"1","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}},{"action":{"date":"2017-03-27T07:56:15","name":"open","host":"host2","user":{"id":"27","name":"Doe, Joe","class":"6"},"obj":{"file":"3","name":"3","version":"1","size":"2433","type":"2","room":{"name":"room1","id":"room1"},"subject":"m5","fab":"a1","dir":"m5-a2"}}}]

0 Karma

mattymo
Splunk Employee
Splunk Employee

ah nice! Glad to see you got it sorted!

- MattyMo
0 Karma

evelenke
Contributor

Thanks, but I think it's kind of bug, although it may be rare output type with '[]' but it happens...
Please also check my comment below , don't know if I need to create new topic for this.

0 Karma

mattymo
Splunk Employee
Splunk Employee

i don't believe its a bug, i thiink its more to do with how splunk reads a json array. If u are using json extractions, an array will be one event, so if you are hand changing an event, it will read all of it because it is a "new" event. at least thats what I think is happening

- MattyMo
0 Karma

evelenke
Contributor

Yes, I also have similar thoughts.
Looking for a workaround, the transformation I've provided below doesn't work, unfortunately.

0 Karma

mattymo
Splunk Employee
Splunk Employee

you just need to get the linebreaker right. regex101.com is my best friend for that.

Check out this answer to see how other users did a similar thing

https://answers.splunk.com/answers/511589/how-to-configure-line-breaking-for-my-sample-json.html#ans...

- MattyMo
0 Karma

evelenke
Contributor

Thank you, that was resolved. Always use regex101 but sometimes Splunk needs something else)

0 Karma

woodcock
Esteemed Legend

It is the followTail = 1 that is doing it; delete this or set to 0 instead.

0 Karma

evelenke
Contributor

The same, actually I've added followTail=1 with hope this will resolve it.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Can you share the entire config for this input?

./splunk btool inputs list --debug

look for the stanza for this particular input and share here!

- MattyMo
0 Karma

evelenke
Contributor

sure:

[monitor://mypath*.json]
disabled = 0
host = myhost
sourcetype = my_json_test1
source = my_json_test1
index = my_json_test1

alwaysOpenFile = 1

followTail=1

0 Karma

mattymo
Splunk Employee
Splunk Employee

windows or nix?

- MattyMo
0 Karma

evelenke
Contributor

windows 10

0 Karma

evelenke
Contributor

The same on Linux.

0 Karma

somesoni2
Revered Legend

By default, Splunk does followTail only for monitor data inputs. Does the update that you make is within first 256 characters of the file?

0 Karma

evelenke
Contributor

No, it's more - each event has approx. 400 chars

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...