topic Re: Do I have a possible KV extraction issue on the universal forwarder? in Getting Data In

Do I have a possible KV extraction issue on the universal forwarder?

ehowardl3 — Mon, 27 Aug 2018 20:21:21 GMT

I have some json events that are fairly long (10K-20K characters). Most events come through fine, except for the fact that some events have an issue with some of the fields towards the end of the event not being automatically extracted. I ran a couple of searches and found that the maximum raw length of the events with no issues is 10,197, while the minimum raw length of the events with the extraction issues is 10,251.

This led me to believe that I needed to bump up maxchars in the kv stanza in limits.conf, since it has a default limit of 10,240. However, upon making the change, the extraction issues remain. Here are my configurations:

On the forwarder:

props.conf:

[test:json]
KV_MODE = json
TIMESTAMP_FIELDS = Date
TRUNCATE = 0
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false

On the search head:

limits.conf:

[kv]
limit = 200
maxchars = 25000
maxcols = 1024

Any ideas?

Thanks!
E

Re: Do I have a possible KV extraction issue on the universal forwarder?

harsmarvania57 — Tue, 28 Aug 2018 09:18:49 GMT

HI @ehowardl3 ,

I am assuming that forwarder means Universal forwarder, in that case can you please apply below configuration on Universal Forwarder ?

props.conf

[test:json]
INDEXED_EXTRACTIONS = json

Re: Do I have a possible KV extraction issue on the universal forwarder?

ehowardl3 — Tue, 28 Aug 2018 12:11:37 GMT

Thank you! This worked. I had tried that previously, however I had:

[test:json]
INDEXED_EXTRACTIONS = JSON

I'm assuming the uppercase JSON was causing an issue.

Re: Do I have a possible KV extraction issue on the universal forwarder?

harsmarvania57 — Tue, 28 Aug 2018 12:29:08 GMT

Great, it worked. I have converted my comment to answer so that you can accept/upvote it.

Thanks,
Harshil