Hi,
I am trying to ingest long JSON files into my Splunk index, where a record could contain more than 10000 characters. To prevent long records from getting truncated, I added a "TRUNCATE=0" into ...
See more...
Hi,
I am trying to ingest long JSON files into my Splunk index, where a record could contain more than 10000 characters. To prevent long records from getting truncated, I added a "TRUNCATE=0" into my props.conf, and the entire record was ingested into the index. All events are forwarded and stored in the index, but I'm having problems with fields that appear towards the end of the JSON records.
I'm currently testing with 2 files:
File A has 382 records, of which 166 are long records.
File B has 252 records, of which all are long records.
All 634 events are returned with a simple search of the index, and I can see all fields in each event, regardless of how long the event is.
However, not all fields are extracted and directly searchable. For example, one of the fields is called "name", and it appears towards the end of each JSON record. On the "Interesting fields" pane, under "name", it shows only a count of 216 events from File A, and none of the remaining 166 + 252 long events in Files A and B. This is the same for other fields that appear towards the end of each JSON record, but fields towards the beginning of the record show all 634 events.
If I negate the 216 events, then these fields do not appear on the Fields pane at all.
Also, while I'm not able to directly search for "name=<name in File B>", I can still select the field from the event and "add to search", and all 252 events would be returned.
I'm not sure why these fields are not properly extracted even though they did not appear to be truncated. How can I extract them properly?
Thank you.