Hi everyone,
My client has indexes with events which are sometimes really large. The problem is that field extraction in such cases doesn't work properly. For example, opening an event shows the whole raw event, but fields below it are trimmed. If the field was a few thousands characters long, in the fields view below the event only about a thousand first characters are shown. Moreover, efforts to manipulate such fields produce unspecified results, e.g.,
| eval len_x = len(field_x)
returns 71, although the field is several thousands characters long. Searches targeting such events sometimes fail, e.g., specifying an event with an ID: event_uid=unique_id (a field-value combination present in the event) doesn't return anything, although a less specific search with the same time frame returns that event. We also tried to tackle the problem at the source, i.e., to shorten the field with excessive length before indexing
| eval field_x = if(len(field_x) > 1000, substr(field_x, 1, 1000) . "(oversized field trimmed)", field_x)
but this only trimmed the fields, without adding the text in the brackets.
So, since I haven't managed to find it in the documentation, I would like to ask the following: is there a limit for the field length and does it depend on the overall event size? How to deal with such long fields? Thanks and kind regards,
Krunoslav Ivesic
... View more