Hi all. Background is I have recently acquired a JSON feed via Kafka but the schema was developed with other uses in mind so it's not working particularly well for our logging requirements. For some reason the event type is being used as a key name. It annoys me greatly as there's no way to run a search for a specific event type. Its also playing havoc with extractions as the full path to the required value needs to be known, but can not be in a search as the event type can change. We can get the values if we know a specific event type (eg. Eventtype4.request.someKey), however it's very difficult to return the same value when the specific event type is unknown (eg. *.request.someKey) The second issue is all event types are present in the event but all but one has a null value. This can get very messy when there could be several hundred types of events.... I have tried stripping the first level and extracting the name of a key that has a non-null value but nothing works particularly all that well.  Any assistance will be appreciated. { Eventtype1: null Eventtype2: null Eventtype3: null Eventtype4: {    request: {               "someKey": "helloworld"               "anotherKey": "anothervalue"    }    response: {             "datachunk": "a few values here"    } } header: {               "timestamp": 123456789012               "someid": "323abcd" } } ... View more

I have been ripping my hair out for the last few nights trying to figure out a solution for this issue. I have a log being ingested by a UF that has some annoying characteristics. Looks a bit like this: ** Process Started ** 2021-06-01 14:40:21 INFO Application is loading something 2017-06-01 14:40:22 INFO And another thing 2017-06-01 14:40:23 WARN Something might have broken ** Process Finished ** ** Process Started ** 2021-06-02 20:15:50 INFO Application has done something interesting ** Process Finished ** Between the two messages are nice, timestamped, single line events. Those ones load up pretty well using defaults but the pesky non-timestamped application messages are causing all sorts of issues. I can't filter them out and it's preferable that events don't start with "** Process Started **". Best hack i have been able to come up with so far is: TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE = \d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2} HEADER_FIELD_LINE_NUMBER = 2 PREAMBLE_REGEX = Process\s+Started The first non-timestamped line is ignored and the rest are bundled into the end of an event. But there must be a better way. ... View more