topic Re: Ingesting event data with timestamps afterwards sometimes in Getting Data In

Ingesting event data with timestamps afterwards sometimes

Thomas-R — Mon, 24 Jan 2022 06:10:18 GMT

Hey All,

I have data that needs to be ingested with multiple lines similar to the following:

************ Start Display Current Environment ************

***data***

************* End Display Current Environment *************

[13/11/21 5:21:15:183 AEDT] 00000001 ***data***
[13/11/21 5:21:15:276 AEDT] 00000001 ***data***
[13/11/21 5:21:15:278 AEDT] 00000001 ***data***

************ Start Display Current Environment ************

***data***

************* End Display Current Environment *************

[17/11/21 5:21:15:183 AEDT] 00000001 ***data***
[17/11/21 5:21:15:276 AEDT] 00000001 ***data***
[17/11/21 5:21:15:278 AEDT] 00000001 ***data***

Please note that the Start and End Display current Environment lines are constant in length and how they start but belong to the timestamp after themselves.

Is there a way to parse this data?

Re: Ingesting event data with timestamps afterwards sometimes

richgalloway — Mon, 24 Jan 2022 13:38:24 GMT

Try these props.conf settings:

[mysourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)\*{12} Start TIME_PREFIX = \[ TIME_FORMAT = %d/%m/%y %H:%M:%S:%$N %Z

Re: Ingesting event data with timestamps afterwards sometimes

Thomas-R — Mon, 24 Jan 2022 22:32:45 GMT

Hi @richgalloway,

I tried that but unfortunately it groups the events as per '************ Start' but sometimes the events start with the date instead, it there a way to sometimes get it to parse if this '************ Start' is first unless there is a date there instead?

Re: Ingesting event data with timestamps afterwards sometimes

richgalloway — Tue, 25 Jan 2022 00:42:05 GMT

Thanks for clarifying the requirements. Try this setting.

LINE_BREAKER = ([\r\n]+)(\*{12} Start|\[\d\d\/)