How to fix line breaking issue Unix timestamp

[samadmemon]

Hi All,

I have the logs in below format which is stored in an S3 bucket :

1567295878959445,hostname,ip,id,session,operation,db,query

The first field I believe is the Unix timestamp. When I am integrating those logs with Splunk Addon for AWS the line breaking is not happening as per the timestamp.
Below is the sample log that I am receiving in Splunk. The log is not breaking based on the timestamp as you can see below:

1567295878959445,hostname,ip,id,session,operation,db,query,1567295878959550,hostname,ip,id,session,operation,db,query'

Could anyone advise the configuration in props.conf to break these logs as per the timestamp? Ideally, log should look like below in Splunk :

1567295878959445,hostname,ip,id,session,operation,db,query -log1

1567295878959550,hostname,ip,id,session,operation,db,query' -log2

Regards,
Samad

[somesoni2]

Give this a try (props.conf on your heavy forwarder/indexer, first Splunk Enterprise instance that comes in your data flow)

[YourSourcetypeHere]
LINE_BREAKER = ([\r\n]+)(?=\d+,)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %s.%6N

Would require Splunk restart to take effect.

[samadmemon]

Thank you for the answer.

However, the above setting is not working.

[rupesh26]

Try this
TIME_FORMAT = %s%3N

[samadmemon]

It is now taking the timestamp without any errors, however the events are not breaking based on the timestamp. I used below setting :

LINE_BREAKER = (\d{10,16}+)

It is breaking the events but it is removing timestamp from the logs.

Regards,
Samad

[rupesh26]

doesn't this LINE_BREAKER work for you ?

LINE_BREAKER = ([\r\n]+)(?=\d+,)

[samadmemon]

Hi Rupesh,

No, this line breaker doesn't work.

[somesoni2]

So, all the logs are in single lines thus the my line breaker didn't work. Logs like this with no separator between the events are tough to break. Do you have control to change the logging format, either changing it to single line OR including some static/unique characters before the event?

[samadmemon]

I don't have control to change the logging format. However, I will check with admin to see if there is any possibility to change the logging format.

Thanks for your help.

[maciep]

what about something like this for your line breaker....assuming you don't need the comma

(,\s*)(?=\d{10})