Getting Data In

How to fix line breaking issue Unix timestamp

Explorer

Hi All,

I have the logs in below format which is stored in an S3 bucket :

1567295878959445,hostname,ip,id,session,operation,db,query

The first field I believe is the Unix timestamp. When I am integrating those logs with Splunk Addon for AWS the line breaking is not happening as per the timestamp.
Below is the sample log that I am receiving in Splunk. The log is not breaking based on the timestamp as you can see below:

1567295878959445,hostname,ip,id,session,operation,db,query,1567295878959550,hostname,ip,id,session,operation,db,query'

Could anyone advise the configuration in props.conf to break these logs as per the timestamp? Ideally, log should look like below in Splunk :

1567295878959445,hostname,ip,id,session,operation,db,query -log1

1567295878959550,hostname,ip,id,session,operation,db,query' -log2

Regards,
Samad

0 Karma

Revered Legend

Give this a try (props.conf on your heavy forwarder/indexer, first Splunk Enterprise instance that comes in your data flow)

[YourSourcetypeHere]
LINE_BREAKER = ([\r\n]+)(?=\d+,)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %s.%6N

Would require Splunk restart to take effect.

Explorer

Thank you for the answer.

However, the above setting is not working.

0 Karma

Path Finder

Try this
TIME_FORMAT = %s%3N

0 Karma

Explorer

It is now taking the timestamp without any errors, however the events are not breaking based on the timestamp. I used below setting :

LINE_BREAKER = (\d{10,16}+)

It is breaking the events but it is removing timestamp from the logs.

Regards,
Samad

0 Karma

Path Finder

doesn't this LINE_BREAKER work for you ?

LINE_BREAKER = ([\r\n]+)(?=\d+,)

0 Karma

Explorer

Hi Rupesh,

No, this line breaker doesn't work.

0 Karma

Revered Legend

So, all the logs are in single lines thus the my line breaker didn't work. Logs like this with no separator between the events are tough to break. Do you have control to change the logging format, either changing it to single line OR including some static/unique characters before the event?

0 Karma

Explorer

I don't have control to change the logging format. However, I will check with admin to see if there is any possibility to change the logging format.

Thanks for your help.

0 Karma

Champion

what about something like this for your line breaker....assuming you don't need the comma

(,\s*)(?=\d{10})
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!