Unable to merge multiple lines for a non json log file into a single event.

[rgoyal1010]

I have a log whose sample format is similar to below. There are some cases where not every line starts with a time stamp. I want to break every time this occurs as a separate event. I tried below changes to the props.conf file but to no avail:

Logs file:

2016-09-07 18:41:23,556 [1589] DEBUG DBConnection - ExecuteReader SQL query time: 111 ms
2016-09-07 18:41:23,762 [1589] DEBUG AccountCacheModel - 2135
2016-09-07 18:41:23,762 [1589] DEBUG AccountCacheModel - A 17

2016-09-07 18:41:23,444 [1589] INFO OracleCommand - ExecuteReader - Oracle: select acm.acct_no , acm.acct_name, nvl(lk.lookup_desc,' ') as strategy, mgr.mgr_last_name, acm.special_acct_code, '' as ace_ok, 0 as is_funding, to_char(incept_date, 'DD/MM/YYYY')
FROM am_own.acm_vw acm
left join pimco_own.acct_mgr amgr on (acm.firm_no = amgr.firm_no AND acm.acct_no = amgr.acct_no AND amgr.mgr_type = 'P1')
left join pimco_own.mgr mgr on amgr.mgr_no = mgr.mgr_no
left join pimco_own.lookup lk on (lk.lookup_code = acm.strategy_ind and lk.lookup_tbl = 'strategy')
WHERE acm.firm_no = 1 AND acm.incept_date != to_date('01/01/1900','mm/dd/yyyy') AND acm.incept_date <= sysdate AND acm.term_date > sysdate
AND acm.special_acct_code NOT IN ('T') AND acm.active_status = 'Y'
UNION
select acm.acct_no , acm.acct_name, nvl(lk.lookup_desc,' ') as strategy, mgr.mgr_last_name, acm.special_acct_code, dot1.ace_ok, 1 as is_funding, to_char(dot.est_incep_date, 'DD/MM/YYYY')
from pimco_own.acm acm
left join pimco_own.acct_mgr amgr on (acm.firm_no = amgr.firm_no AND acm.acct_no = amgr.acct_no AND amgr.mgr_type = 'P1')
left join pimco_own.mgr mgr on amgr.mgr_no = mgr.mgr_no
left join pimco_own.lookup lk on (lk.lookup_code = acm.strategy_ind and lk.lookup_tbl = 'strategy')
left join pimco_own.dot_ims_transition dot on acm.acct_no = dot.acct_no
left join pimco_own.dot_pimco_transition dot1 on dot.acct_no = dot1.acct_no

Options used:

Case 1:
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d
SHOULD_LINEMERGE = true

Case 2:
LINE_BREAKER = ^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d

Case 3:
BREAK_ONLY_BEFORE = ^\d+-\d+-\d+\s+\d+:\d+:\d+,\d+\s+[\d+]
SHOULD_LINEMERGE = true

Case 4:
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true

[somesoni2 · Accepted solution]

Try this

[yoursourcetype]
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2})
SHOULD_LINEMERGE = false
TIME_FORMAT = 2016-09-07 18:41:23,556
TIME_PREFIX = %Y-%m-%d %H:%M:%S,%N
MAX_TIMESTAMP_LOOKAHEAD = 23

[rgoyal1010]

Perfect ! Worked like a charm 🙂 Awesome stuff