Getting Data In

Unable to merge multiple lines for a non json log file into a single event.

rgoyal1010
New Member

I have a log whose sample format is similar to below. There are some cases where not every line starts with a time stamp. I want to break every time this occurs as a separate event. I tried below changes to the props.conf file but to no avail:

Logs file:

2016-09-07 18:41:23,556 [1589] DEBUG DBConnection - ExecuteReader SQL query time: 111 ms
2016-09-07 18:41:23,762 [1589] DEBUG AccountCacheModel - 2135
2016-09-07 18:41:23,762 [1589] DEBUG AccountCacheModel - A 17

2016-09-07 18:41:23,444 [1589] INFO OracleCommand - ExecuteReader - Oracle: select acm.acct_no , acm.acct_name, nvl(lk.lookup_desc,' ') as strategy, mgr.mgr_last_name, acm.special_acct_code, '' as ace_ok, 0 as is_funding, to_char(incept_date, 'DD/MM/YYYY')
FROM am_own.acm_vw acm
left join pimco_own.acct_mgr amgr on (acm.firm_no = amgr.firm_no AND acm.acct_no = amgr.acct_no AND amgr.mgr_type = 'P1')
left join pimco_own.mgr mgr on amgr.mgr_no = mgr.mgr_no
left join pimco_own.lookup lk on (lk.lookup_code = acm.strategy_ind and lk.lookup_tbl = 'strategy')
WHERE acm.firm_no = 1 AND acm.incept_date != to_date('01/01/1900','mm/dd/yyyy') AND acm.incept_date <= sysdate AND acm.term_date > sysdate
AND acm.special_acct_code NOT IN ('T') AND acm.active_status = 'Y'
UNION
select acm.acct_no , acm.acct_name, nvl(lk.lookup_desc,' ') as strategy, mgr.mgr_last_name, acm.special_acct_code, dot1.ace_ok, 1 as is_funding, to_char(dot.est_incep_date, 'DD/MM/YYYY')
from pimco_own.acm acm
left join pimco_own.acct_mgr amgr on (acm.firm_no = amgr.firm_no AND acm.acct_no = amgr.acct_no AND amgr.mgr_type = 'P1')
left join pimco_own.mgr mgr on amgr.mgr_no = mgr.mgr_no
left join pimco_own.lookup lk on (lk.lookup_code = acm.strategy_ind and lk.lookup_tbl = 'strategy')
left join pimco_own.dot_ims_transition dot on acm.acct_no = dot.acct_no
left join pimco_own.dot_pimco_transition dot1 on dot.acct_no = dot1.acct_no

Options used:

Case 1:
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d
SHOULD_LINEMERGE = true

Case 2:
LINE_BREAKER = ^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d

Case 3:
BREAK_ONLY_BEFORE = ^\d+-\d+-\d+\s+\d+:\d+:\d+,\d+\s+[\d+]
SHOULD_LINEMERGE = true

Case 4:
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true

0 Karma
1 Solution

somesoni2
Revered Legend

Try this

[yoursourcetype]
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2})
SHOULD_LINEMERGE = false
TIME_FORMAT = 2016-09-07 18:41:23,556
TIME_PREFIX = %Y-%m-%d %H:%M:%S,%N
MAX_TIMESTAMP_LOOKAHEAD = 23

View solution in original post

somesoni2
Revered Legend

Try this

[yoursourcetype]
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2})
SHOULD_LINEMERGE = false
TIME_FORMAT = 2016-09-07 18:41:23,556
TIME_PREFIX = %Y-%m-%d %H:%M:%S,%N
MAX_TIMESTAMP_LOOKAHEAD = 23

rgoyal1010
New Member

Perfect ! Worked like a charm 🙂 Awesome stuff

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...