topic Re: Multiline events not line breaking as expected in Getting Data In

Multiline events not line breaking as expected

lisheridan — Mon, 28 Sep 2020 10:02:49 GMT

I have some data that looks like:

TIMESTAMP: 2011-10-31 13:51:25
top - 13:51:25 up 6 days, 19:53, 5 users, load average: 21.00, 20.57, 19.83
Tasks: 130 total, 0 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.5% us, 0.7% sy, 0.0% ni, 96.4% id, 1.3% wa, 0.0% hi, 0.1% si
Mem: 32906264k total, 32847544k used, 58720k free, 346852k buffers
Swap: 33615352k total, 6804k used, 33608548k free, 7764416k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25772 admin 17 0 22.8g 22g 7040 S 19.9 72.1 10:17.11 rfsd
25780 admin 16 0 22.8g 22g 7040 S 19.9 72.1 10:18.01 rfsd
25777 admin 16 0 22.8g 22g 7040 S 17.9 72.1 10:18.10 rfsd
25459 admin 16 0 22.8g 22g 7040 S 11.9 72.1 8:40.27 rfsd
25493 admin 16 0 22.8g 22g 7040 S 6.0 72.1 2:03.05 rfsd

TIMESTAMP: 2011-10-31 13:52:25
top - 13:52:25 up 6 days, 19:53, 5 users, load average: 21.00, 20.57, 19.83
Tasks: 130 total, 0 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.5% us, 0.7% sy, 0.0% ni, 96.4% id, 1.3% wa, 0.0% hi, 0.1% si
Mem: 32906264k total, 32847544k used, 58720k free, 346852k buffers
Swap: 33615352k total, 6804k used, 33608548k free, 7764416k cached

I want to line break only before "TIMESTAMP". Here is my props.conf:

[source::/var/log/stats/rfsd_top*]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = TIMESTAMP
MAX_EVENTS = 400

I sometimes get an event with just the "TIMESTAMP.." line while other times I get the correct event intact. The event size is 132 lines. How can I get this to work?

Re: Multiline events not line breaking as expected

gkanapathy — Tue, 01 Nov 2011 00:27:54 GMT

[source::/var/log/stats/rfsd_top*]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)TIMESTAMP:
TRUNCATE = 40000
TIME_PREFIX = ^TIMESTAMP:
TIME_FORMAT = %Y-%m-%d %H:%M:%S

Re: Multiline events not line breaking as expected

_d_ — Tue, 01 Nov 2011 00:33:53 GMT

Give this stanza a try:

[source::/var/log/stats/rfsd_top*] TIME_PREFIX = ^TIMESTAMP:\s+ TIME_FORMAT= %Y-%m-%d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 20 LINE_BREAKER = ([\r\n]+)(?=^TIMESTAMP:\s+\d{4}\-\d{2}\-\d{2}) SHOULD_LINEMERGE = false

Hope this helps

> please upvote and accept answer if you find it useful - thanks!

Re: Multiline events not line breaking as expected

lisheridan — Mon, 28 Sep 2020 10:02:52 GMT

Unfortunately neither of those worked but thx for trying (even with SHOULD_LINEMERGE = true while using BREAK_ONLY_BEFORE).

Re: Multiline events not line breaking as expected

lguinn2 — Mon, 28 Sep 2020 10:02:54 GMT

Shouldn't it be

SHOULD_LINEMERGE = true

if you are going to use BREAK_ONLY_BEFORE?

Re: Multiline events not line breaking as expected

lguinn2 — Mon, 28 Sep 2020 10:02:57 GMT

To your original stanza, try adding

TRUNCATE = 40000

MAX_TIMESTAMP_LOOKAHEAD = 42

Re: Multiline events not line breaking as expected

kristian_kolb — Mon, 28 Sep 2020 10:02:59 GMT

You're right, gkanapathy probably made a small mistake. Replace BREAK_ONLY_BEFORE with LINE_BREAKER.

/kristian

Re: Multiline events not line breaking as expected

gkanapathy — Tue, 01 Nov 2011 15:04:22 GMT

yes, thank you. corrected above.