Filter portions of multi-line logs using a Heavy Forwarder

Raghav2384 — Mon, 28 Sep 2020 19:16:34 GMT

Hello Experts, I had posted the same question couple of days ago and had to re-post because of the formatting issues. We are consuming certain logs with DEBUG level but realized only a portion of those we need to index. We have decided to extract those required portions and discard rest of the stuff. These are logs averaging 50 lines per event. I just want to make sure we do it right the first time as the usage is pretty deep and do not want to interrupt. Here's a sample log

172.18.232.85 [2015-03-04 15:20:25,083] ===============================================
POLICY RESULT ERROR: Unable to authorize session - no authorization result found
Key1 = value1
Key2 = Value2
Key3 = Value3
Key4 = Value4
Key5 = Value5
Key6 = Value6
Key7 = Value7
Key8 = Value8
Key9 = Value9
DEBUG MSGS:
INFO : (core) Tagging message with ID: Important Stuff here
INFO : (radius) RADIUS device group assigned to: Phoenix
INFO : (radius) Loading session by the audit session id: XYZ
INFO : (core) Lock obtained on key: auditSessionId:ab1234bcngd
INFO : (core) Start session triggered
INFO : (radius) Radius usage reported 123456
INFO : (radius) Found generic location parameter ssid/DomainID
INFO : (radius) Found generic location parameter ap_mac/12-23-45-67-89
INFO : (location) Using generic ssid\XYZABC for location lookup.
INFO : (location) Using generic ap_mac\12-23-45-67-89 for location lookup.
INFO : (location) Location found for generic matching: ssid\DomainID
WARN : (auth) Failed USUM_AUTHORIZATION no password found for user
WARN : (core) Removing session since no authorization result found
WARN : (service) Stopping creation since the session has no services
INFO : (balance) Error found, rolling back transaction

ERROR : (core) Error processing policy request: Unable to authorize session - no authorization result found

we do not want to index entire Debug...I just need to grab the fields/Text in Bold and send rest to the null Queue. All of these are multiline logs and i am not able to get close in achieving this. Appreciate any pointers

Thanks,
Raghav

Re: Filter portions of multi-line logs using a Heavy Forwarder

Raghav2384 — Tue, 24 Mar 2015 15:41:15 GMT

Any ideas?

Re: Filter portions of multi-line logs using a Heavy Forwarder

somesoni2 — Tue, 24 Mar 2015 16:38:21 GMT

Is the format of the logs always same?? We need to identify patterns for which lines to retain and which lines to discard.

Re: Filter portions of multi-line logs using a Heavy Forwarder

Raghav2384 — Tue, 24 Mar 2015 16:59:43 GMT

Yes, standard is, the pieces i want from debug portion remains same

Re: Filter portions of multi-line logs using a Heavy Forwarder

woodcock — Tue, 11 Aug 2015 22:58:51 GMT

You can do it like this in your props.conf:

[MyBigFatSourcetype]
SEDCMD-slimfast = s/^INFO : (core) Start.*$// s/^INFO : (radius) Radius.*$// s/^INFO : (location).*$// s/^WARN :.*$// s/^INFO : (balance).*$//

This will definitely work but the downside is that, although the text will definitely be gone, I think it will leave blank line gaps in the raw events which may be distracting/confusing.

topic Re: Filter portions of multi-line logs using a Heavy Forwarder in Getting Data In

Filter portions of multi-line logs using a Heavy Forwarder

ERROR : (core) Error processing policy request: Unable to authorize session - no authorization result found

Re: Filter portions of multi-line logs using a Heavy Forwarder

Re: Filter portions of multi-line logs using a Heavy Forwarder

Re: Filter portions of multi-line logs using a Heavy Forwarder

Re: Filter portions of multi-line logs using a Heavy Forwarder