Event Breaking events Zscaler

[CarolinaHB]

Hello, 

I need to event break the following events, but they have a different date format. At the beginning, only at the end, it ends with the 'keyprotectiontype' field, which sometimes has 'NA'. Additionally, it must always have the 'reason' field at the beginning.

Apr 2 22:18:08 04-02 22: 17:39#011reason=Allowed#011event_id=7353490211603742721#011protocol=HTTP#011action=Allowed#011transactionsize=345241#011responsesize=344806#011requestsize=435#011urlcategory=Operating System and Software Updates#011serverip=92.123.121.156#011requestmethod=GET#011refererURL=None#011useragent=Microsoft BITS/7.8#011product=NSS#011location=Road Warrior#011ClientIP=12.2.11.10#011status=206#011user=lvtorrea@lula.com.es#011url=2.tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/20c818db-67ad-44d4-8409-4d9dd7986af1?P1=1712128627&P2=404&P3=2&P4=OEkaO+U5XHKvf+lM41oEFDeIKRAD9S6SWgch3BSzA/yxusk1LA44YVdjNg94soDh+D8bYKjPHLpS4296pI6Tcw==#011vendor=Zscaler#011hostname=dkdkdk #011clientpublicIP=1.111.120.11#011threatcategory=None#011threatname=None#011filetype=None#011appname=General Browsing#011pagerisk=0#011threatseverity=None#011department=XXXXX (1422)#011urlsupercategory=Information Technology#011appclass=General Browsing#011dlpengine=None#011urlclass=Business Use#011threatclass=None#011dlpdictionaries=None#011fileclass=None#011bwthrottle=NO#011contenttype=application/octet_stream#011unscannabletype=None#011devicehostname=MAA#011deviceowner=lvtorrea#011keyprotectiontype= Software Protection#0122024-04-02 22:17:39#011reason=Allowed#011event_id=7353490211788947457#011protocol=SSL#011action=Allowed#011transactionsize=9568#011responsesize=4934#011requestsize=4634#011urlcategory=Microsoft_WVD_URL#011serverip=20.189.173.26#011requestmethod=NA#011refererURL=None#011useragent=Unknown#011product=NSS#011location=Road Warrior#011ClientIP=192.168.0.147#011status=NA#011user=jlvaldezo@lula.com.es#011url=us-v10c.events.data.microsoft.com#011vendor=Zscaler#011hostname=dkdkdk#011clientpublicIP=1.19.72.10#011threatcategory=None#011threatname=None#011filetype=None#011appname=General Browsing#011pagerisk=0#011threatseverity=None#011department=xxxxxxx MANAGEMENT#011urlsupercategory=User-defined#011appclass=General Browsing#011dlpengine=None#011urlclass=Bandwidth Loss#011threatclass=None#011dlpdictionaries=None#011fileclass=None#011bwthrottle=NO#011contenttype=Other#011unscannabletype=None#011devicehostname=KDKD#011deviceowner=jlvaldezo#011keyprotectiontype=N/A#012202

Can you help me?

[scelikok]

Hi @CarolinaHB,

I noticed that "#012" exists in your event as end of event marker.

You can use below as a line breaker;

LINE_BREAKER=#012()

[saranvishva]

It will work but extract with "#012 ".

[PickleRick]

#012 here is Line Feed character (\n) escaped by rsyslog (as well as #011 is an escaped \t).

Question is why it's escaped. It would be easiest if the events were broken by rsyslog.

[ashurack_qmulos · Accepted solution]

I'm by no means an rsyslog guru but ran into it recently.  There may be a better way to solve this but the quick fix was to turn off both supportOctetCountedFraming (input) and escapeControlCharacterTab (global).

$EscapeControlCharacterTab off

[...other config...]

input(type="imtcp" port="<port>" name="<name>" ruleset="<ruleset>" supportOctetCountedFraming="off")

[PickleRick]

I don't know what is the original rsyslog configuration (and even where that rsyslog is :-)).

But your option will only make the tab character (un)escaped.

The general option for escaping characters is parser.escapeControlCharactersOnReceive

[ashurack_qmulos]

Setting supportOctetCountedFraming="off" on the input fixes newlines being encoded to #012.

[PickleRick]

It does this effect but it works a bit differently. With octet counted option rsyslog split the input connection (because it works with tcp input only) based on the length of the event which should be given at the beginning of the event if I remember correctly. So the main problem is not that the new lines are encoded as #012 but that the events are not split at newline characters as they should be. If you turn of the octet counted option, the incoming tcp stream is broken into separate events on newline character so there is nothing to encode as #012 anymore.