topic Re: Event Breaking events Zscaler in Getting Data In

Event Breaking events Zscaler

CarolinaHB — Wed, 03 Apr 2024 04:42:53 GMT

Hello,

I need to event break the following events, but they have a different date format. At the beginning, only at the end, it ends with the 'keyprotectiontype' field, which sometimes has 'NA'. Additionally, it must always have the 'reason' field at the beginning.

Apr 2 22:18:08 04-02 22: 17:39#011reason=Allowed#011event_id=7353490211603742721#011protocol=HTTP#011action=Allowed#011transactionsize=345241#011responsesize=344806#011requestsize=435#011urlcategory=Operating System and Software Updates#011serverip=92.123.121.156#011requestmethod=GET#011refererURL=None#011useragent=Microsoft BITS/7.8#011product=NSS#011location=Road Warrior#011ClientIP=12.2.11.10#011status=206#011user=lvtorrea@lula.com.es#011url=2.tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files/20c818db-67ad-44d4-8409-4d9dd7986af1?P1=1712128627&P2=404&P3=2&P4=OEkaO+U5XHKvf+lM41oEFDeIKRAD9S6SWgch3BSzA/yxusk1LA44YVdjNg94soDh+D8bYKjPHLpS4296pI6Tcw==#011vendor=Zscaler#011hostname=dkdkdk #011clientpublicIP=1.111.120.11#011threatcategory=None#011threatname=None#011filetype=None#011appname=General Browsing#011pagerisk=0#011threatseverity=None#011department=XXXXX (1422)#011urlsupercategory=Information Technology#011appclass=General Browsing#011dlpengine=None#011urlclass=Business Use#011threatclass=None#011dlpdictionaries=None#011fileclass=None#011bwthrottle=NO#011contenttype=application/octet_stream#011unscannabletype=None#011devicehostname=MAA#011deviceowner=lvtorrea#011keyprotectiontype= Software Protection#0122024-04-02 22:17:39#011reason=Allowed#011event_id=7353490211788947457#011protocol=SSL#011action=Allowed#011transactionsize=9568#011responsesize=4934#011requestsize=4634#011urlcategory=Microsoft_WVD_URL#011serverip=20.189.173.26#011requestmethod=NA#011refererURL=None#011useragent=Unknown#011product=NSS#011location=Road Warrior#011ClientIP=192.168.0.147#011status=NA#011user=jlvaldezo@lula.com.es#011url=us-v10c.events.data.microsoft.com#011vendor=Zscaler#011hostname=dkdkdk#011clientpublicIP=1.19.72.10#011threatcategory=None#011threatname=None#011filetype=None#011appname=General Browsing#011pagerisk=0#011threatseverity=None#011department=xxxxxxx MANAGEMENT#011urlsupercategory=User-defined#011appclass=General Browsing#011dlpengine=None#011urlclass=Bandwidth Loss#011threatclass=None#011dlpdictionaries=None#011fileclass=None#011bwthrottle=NO#011contenttype=Other#011unscannabletype=None#011devicehostname=KDKD#011deviceowner=jlvaldezo#011keyprotectiontype=N/A#012202

Can you help me?

Re: Event Breaking events Zscaler

scelikok — Wed, 03 Apr 2024 04:56:58 GMT

Hi @CarolinaHB,

I noticed that "#012" exists in your event as end of event marker.

You can use below as a line breaker;

LINE_BREAKER=#012()

Re: Event Breaking events Zscaler

PickleRick — Wed, 03 Apr 2024 08:44:30 GMT

#012 here is Line Feed character (\n) escaped by rsyslog (as well as #011 is an escaped \t).

Question is why it's escaped. It would be easiest if the events were broken by rsyslog.

Re: Event Breaking events Zscaler

saranvishva — Wed, 03 Apr 2024 09:45:49 GMT

It will work but extract with "#012 ".

Re: Event Breaking events Zscaler

ashurack_qmulos — Wed, 03 Apr 2024 13:02:48 GMT

I'm by no means an rsyslog guru but ran into it recently. There may be a better way to solve this but the quick fix was to turn off both supportOctetCountedFraming (input) and escapeControlCharacterTab (global).

$EscapeControlCharacterTab off [...other config...] input(type="imtcp" port="<port>" name="<name>" ruleset="<ruleset>" supportOctetCountedFraming="off")

Re: Event Breaking events Zscaler

PickleRick — Wed, 03 Apr 2024 13:27:41 GMT

I don't know what is the original rsyslog configuration (and even where that rsyslog is :-)).

But your option will only make the tab character (un)escaped.

The general option for escaping characters is parser.escapeControlCharactersOnReceive

Re: Event Breaking events Zscaler

ashurack_qmulos — Wed, 03 Apr 2024 14:31:46 GMT

Setting supportOctetCountedFraming="off" on the input fixes newlines being encoded to #012.

Re: Event Breaking events Zscaler

PickleRick — Wed, 03 Apr 2024 20:09:58 GMT

It does this effect but it works a bit differently. With octet counted option rsyslog split the input connection (because it works with tcp input only) based on the length of the event which should be given at the beginning of the event if I remember correctly. So the main problem is not that the new lines are encoded as #012 but that the events are not split at newline characters as they should be. If you turn of the octet counted option, the incoming tcp stream is broken into separate events on newline character so there is nothing to encode as #012 anymore.