Help with line breaking and stripping extra lines from events

[jordanmedved]

Hello,

I have been having trouble onboarding some logs that have some extra data at the top and are not breaking into individual events.

I would like to remove the first 7 lines (I tried SEDCMD in props) and then break the following into individual events that start with "CEF:0". Any help would be appreciated.

Sample log that came in as 1 event.

accountId:1111111
configId:1111
checksum:fffffffffffffffffffffff
format:CEF
startTime:1591023419998
endTime:1591023786052
|==|
CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| fileId=763000040111111111 sourceServiceName=site.site.com siteid=41611111 suid=1111111 requestClientApplication=Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) deviceFacility=ams cs2=false cs2Label=Javascript Support cs3=false cs3Label=Support cs1=NA cs1Label=Cap Support cs4=bf0e3ba9-cad7-42e3-917d-ffffffffffff cs4Label=VID cs5=a069314a28fc3f38df1a7fd08797ff70400c236c3f43c214a588d2c6b92fada93f21b37a01969be556f0370e4534fbd14969aa1f882f5680157c6c2cf9ffffff cs5Label=clappsig dproc=Unclassified cs6=Bot cs6Label=clapp ccode=DE cs7=51.2993 cs7Label=latitude cs8=9.491 cs8Label=longitude Customer=company start=1591023257044 request=site.site.com/products/ requestMethod=GET qstr=offset=70&max\ cn1=200 app=HTTPS act=REQ_PASSED deviceExternalId=107081971111111111 sip=x.x.x.x spt=443 in=6214 xff=x.x.x.x cpt=28286 src=x.x.x.x ver=TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 end=1591023257493
CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| fileId=763000040111111111 sourceServiceName=site.site.com siteid=11111111 suid=1111111 requestClientApplication=Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) deviceFacility=ams cs2=false cs2Label=Javascript Support cs3=false cs3Label=Support cs1=NA cs1Label=Cap Support cs4=bf0e3ba9-cad7-42e3-917d-ffffffffffff cs4Label=VID cs5=a069314a28fc3f38df1a7fd08797ff70400c236c3f43c214a588d2c6b92fada93f21b37a01969be556f0370e453411111111111f882f5680157c6c2cf9ac15cc cs5Label=clappsig dproc=Unclassified cs6=Bot cs6Label=clapp ccode=DE cs7=51.2993 cs7Label=latitude cs8=9.491 cs8Label=longitude Customer=company start=1591023260718 request=site.site.com/ requestMethod=GET qstr=offset=70&max cn1=302 app=HTTPS act=REQ_PASSED deviceExternalId=107082561111111111 sip=x.x.x.x spt=443 in=368 xff=x.x.x.x cpt=28286 src=x.x.x.x ver=TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 end=1591023260838

[richgalloway]

It would help to know your current props.conf settings, but try these:

[mysourcetype]
SEDCMD = s/accountId:[\s\S]+\|==\|//
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)CEF:
TIME_PREFIX = end=
TIME_FORMAT = %s%3N