Parsing Forcepoint CASB CEF logs in Splunk

[geoffmoraes]

I need some help with parsing Forcepoint CASB CEF logs in Splunk. The data does not seem to parse the epoch time stamps and all comes in as one event. I need to break these up into individual events and also parse the epoch time stamp in the format  "%Y-%m-%d %H:%M:%S" on ingestion into splunk. 

SAMPLE DATA

CEF:0|Forcepoint CASB|Cloud Service Monitoring|1.0|63575192763|Activity|0|act=Monitor app= cat=Normal Activity cs1= destinationServiceName=Office365 deviceExternalId= deviceFacility=true deviceProcessName=loadBalancers dhost= dpriv=User dst=0.0.0.0 duser=14dd43c6-a792-4c07-a33b-c5e561a129de dvc=10.1.2.12 dvchost=somedomain end=1600408516000 externalId=0 fsize=-1 msg=//United States/Unknown outcome=Success proto= reason=modify request= requestClientApplication=Unknown/Unknown/"" rt=1600408516000 sourceServiceName=Unmanaged src=someIP start=1600408516000 suser=  cs2= cs3= cs5=false cs6= dproc=Unknown flexString1=mc_rg-int-qa-eus02_aks-int01-qa-eus02_eastus2,kubernetes-internal cs4=14dd43c6-a792-4c07-a33b-c5e561a129de flexString2= AD.ThreatRadarCategory= AD.TORNetworks= AD.MaliciousIPs= AD.AnonymousProxies= AD.IPChain=someIP AD.IPOrigin=External AD.samAccountName=14dd43c6-a792-4c07-a33b-c5e561a129de
CEF:0|Forcepoint CASB|Cloud Service Monitoring|1.0|63575216734|Activity|0|act=Monitor app= cat=Normal Activity cs1= destinationServiceName=Office365 deviceExternalId= deviceFacility=false deviceProcessName= dhost= dpriv=User dst=0.0.0.0 duser=someuser_849fcb2777e9@somedomain.onmicrosoft.com dvc=10.1.2.12 dvchost=somedomain end=1600406172000 externalId=0 fsize=-1 msg=//United States/Unknown outcome=Success proto= reason=login request= requestClientApplication=Unknown/Unknown/"" rt=1600406172000 sourceServiceName=Unmanaged src=someIP start=1600406172000 suser=  cs2= cs3= cs5=false cs6= dproc=Unknown flexString1= cs4=someaccount9@somedomain.onmicrosoft.com flexString2= AD.ThreatRadarCategory= AD.TORNetworks= AD.MaliciousIPs= AD.AnonymousProxies= AD.IPChain=someIP AD.IPOrigin=Internal AD.samAccountName=someaccount9@somedomain.onmicrosoft.com
CEF:0|Forcepoint CASB|Cloud Service Monitoring|1.0|63575216736|Activity|0|act=Monitor app= cat=Normal Activity cs1= destinationServiceName=Office365 deviceExternalId= deviceFacility=true deviceProcessName= dhost= dpriv=User dst=0.0.0.0 duser=someaccount@somedomain.onmicrosoft.com dvc=10.1.2.12 dvchost=somedomain end=1600405713000 externalId=0 fsize=-1 msg=//United States/Unknown outcome=Success proto= reason=login request= requestClientApplication=Unknown/Unknown/"" rt=1600405713000 sourceServiceName=Unmanaged src=someIP start=1600405713000 suser=  cs2= cs3= cs5=false cs6= dproc=Unknown flexString1= cs4=someaccount@somedomain.onmicrosoft.com flexString2= AD.ThreatRadarCategory= AD.TORNetworks= AD.MaliciousIPs= AD.AnonymousProxies= AD.IPChain=someIP AD.IPOrigin=External AD.samAccountName=someaccount@somedomain.onmicrosoft.com
CEF:0|Forcepoint CASB|Cloud Service Monitoring|1.0|63575216738|Activity|0|act=Monitor app= cat=Normal Activity cs1= destinationServiceName=Office365 deviceExternalId= deviceFacility=false deviceProcessName= dhost= dpriv=User dst=0.0.0.0 duser=someaccount@somedomain.com dvc=10.1.2.12 dvchost=somedomain end=1600405674000 externalId=0 fsize=-1 msg=/1225/United States/Unknown outcome=Success proto= reason=login request= requestClientApplication=Unknown/Unknown/"" rt=1600405674000 sourceServiceName=Unmanaged src=someIP start=1600405674000 suser= cs2= cs3= cs5=false cs6= dproc=Unknown flexString1= cs4=someaccount flexString2= AD.ThreatRadarCategory= AD.TORNetworks= AD.MaliciousIPs= AD.AnonymousProxies= AD.IPChain=someIP AD.IPOrigin=Internal AD.samAccountName=someaccount
CEF:0|Forcepoint CASB|Cloud Service Monitoring|1.0|63575216735|Activity|0|act=Monitor app= cat=Normal Activity cs1= destinationServiceName=Office365 deviceExternalId= deviceFacility=false deviceProcessName= dhost= dpriv=User dst=0.0.0.0 duser=someaccount9@somedomain.onmicrosoft.com dvc=10.1.2.12 dvchost=somedomain end=1600406165000 externalId=0 fsize=-1 msg=//United States/Unknown outcome=Success proto= reason=login request= requestClientApplication=Unknown/Unknown/"" rt=1600406165000 sourceServiceName=Unmanaged src=someIP start=1600406165000 suser=  cs2= cs3= cs5=false cs6= dproc=Unknown flexString1= cs4=someaccount@somedomain.onmicrosoft.com flexString2= AD.ThreatRadarCategory= AD.TORNetworks= AD.MaliciousIPs= AD.AnonymousProxies= AD.IPChain=someIP AD.IPOrigin=Internal AD.samAccountName=someaccount@somedomain.onmicrosoft.com

Assume that the sourcetype is called "cefevents" for this example.

 PROPS.CONF

[cefevents]
LINE_BREAKER = CEF:0.+[\r\n]?
MAX_TIMESTAMP_LOOKAHEAD = 600
NO_BINARY_CHECK = true
KV_MODE = none
SHOULD_LINEMERGE = false
TIME_FORMAT = %s%Q
TIME_PREFIX = \s(start|end|rt)\=

The regex looks ok on regex101 https://regex101.com/r/hf7ZJs/1 but doesn't work on this data.

I have also attempted using the props.conf from https://splunkbase.splunk.com/app/487/ but that does not help either.

[geoffmoraes · Accepted solution]

This has been resolved.

I had the props.conf under 

$SPLUNKHOME\etc\deployment-apps\<app_name>\local\

I removed that file and instead added its contents to the existing props.conf under 

 $SPLUNKHOME\etc\apps\search\local\

and the line-breaking and time-stamp parsing work as expected.

[thambisetty]

Did you set sourcetype? you should should use sourcetype that you have mentioned in inputs.conf in heavy forwarder props.conf.

[geoffmoraes]

@thambisetty Yes, the sourcetype in props.conf matches what is set in inputs.conf for this custom Forcepoint CASB app.

[thambisetty]

TIME_PREFIX - you can't set multiple prefixes. set any of them. I have used end=( you can change it)

TIME_FORMAT - you can format the way you want at the time of search. we can't enforce to show time format for each source type. TIME_FORMAT will tell splunk in which format time is there in event. so that Spunk understands time correctly.

[ __auto__learned__ ]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
TIME_PREFIX=end\=

[geoffmoraes]

@thambisettyIf I take a sample cef file and upload it to splunk, these values work as expected.

However, if I create a custom deployment app on a Heavy Forwarder with these values in props.conf, it has no effect on the incoming logs. The logs have no line breaks and the time stamp is that of the log file drop on the server with the UF.