Getting Data In
Highlighted

breaking multiline forefront events

Path Finder

Hello,

I'm trying to break logs collected from Microsoft Forefront Client Security into separate events. Here is a sample of the logs:

--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 08:57:48
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
************************************************************
Microsoft Forefront Client Security Log, (c) 2006
Stopped On Fri Feb 10 2012 09:20:43 (Exit Code = 0x0)
************************************************************
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 09:20:45
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
************************************************************

Begin Resource Scan
Scan ID:{4D865233-FA4A-42C5-87A3-962CC4E6B95A}
Scan Source:4
Start Time:Mon May 09 2011 15:10:29
End Time:Mon May 09 2011 15:10:30
Explicit resource to scan
Resource Schema:taskscheduler
Resource Path:@S-1-5-21-3171008878-1269407668-386151811-1006\C:\WINDOWS\tasks\MP Scheduled Quick Scan.job
Result Count:1
Known File
Number of Resources:1
Resource Schema:file
Resource Path:c:\Program Files\Microsoft Forefront\Client Security\Client\Antimalware\MpCmdRun.exe
Extended Info:481036337152
End Scan
************************************************************

I'd like for the data in between the asterisks to be one event.

I've tried several versions of the props.conf stanza with nothing working all of the time. What is the best and most efficient way to do this. Thanks.

0 Karma
Highlighted

Re: breaking multiline forefront events

Splunk Employee
Splunk Employee

Something like this:

inputs.conf

[monitor:///Users/dmaislin/Desktop/forefront]
disabled = false
followTail = 0
index = forefront
sourcetype = forefront

props.conf

 [forefront]
    BREAK_ONLY_BEFORE = ---
    NO_BINARY_CHECK = 1
    SHOULD_LINEMERGE = true
    pulldown_type = 1
    EXTRACT-productVersion = ^Product\sVersion:\s(?P<productVersion>.+?)$
    EXTRACT-engineVersion = ^Engine\sVersion:\s(?P<engineVersion>.+?)$
    EXTRACT-ASSignature = ^AS\sSignature\sVersion:\s(?P<ASSignatureVersion>.+?)$
    EXTRACT-AVSignature = ^AV\sSignature\sVersion:\s(?P<AVSignatureVersion>.+?)$
    EXTRACT-ScanID = ^Scan\sID:(?P<scanID>.+?)$
    EXTRACT-ScanSource = ^Scan\sSource:(?P<scanSource>.+?)$
    EXTRACT-StartTime = ^Start\sTime:(?P<startTime>.+?)$
    EXTRACT-EndTime = ^End\sTime:(?P<endTime>.+?)$
    EXTRACT-ResourceSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
    EXTRACT-ResourcePath = ^Resource\sPath:(?P<resourcePath>.+?)$
    EXTRACT-ResultCount = ^Result\sCount:(?P<resultCount>.+?)$
    EXTRACT-NumberOfResources = ^Number\sof\sResources:(?P<numberOfResources>.+?)$
    EXTRACT-Resource\sSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
    EXTRACT-Resource\sPath = ^Resource\sPath:(?P<resourcePath>.+?)$
    EXTRACT-Extended\sInfo = ^Extended\sInfo:(?P<extendedInfo>.+?)$
0 Karma
Highlighted

Re: breaking multiline forefront events

Path Finder

Thanks dmaislin, but that didn't seem to do what I'm wanting.

Each line in between the "***************" lines should be one event. Here is the props.conf stanza that I'm using right now that isn't working

[ms_forefront]
NO_BINARY_CHECK = true
SEDCMD-stripnull = s/\\x00//g
LINE_BREAKER = ([\r\n]+)(\*{60})
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER = (\*{60})
BREAK_ONLY_BEFORE_DATE = false

I have to wonder if the SEDCMD has something to do with it. Without the sedcmd, I get a "\x00" inserted in between each character of my log.

0 Karma
Highlighted

Re: breaking multiline forefront events

Splunk Employee
Splunk Employee

Try this:

[forefront]
BREAK_ONLY_BEFORE = ^\*.+?$
BREAK_ONLY_BEFORE_DATE = FALSE
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
pulldown_type = 1
EXTRACT-productVersion = ^Product\sVersion:\s(?P<productVersion>.+?)$
EXTRACT-engineVersion = ^Engine\sVersion:\s(?P<engineVersion>.+?)$
EXTRACT-ASSignature = ^AS\sSignature\sVersion:\s(?P<ASSignatureVersion>.+?)$
EXTRACT-AVSignature = ^AV\sSignature\sVersion:\s(?P<AVSignatureVersion>.+?)$
EXTRACT-ScanID = ^Scan\sID:(?P<scanID>.+?)$
EXTRACT-ScanSource = ^Scan\sSource:(?P<scanSource>.+?)$
EXTRACT-StartTime = ^Start\sTime:(?P<startTime>.+?)$
EXTRACT-EndTime = ^End\sTime:(?P<endTime>.+?)$
EXTRACT-ResourceSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
EXTRACT-ResourcePath = ^Resource\sPath:(?P<resourcePath>.+?)$
EXTRACT-ResultCount = ^Result\sCount:(?P<resultCount>.+?)$
EXTRACT-NumberOfResources = ^Number\sof\sResources:(?P<numberOfResources>.+?)$
EXTRACT-Resource\sSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
EXTRACT-Resource\sPath = ^Resource\sPath:(?P<resourcePath>.+?)$
EXTRACT-Extended\sInfo = ^Extended\sInfo:(?P<extendedInfo>.+?)$
0 Karma
Highlighted

Re: breaking multiline forefront events

Path Finder

This is the event in Splunk:
Microsoft Forefront Client Security Log, (c) 2006


Microsoft Forefront Client Security Log, (c) 2006

Started On Fri Feb 10 2012 14:15:44


Product Version: 1.5.1941.0

Engine Version: 1.1.8001.0

AS Signature Version: 1.119.1709.0

AV Signature Version: 1.119.1709.0


The second and third row of -----'s are actualy asterisks, not sure why the comment changed them.

0 Karma
Highlighted

Re: breaking multiline forefront events

Splunk Employee
Splunk Employee

Mine are line breaking correctly. Are you sure you deleted them and reindexed? Did you add a crcSalt=<SOURCE> to inputs.conf?

1   »  2/10/12
9:29:47.000 AM  
************************************************************
2   »  2/10/12
9:29:47.000 AM  
************************************************************
Begin Resource Scan
Scan ID:{4D865233-FA4A-42C5-87A3-962CC4E6B95A}
Scan Source:4
Start Time:Mon May 09 2011 15:10:29
End Time:Mon May 09 2011 15:10:30
Explicit resource to scan
Resource Schema:taskscheduler
Resource Path:@S-1-5-21-3171008878-1269407668-386151811-1006\C:\WINDOWS\tasks\MP Scheduled Quick Scan.job
Result Count:1
Show all 16 lines
3   »  2/10/12
9:29:47.000 AM  
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
4   »  2/10/12
9:29:47.000 AM  
************************************************************
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 09:20:45
5   »  2/10/12
9:29:47.000 AM  
************************************************************
Microsoft Forefront Client Security Log, (c) 2006
Stopped On Fri Feb 10 2012 09:20:43 (Exit Code = 0x0)
6   »  2/10/12
9:29:47.000 AM  
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
7   »  2/10/12
9:29:47.000 AM  
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 08:57:48
0 Karma
Highlighted

Re: breaking multiline forefront events

Path Finder

I am not deleting them and reindexing, I'm making the changes, restarting splunk, and then triggering more events to be written to my forefront logs.

Yes, I do have crcSalt = in my input.

Would the SEDCMD that I have to put in my props.conf affect anything?

0 Karma
Highlighted

Re: breaking multiline forefront events

Splunk Employee
Splunk Employee

Not sure, take it out and try since that is an at index time command.

0 Karma
Highlighted

Re: breaking multiline forefront events

Path Finder

It does have something to do with it. I created a second log file and copied/pasted log data into it and it broke the lines fine.

If I remove the sedcmd then i get an \x00 in between each character of my log. I guess I need to look and see exactly what the sedcmd is doing.

0 Karma
Highlighted

Re: breaking multiline forefront events

Splunk Employee
Splunk Employee

How are you getting these logs to Splunk? Is it with a forwarder or with FTP?

0 Karma