Techniques for limiting size of windows event logs

[mrflibbleuk]

Hi,

We currently have a number of servers that have the universal forwarder installed and set to forward to an Enterprise Splunk instance. Unfortunately the security logs themselves are generating a massive amount of data. We want to limit the data sent to Splunk, my understanding is we can use transforms etc to achieve this BUT this needs a heavy forwarder installed (i.e. a full Splunk installation). I am not sure of the license implications for this.

Does anyone here have any thoughts on ways around this one? I did think of maybe filtering the events in windows then exporting to a file and then picking it up but its a bit clunky.
Maybe use another Windows Forwarder style app and then forward the event logs as syslog entries to Splunk.

Would appreciate any thoughts/ideas?

[bmacias84]

Heavy Forwards do not affect your license, indexing your data does. You could configure you indexer to filter events using a transform, but thats added load on your indexer. Not sure how good your coding skills are, but you could use a scripted input using powershell only collecting/filtering events that are important to you. I've done similar VB/Powershell scripts to pull events.

The other painful exercise would be to work with your Windows Team to clean/fix all your Warns, denies, and errors. Whats the good of logging it if you are not going to fix it.

If I am not mistaken you License cost should not occur untill you hit hte indexQueue.

I could also post an old clunky VBs script to give you some ideas.

Update:

This a vb script I used years ago. Its not pretty by any standard but it the need. Powershell would could be alot simpler. This provided as is. Not my best work (little imbarrassed to show it)

''''''Script Create By Bmacias84

''''''Description: Using WMI Win32_NTLogEvent script searches last 5 minutes of Event logs that matches condition then print to log file eventlogerror.tsv.

''''''Purpose: To detect any warning, errors, or events of interest

''Win32_NTLogEVENT Class discription

'  uint16   Category;

'  string   CategoryString;

'  string   ComputerName;

'  uint8    Data[];

'  uint16   EventCode;

'  uint32   EventIdentifier;

'  uint8    EventType;

'  string   InsertionStrings[];

'  string   Logfile;

'  string   Message;

'  uint32   RecordNumber;

'  string   SourceName;

'  datetime TimeGenerated;

'  datetime TimeWritten;

'  string   Type;

'  string   User;

Dim objWMIService, colSysEventLogs, colAppEventLogs, objRegEx, aEvent, sEvent, objFSO, objLogFile

Dim arrMesssage

Dim intDay, intMonth, intYear, intHour, intMin, intNow

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")

Set colSysEventLogs = objWMIService.ExecQuery("Select * from Win32_NTLogEvent Where Logfile = 'System'" )

Set colAppEventLogs = objWMIService.ExecQuery("Select * from Win32_NTLogEvent Where Logfile = 'Application'" )

Set objFSO = CreateObject("Scripting.FileSystemObject")

If objFSO.FileExists("%logdir%\eventlogerror.tsv") = True Then

    objFSO.DeleteFile "%logdir%\eventlogerror.tsv", True

End If

Set objLogFile = objFSO.OpenTextFile _

    ("%logdir%\eventlogerror.tsv", 8, True) 

For Each sEvent In colSysEventLogs

    intDay = Mid(sEvent.TimeWritten,7,2)

    intMonth = Mid(sEvent.TimeWritten,5,2)

    intYear = Mid(sEvent.TimeWritten,1,4)

    intHour = Mid(sEvent.TimeWritten,9,2)

    intMin = Mid(sEvent.TimeWritten,11,2)

    intNow = Now()

    If DateDiff("n",CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)),intNow) < 5 Then

        If (sEvent.SourceName = "W3SVC" And (sEvent.Type = "Warning" Or sEvent.Type = "Error")) Or (sEvent.SourceName = "IISCTLS" And (sEvent.EventCode = "6" Or sEvent.EventCode = "2" Or sEvent.EventCode = "4" Or sEvent.EventCode = "1")) Then

            objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & sEvent.ComputerName & "," & sEvent.EventCode & "," &  sEvent.SourceName & "," & sEvent.Message)

        Else

            Set objRegEx = CreateObject("VBScript.RegExp")

                objRegEx.Pattern = "C\+\+"

                objRegEx.Global = True

                objRegEx.IgnoreCase = True

            Set arrMesssage = objRegEx.Execute(sEvent.Message)

            If arrMesssage.Count > 0 Then

                objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & sEvent.ComputerName & "," & sEvent.EventCode & "," &  sEvent.SourceName & ","  & sEvent.Message)

            Else

            End If

        End If

    Else

        Exit For

    End If

Next

For Each aEvent In colAppEventLogs

    intDay = Mid(aEvent.TimeWritten,7,2)

    intMonth = Mid(aEvent.TimeWritten,5,2)

    intYear = Mid(aEvent.TimeWritten,1,4)

    intHour = Mid(aEvent.TimeWritten,9,2)

    intMin = Mid(aEvent.TimeWritten,11,2)

    intNow = Now()

    If DateDiff("n",CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)),intNow) < 5 Then

        If (aEvent.SourceName = "NET" Or aEvent.SourceName = "ASP.NET 2.0.xxxx") And (aEvent.Type = "Warning" Or aEvent.Type = "Error") Then

            objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & aEvent.ComputerName & "," & aEvent.EventCode & "," &  aEvent.SourceName & "," & aEvent.Message)

        End If

    Else

        Exit For

    End If

Next

objLogFile.Close

[bmacias84]

Modifying the Props.conf and transforms.conf will work on data currently making its way to the indexqueue, but not on data that has been indexed already, might need to double check. http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/Howindexingworks

Alternatively you could configure a Heavy Intermediate forwarder to recieve Universal Forwarder data. The Heavy Intermediate Forwarder would filter the data and send it to your indexer. http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Forwarderdeploymenttopologies#Intermediate_fo...

[mrflibbleuk]

Unfortunately I am not good at coding so any help would be much appreciated. I am pretty new to Splunk as well so apologies for any daft questions.

So the info I have seen around modifying props.conf and transforms.conf to limit event logs that are sent will only work on indexed data? therefore requiring a license?

[pwattssplunk]

Licensing is only applied to indexed data, so if you're using a heavy forwarded to perform transforms (pruning or otherwise) prior to being sent to the indexer you aren't consuming license bandwidth.