Getting Data In

Techniques for limiting size of windows event logs

New Member

Hi,

We currently have a number of servers that have the universal forwarder installed and set to forward to an Enterprise Splunk instance. Unfortunately the security logs themselves are generating a massive amount of data. We want to limit the data sent to Splunk, my understanding is we can use transforms etc to achieve this BUT this needs a heavy forwarder installed (i.e. a full Splunk installation). I am not sure of the license implications for this.

Does anyone here have any thoughts on ways around this one? I did think of maybe filtering the events in windows then exporting to a file and then picking it up but its a bit clunky.
Maybe use another Windows Forwarder style app and then forward the event logs as syslog entries to Splunk.

Would appreciate any thoughts/ideas?

Tags (1)
0 Karma

Champion

Heavy Forwards do not affect your license, indexing your data does. You could configure you indexer to filter events using a transform, but thats added load on your indexer. Not sure how good your coding skills are, but you could use a scripted input using powershell only collecting/filtering events that are important to you. I've done similar VB/Powershell scripts to pull events.

The other painful exercise would be to work with your Windows Team to clean/fix all your Warns, denies, and errors. Whats the good of logging it if you are not going to fix it.

If I am not mistaken you License cost should not occur untill you hit hte indexQueue.

I could also post an old clunky VBs script to give you some ideas.

Update:

This a vb script I used years ago. Its not pretty by any standard but it the need. Powershell would could be alot simpler. This provided as is. Not my best work (little imbarrassed to show it)


''''''Script Create By Bmacias84
''''''Description: Using WMI Win32_NTLogEvent script searches last 5 minutes of Event logs that matches condition then print to log file eventlogerror.tsv.
''''''Purpose: To detect any warning, errors, or events of interest
''Win32_NTLogEVENT Class discription
' uint16 Category;
' string CategoryString;
' string ComputerName;
' uint8 Data[];
' uint16 EventCode;
' uint32 EventIdentifier;
' uint8 EventType;
' string InsertionStrings[];
' string Logfile;
' string Message;
' uint32 RecordNumber;
' string SourceName;
' datetime TimeGenerated;
' datetime TimeWritten;
' string Type;
' string User;
Dim objWMIService, colSysEventLogs, colAppEventLogs, objRegEx, aEvent, sEvent, objFSO, objLogFile
Dim arrMesssage
Dim intDay, intMonth, intYear, intHour, intMin, intNow
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")
Set colSysEventLogs = objWMIService.ExecQuery("Select * from Win32_NTLogEvent Where Logfile = 'System'" )
Set colAppEventLogs = objWMIService.ExecQuery("Select * from Win32_NTLogEvent Where Logfile = 'Application'" )
Set objFSO = CreateObject("Scripting.FileSystemObject")
If objFSO.FileExists("%logdir%\eventlogerror.tsv") = True Then
objFSO.DeleteFile "%logdir%\eventlogerror.tsv", True
End If
Set objLogFile = objFSO.OpenTextFile _
("%logdir%\eventlogerror.tsv", 8, True)
For Each sEvent In colSysEventLogs
intDay = Mid(sEvent.TimeWritten,7,2)
intMonth = Mid(sEvent.TimeWritten,5,2)
intYear = Mid(sEvent.TimeWritten,1,4)
intHour = Mid(sEvent.TimeWritten,9,2)
intMin = Mid(sEvent.TimeWritten,11,2)
intNow = Now()
If DateDiff("n",CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)),intNow) < 5 Then
If (sEvent.SourceName = "W3SVC" And (sEvent.Type = "Warning" Or sEvent.Type = "Error")) Or (sEvent.SourceName = "IISCTLS" And (sEvent.EventCode = "6" Or sEvent.EventCode = "2" Or sEvent.EventCode = "4" Or sEvent.EventCode = "1")) Then
objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & sEvent.ComputerName & "," & sEvent.EventCode & "," & sEvent.SourceName & "," & sEvent.Message)
Else
Set objRegEx = CreateObject("VBScript.RegExp")
objRegEx.Pattern = "C\+\+"
objRegEx.Global = True
objRegEx.IgnoreCase = True
Set arrMesssage = objRegEx.Execute(sEvent.Message)
If arrMesssage.Count > 0 Then
objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & sEvent.ComputerName & "," & sEvent.EventCode & "," & sEvent.SourceName & "," & sEvent.Message)
Else
End If
End If
Else
Exit For
End If
Next
For Each aEvent In colAppEventLogs
intDay = Mid(aEvent.TimeWritten,7,2)
intMonth = Mid(aEvent.TimeWritten,5,2)
intYear = Mid(aEvent.TimeWritten,1,4)
intHour = Mid(aEvent.TimeWritten,9,2)
intMin = Mid(aEvent.TimeWritten,11,2)
intNow = Now()
If DateDiff("n",CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)),intNow) < 5 Then
If (aEvent.SourceName = "NET" Or aEvent.SourceName = "ASP.NET 2.0.xxxx") And (aEvent.Type = "Warning" Or aEvent.Type = "Error") Then
objLogFile.WriteLine(CDate(DateSerial(intYear,intMonth,intDay) & " " & TimeSerial(intHour,intMin,0)) & "," & aEvent.ComputerName & "," & aEvent.EventCode & "," & aEvent.SourceName & "," & aEvent.Message)
End If
Else
Exit For
End If
Next
objLogFile.Close

0 Karma

Champion

Modifying the Props.conf and transforms.conf will work on data currently making its way to the indexqueue, but not on data that has been indexed already, might need to double check. http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/Howindexingworks

Alternatively you could configure a Heavy Intermediate forwarder to recieve Universal Forwarder data. The Heavy Intermediate Forwarder would filter the data and send it to your indexer. http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Forwarderdeploymenttopologies#Intermediate_fo...

0 Karma

New Member

Unfortunately I am not good at coding so any help would be much appreciated. I am pretty new to Splunk as well so apologies for any daft questions.

So the info I have seen around modifying props.conf and transforms.conf to limit event logs that are sent will only work on indexed data? therefore requiring a license?

0 Karma

Splunk Employee
Splunk Employee

Licensing is only applied to indexed data, so if you're using a heavy forwarded to perform transforms (pruning or otherwise) prior to being sent to the indexer you aren't consuming license bandwidth.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!