I've set up a scripted input for a Windows forwarder using Powershell. The script works and outputs the data that I need. The output is multiple lines each starting with a date-time stamp. The first two lines come back as a single event and all of the remaining lines come back as individual events (~40 lines). I'd like to have each line come back as a single event or if that's not possible have all of the lines come back as a single event. What can I do?
To sum up.
How do get the first two lines from the script to come back as individual events?
The output looks like this:
05/16/2012 14:23:39 esxhost=ssvmaccc1n2 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=1628 MemoryTotalMB=524275 MemoryUsageMB=144398 VMCount=14 cluster=MACCCluster 05/16/2012 14:23:40 esxhost=ssvmaccc1n1 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=1223 MemoryTotalMB=524275 MemoryUsageMB=37190 VMCount=2 cluster=MACCCluster 05/16/2012 14:23:41 esxhost=ssvcloudn1 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=285 MemoryTotalMB=524277 MemoryUsageMB=4780 VMCount=0 cluster=CloudCluster 05/16/2012 14:23:42 esxhost=ssvcloudn2 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=2675 MemoryTotalMB=524277 MemoryUsageMB=9519 VMCount=3 cluster=CloudCluster 05/16/2012 14:23:43 esxhost=ssvcloudn3 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=320 MemoryTotalMB=524277 MemoryUsageMB=4772 VMCount=0 cluster=CloudCluster 05/16/2012 14:23:44 esxhost=ssvcloudn4 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=384 MemoryTotalMB=524277 MemoryUsageMB=4647 VMCount=0 cluster=CloudCluster
Why not saving the output of your script into a file, then use the Data Input "Preview data before indexing" wizard to see how Splunk would understand it?
My opinion is because you have all events starting by a date + time, Splunk will be able to split it correctly and identify the time.
Hope this helps.
I've actually done the data preview with this and Splunk does handle it just fine.
The problem is that given the above input the first two lines show up as a single event while each remaining line shows up as its own event. So for 45 lines I get 44 events when I should be getting 45.
In props.conf, you want to disable linebreaking. Just create (or use an existing) stanza for the sourcetype for your scripted input:
[your_sourcetype] SHOULD_LINEMERGE = false
For more info, check out the props.conf documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf
I'm not familiar with powershell but it sounds like you authored the script and could modify it yourself. If that's the case you could define your own line breaker. The line breaker is then defined in props.conf and tells Splunk "hey Splunk, when you see my line breaker, it means an event is ending and a new one is about to begin (unless it's the last event of course)".
you script output could add a string between each event that you're sure will never be also found in your event...like 'regexuneverfind'
05/16/2012 14:23:39 esxhost=ssvmaccc1n2 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=1628 MemoryTotalMB=524275 MemoryUsageMB=144398 VMCount=14 cluster=MACCCluster
05/16/2012 14:23:43 esxhost=ssvcloudn3 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=320 MemoryTotalMB=524277 MemoryUsageMB=4772 VMCount=0 cluster=CloudCluster
05/16/2012 14:23:44 esxhost=ssvcloudn4 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=384 MemoryTotalMB=524277 MemoryUsageMB=4647 VMCount=0 cluster=CloudCluste
In props.conf tell splunk what breaks your events by defining the regex that matches your line breaker:
[yourname] REPORT-yourname = yourname TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = FALSE TRUNCATE = 0 LINE_BREAKER = (regexuneverfind)