Getting Data In

scripted input as multiple events

colinj
Path Finder

Howdy,

I've set up a scripted input for a Windows forwarder using Powershell. The script works and outputs the data that I need. The output is multiple lines each starting with a date-time stamp. The first two lines come back as a single event and all of the remaining lines come back as individual events (~40 lines). I'd like to have each line come back as a single event or if that's not possible have all of the lines come back as a single event. What can I do?

To sum up.

  • scripted input
  • multi-line output
  • each line has a timestamp
  • first two lines are a single event
  • each remaining line is its own event

How do get the first two lines from the script to come back as individual events?

The output looks like this:

05/16/2012 14:23:39 esxhost=ssvmaccc1n2 NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=1628    MemoryTotalMB=524275    MemoryUsageMB=144398    VMCount=14  cluster=MACCCluster
05/16/2012 14:23:40 esxhost=ssvmaccc1n1 NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=1223    MemoryTotalMB=524275    MemoryUsageMB=37190 VMCount=2   cluster=MACCCluster
05/16/2012 14:23:41 esxhost=ssvcloudn1  NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=285 MemoryTotalMB=524277    MemoryUsageMB=4780  VMCount=0   cluster=CloudCluster
05/16/2012 14:23:42 esxhost=ssvcloudn2  NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=2675    MemoryTotalMB=524277    MemoryUsageMB=9519  VMCount=3   cluster=CloudCluster
05/16/2012 14:23:43 esxhost=ssvcloudn3  NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=320 MemoryTotalMB=524277    MemoryUsageMB=4772  VMCount=0   cluster=CloudCluster
05/16/2012 14:23:44 esxhost=ssvcloudn4  NumCpu=16   CpuTotalMhz=36224   CpuUsageMhz=384 MemoryTotalMB=524277    MemoryUsageMB=4647  VMCount=0   cluster=CloudCluster
Tags (3)

jpass
Contributor

I'm not familiar with powershell but it sounds like you authored the script and could modify it yourself. If that's the case you could define your own line breaker. The line breaker is then defined in props.conf and tells Splunk "hey Splunk, when you see my line breaker, it means an event is ending and a new one is about to begin (unless it's the last event of course)".

you script output could add a string between each event that you're sure will never be also found in your event...like 'regexuneverfind'
Example:
05/16/2012 14:23:39 esxhost=ssvmaccc1n2 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=1628 MemoryTotalMB=524275 MemoryUsageMB=144398 VMCount=14 cluster=MACCCluster
regexyoullneverfind
05/16/2012 14:23:43 esxhost=ssvcloudn3 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=320 MemoryTotalMB=524277 MemoryUsageMB=4772 VMCount=0 cluster=CloudCluster
regexyoullneverfind
05/16/2012 14:23:44 esxhost=ssvcloudn4 NumCpu=16 CpuTotalMhz=36224 CpuUsageMhz=384 MemoryTotalMB=524277 MemoryUsageMB=4647 VMCount=0 cluster=CloudCluste

In props.conf tell splunk what breaks your events by defining the regex that matches your line breaker:

[yourname]
REPORT-yourname = yourname
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = FALSE
TRUNCATE = 0
LINE_BREAKER = (regexuneverfind)
0 Karma

pstout
Splunk Employee
Splunk Employee

In props.conf, you want to disable linebreaking. Just create (or use an existing) stanza for the sourcetype for your scripted input:

[your_sourcetype]
SHOULD_LINEMERGE = false

For more info, check out the props.conf documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

0 Karma

jwalthour
Communicator

I downvoted this post because this does not work.

0 Karma

OL
Communicator

Hello,

Why not saving the output of your script into a file, then use the Data Input "Preview data before indexing" wizard to see how Splunk would understand it?

My opinion is because you have all events starting by a date + time, Splunk will be able to split it correctly and identify the time.

Hope this helps.
Olivier

0 Karma

colinj
Path Finder

I've actually done the data preview with this and Splunk does handle it just fine.

The problem is that given the above input the first two lines show up as a single event while each remaining line shows up as its own event. So for 45 lines I get 44 events when I should be getting 45.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...