topic Why is the text Log Broken into Multiple Events? in Getting Data In

Why is the text Log Broken into Multiple Events?

bteele — Mon, 29 Jan 2018 16:30:57 GMT

We have Powershell logs being written to text files along with a Windows path. We have a Splunk app monitoring that location for data. Splunk is ingesting the data from the files, but the data is being broken into numerous events. For small text files, it might be 2 or 3 events in Splunk for one file. For larger files, it can be upwards of 7 events.

Does anyone know why Splunk is ingesting the data in pieces? We'd prefer to have all of the data from one file in a single event.

App inputs config:

[monitor://C:\Windows\Logs\Powershell]
index = winpowershell
disabled = 0

Example Text Content:

#line of asterisks###################
Windows PowerShell transcript start
Start time: 20180129104808
Username: blah\blah
RunAs User: blah\blah
Machine: nameblah (OS version blah)
Host Application: C:\Windows\system32\longscriptblah
Process ID: number
PSVersion: number
PSEdition: thing
PSCompatibleVersions: numbers
BuildVersion: number
CLRVersion: number
WSManStackVersion: number
PSRemotingProtocolVersion: number
SerializationVersion: number
#line of asterisks###################
PS>& 'path'
PS51
PS>$global:?
True
#line of asterisks###################
Windows PowerShell transcript end
End time: 20180129104808
#line of asterisks###################

Example Splunk Events:

Event 1: 
#line of asterisks###################
Windows PowerShell transcript start

Event 2:

Start time: 20180129104808
Username: blah\blah
RunAs User: blah\blah
Machine: nameblah (OS version blah)
Host Application: C:\Windows\system32\longscriptblah
Process ID: number
PSVersion: number
PSEdition: thing
PSCompatibleVersions: numbers
BuildVersion: number
CLRVersion: number
WSManStackVersion: number
PSRemotingProtocolVersion: number
SerializationVersion: number
#line of asterisks###################
PS>& 'path'
PS51
PS>$global:?
True
#line of asterisks###################
Windows PowerShell transcript end

Event 3:

End time: 20180129104808
#line of asterisks###################

Re: Why is the text Log Broken into Multiple Events?

richgalloway — Mon, 29 Jan 2018 22:04:43 GMT

What are the props.conf settings for the souretype?
Depending on how frequently the file is updated, you may want to set time_before_close to a value greater than 3 in inputs.conf.

Re: Why is the text Log Broken into Multiple Events?

davpx — Tue, 29 Sep 2020 17:55:11 GMT

You will want to configure proper line breaking on this input by specifying the following parameters for this in props.conf on your indexer. For this file it should look something like...

[source::C:\Windows\Logs\Powershell]
SHOULD_LINEMERGE = false
LINE_BREAKER = (End time:\s+\d+)([\r\n]+)
TIME_FORMAT = %Y%m%d%H%M%S
TIME_PREFIX = ^Start Time:
MAX_TIMESTAMP_LOOKAHEAD = 14

Re: Why is the text Log Broken into Multiple Events?

vincenteous — Mon, 29 Jan 2018 22:38:57 GMT

Hi bteele,

I think you can use the trick in the link here to fulfill your requirement.
Basically, you have to use props.conf file in your indexer to merge the incoming events from your powershell sourcetype and set up line breaker to a non-existent word/character.

Hope this helps.

Re: Why is the text Log Broken into Multiple Events?

bteele — Wed, 14 Feb 2018 16:40:45 GMT

Thanks. I updated the indexer props file with this, but it's still breaking the text up into different events. I haven't had much time to troubleshoot it further, but wanted to thank you for your help so far.

Any idea where I should start troubleshooting?