topic Re: Line breaking powershell output in Getting Data In

Line breaking powershell output

a_splunk_user — Tue, 29 Sep 2020 14:07:42 GMT

I've attempted multiple times mixing up LINE_BREAKER, BREAK_ONLY_BEFORE, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, nothing seems to work - I've obviously missed something.

Sample:
...........................
EventDate : 2017-05-19 14:20
Host : Server8
InstanceName : oracle
CPU_Percent : 15.19

EventDate : 2017-05-19 14:20
Host : Server8
InstanceName : oracle
CPU_Percent : 12.40

EventDate : 2017-05-19 14:20
Host : Server8
InstanceName : powershell
CPU_Percent : 0.19
...........................

The data comes in as sourcetype Windows:processmonitor from a UF and I currently have the following in my indexer props.conf:
...........................
[Windows:processmonitor]

LINE_BREAKER = EventDate

BREAK_ONLY_BEFORE = EventDate

SHOULD_LINEMERGE = false

TIME_FORMAT = %Y-%m-%d %H:%M
BREAK_ONLY_BEFORE_DATE = true
...........................

Any advice greatly appreciated!

Re: Line breaking powershell output

aakwah — Fri, 19 May 2017 22:27:12 GMT

Hello,

What is missing is SHOULD_LINEMERGE = true, The following stanza in props.conf should work fine:

[Windows:processmonitor]
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M

Regards

Re: Line breaking powershell output

woodcock — Fri, 19 May 2017 22:50:13 GMT

Try this:

[Windows:processmonitor]
LINE_BREAKER = ([\r\n]+)EventDate
SHOULD_LINEMERGE = false
TIME_PREFIX = EventDate\s*:\s*
TIME_FORMAT = %Y-%m-%d %H:%M
MAX_TIMESTAMP_LOOKAHEAD = 16

Re: Line breaking powershell output

a_splunk_user — Tue, 29 Sep 2020 14:09:37 GMT

Thanks all, however no luck so far in either case, unfortunately.

I did however notice in the health check that there were issues in the 'Event-processing issues' section, relating to events for this sourcetype being too long in bytes, and event max was also an issue.
...............

Message

Some recently ingested events are triggering event-processing warnings and indicate the presence of one or more of these scenarios:
1. Lines in the event are too long, exceeding props.conf / TRUNCATE
2. There are too many lines per event, exceeding props.conf / MAX_EVENTS
3. The extraction of event time stamps was partially or completely unsuccessful
These event-processing issues can have a negative impact on the performance of data ingestion.
Suggested Action

Check the events that are triggering these warnings. Adjust event-processing settings as needed to ensure their proper ingestion.
...................

So I added TRUNCATE and MAX_EVENTS to the stanza, to result in the following:
.......................
[procmonitor]
TRUNCATE = 15000
MAX_EVENTS = 300
LINE_BREAKER = ([\r\n]+)EventDate
SHOULD_LINEMERGE = false
TIME_PREFIX = EventDate\s*:\s*
TIME_FORMAT = %Y-%m-%d %H:%M
MAX_TIMESTAMP_LOOKAHEAD = 16
.......................
Note I also changed the sourcetype name just to be sure there was no issue there.

Now the length and count issues are no longer showing up, but the data is the same one big event.

I'm wondering if what I see on the screen and what Splunk is looking at are two different things. I'm also wondering if I should modify my script to make the format more digestible to Splunk, somehow - perhaps xml.