Events getting distorted in splunk production

[swamysanjanaput]

Hi Splunkers,

I am trying to ingest os_metrics logs from one of our prod server to splunk. In QA and dev instance, events are breaking correctly. I pushed the same configs(see below) to production server however i see distorted events when searching the data in prod SH for e.g Thu 10/10/2019 0:43:56.32 Checking "ABC" as one event and ping results as another event. Similarly Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR" as one event and physical address details as another event(below is the sample data)

Splunk is reading old data files from production server and i am able to see old data breaking into events correctly but when new data started to ingest, i see them all getting distorted So, Do i have to place props in our SH cluster or is it something to do with props?

Can someone please help me to resolve this issue? Thanks in advance.

Sample data:
Thu 10/10/2019 0:43:56.32 Checking "ABC"

Pinging ABC [ip] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 0.0.0.0:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR"

Physical Address Transport Name

=================== ==========================================================
\Device\Tcpip_{}

N/A Media disconnected

N/A Media disconnected

N/A Media disconnected

props.conf
[xyz]
NO_BINARY_CHECK=true
CHARSET=UTF-8
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE = \w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+
disabled=false

inputs.conf
[monitor://abc*.log]
disabled = 0
index = xxxxx
sourcetype = xyz

[gcusello]

Hi swamysanjanaputta,
have you in production environment also Heavy Forwarders between sources and Indexers?
If yes, put the props.conf also on Heavy Forwarders (and restart Splunk on them).

Ciao.
Giuseppe

[swamysanjanaput]

Hi, Yes i had initially deployed props to HFs, not sure why data is getting distorted, i see 50% events distorted and other 50% breaking into events correctly. so should i place props on Search Head cluster?

[somesoni2]

Change your sourcetype definition in props.conf with this

[xyz]
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+)
TIME_PREFIX = ^
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%N

[swamysanjanaput]

Thanks for the props, still facing the same issue. I had placed props in HFs aswell but not sure why data is getting distorted. So, do i have to place the props in SH cluster? Please advise..