topic Re: Events getting distorted in splunk production in Getting Data In

Events getting distorted in splunk production

swamysanjanaput — Wed, 30 Sep 2020 02:37:32 GMT

Hi Splunkers,

I am trying to ingest os_metrics logs from one of our prod server to splunk. In QA and dev instance, events are breaking correctly. I pushed the same configs(see below) to production server however i see distorted events when searching the data in prod SH for e.g Thu 10/10/2019 0:43:56.32 Checking "ABC" as one event and ping results as another event. Similarly Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR" as one event and physical address details as another event(below is the sample data)

Splunk is reading old data files from production server and i am able to see old data breaking into events correctly but when new data started to ingest, i see them all getting distorted So, Do i have to place props in our SH cluster or is it something to do with props?

Can someone please help me to resolve this issue? Thanks in advance.

Sample data:
Thu 10/10/2019 0:43:56.32 Checking "ABC"

Pinging ABC [ip] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 0.0.0.0:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR"

Physical Address Transport Name

=================== ==========================================================
\Device\Tcpip_{}

N/A Media disconnected

N/A Media disconnected

N/A Media disconnected

props.conf
[xyz]
NO_BINARY_CHECK=true
CHARSET=UTF-8
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE = \w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+
disabled=false

inputs.conf
[monitor://abc*.log]
disabled = 0
index = xxxxx
sourcetype = xyz

Re: Events getting distorted in splunk production

somesoni2 — Tue, 15 Oct 2019 14:36:30 GMT

Change your sourcetype definition in props.conf with this

[xyz]
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+)
TIME_PREFIX = ^
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%N

Re: Events getting distorted in splunk production

gcusello — Tue, 15 Oct 2019 14:39:58 GMT

Hi swamysanjanaputta,
have you in production environment also Heavy Forwarders between sources and Indexers?
If yes, put the props.conf also on Heavy Forwarders (and restart Splunk on them).

Ciao.
Giuseppe

Re: Events getting distorted in splunk production

swamysanjanaput — Fri, 25 Oct 2019 10:12:30 GMT

Hi, Yes i had initially deployed props to HFs, not sure why data is getting distorted, i see 50% events distorted and other 50% breaking into events correctly. so should i place props on Search Head cluster?

Re: Events getting distorted in splunk production

swamysanjanaput — Fri, 25 Oct 2019 10:16:26 GMT

Thanks for the props, still facing the same issue. I had placed props in HFs aswell but not sure why data is getting distorted. So, do i have to place the props in SH cluster? Please advise..