topic Issue with monitoring one specific log file in Getting Data In

Issue with monitoring one specific log file

AKG1_old1 — Wed, 03 Apr 2019 12:50:28 GMT

Hi,

I am monitoring multiple files/directory under different sourcetype. For one specific log file I am getting wiered behavior.
It's not being monitored Continuously, even though file is getting updated regularly.

I am not getting any relevant error at both Splunk and forwarder side.

Whenever I install new forwarder and configure this file to read, file is being picked only once and stop updating . (It's like reading a batch file)

inputs.conf

[monitor:///net/hp707srv/hp707srv2/apps/QCST_RSAT_v3.1.42_MASTER/qcstTools/qcst_out_alerts.log]
disabled = false
host = MTE_TEST
index = mlc_live
sourcetype = MTE_ALERT
crcSalt = <Source>

Re: Issue with monitoring one specific log file

Laszlo_K — Wed, 03 Apr 2019 13:43:59 GMT

Have you considered crcSalt as described in https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ?

Re: Issue with monitoring one specific log file

lakshman239 — Wed, 03 Apr 2019 15:35:40 GMT

When you define monitor stanza (the others in your inputs.conf in the UF/HF), are you ensuring that no other stanza is resolving to the above path ///net/hp707srv/hp707srv2/apps/QCST_RSAT_v3.1.42_MASTER/qcstTools/ ?

Also, how often does this file get updated and rotated? did you try crcSalt /crc checksum length?

Re: Issue with monitoring one specific log file

AKG1_old1 — Wed, 03 Apr 2019 15:46:05 GMT

I have tried installing fresh forwarder for monitoring only this file. After starting the forwarder full file injested in Splunk but later on it's not getting updated.

I have used crcSalt = as well but didn't work.

Around 30-50 lines are updated in one hour.

Re: Issue with monitoring one specific log file

AKG1_old1 — Wed, 03 Apr 2019 16:08:42 GMT

yesI have tried with

 crcSalt = <Source>

Re: Issue with monitoring one specific log file

lakshman239 — Thu, 04 Apr 2019 04:05:17 GMT

Assuming, you get new events every 1hr, are you seeing any warning/errors in splunkd.log from the time your file is first indexed to say till next 1 or 2 hrs? [ e.g file crc checksum error, file ignored, parsing error]. Also, using the metrics.log, can you check if you are constantly receiving other _internal logs from the host, so we can isolate the issue to only this specific file. I assume this is a normal text file.

Re: Issue with monitoring one specific log file

AKG1_old1 — Wed, 30 Sep 2020 00:01:38 GMT

Issue is resolved by updating TIME_FORMAT In props.conf
Earlier TIME_FORMAT was not defined. but wiered thing is it was working fine initially for a month with no TIME_FORMAT . My assumption is if its not defined it takes current time bydefault.

props.conf
[MTE_ALERT]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true
REPORT-MTE_ALERT = REPORT-MTE_ALERT
TIME_FORMAT = %d/%m/%Y | %H:%M:%S
TIME_PREFIX = ^

Re: Issue with monitoring one specific log file

AKG1_old1 — Thu, 04 Apr 2019 08:40:31 GMT

@lakshman239 : Thanks for help. its got resovled.