topic Why did a Splunk forwarder stop sending to log packet? in Getting Data In

Why did a Splunk forwarder stop sending to log packet?

lifekis — Tue, 07 Jan 2020 07:36:23 GMT

It was working fine until 1 month ago.
There was no Splunk forwarder and network configuration change.
No packets from the forwarder to the indexer.
How can I solve the problem?

Re: Why did a Splunk forwarder stop sending to log packet?

gcusello — Tue, 07 Jan 2020 11:54:55 GMT

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

Re: Why did a Splunk forwarder stop sending to log packet?

lifekis — Wed, 08 Jan 2020 00:02:40 GMT

I haven't received it since December 13, 2019 (GMT + 9). There is a log in the forwarder but I don't know why not forwarding it.

Re: Why did a Splunk forwarder stop sending to log packet?

gnangia — Wed, 08 Jan 2020 00:19:26 GMT

Here are the steps I would take to troubleshoot -
(1) Login to the server to make sure splunk process is running
(2) Check that your outputs.conf is still on the system
(3) Try to do a network connectivity test to your HF/Indexer
(4) Check splunkd.log to see if there are any errors

Re: Why did a Splunk forwarder stop sending to log packet?

gcusello — Wed, 08 Jan 2020 07:54:24 GMT

Hi @lifekis,
ok, let start dubugging:
at first check if you have internal logs from that server (index=_internal host=your_host).
If you haven't them, the problem is on the connection, so test the route (from the UF telnet ip_splunk_indexer 9997).
if Nok verify firewall route between servers, if Ok check in outputs.conf on UF if the address of the Indexers is correct.

If instead you have Splunk internal logs but not the logs you want, check in inputs.conf if the path and the filename of the monitor stanza is still correct.

Ciao.
Giuseppe

Re: Why did a Splunk forwarder stop sending to log packet?

lifekis — Fri, 10 Jan 2020 07:45:36 GMT

Thank you for answer! I checked the disconnection log in splunkd.log.