Why did a Splunk forwarder stop sending to log packet?

[lifekis]

It was working fine until 1 month ago.
There was no Splunk forwarder and network configuration change.
No packets from the forwarder to the indexer.
How can I solve the problem?

[gnangia]

Here are the steps I would take to troubleshoot -
(1) Login to the server to make sure splunk process is running
(2) Check that your outputs.conf is still on the system
(3) Try to do a network connectivity test to your HF/Indexer
(4) Check splunkd.log to see if there are any errors

[gcusello · Accepted solution]

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

[lifekis]

I haven't received it since December 13, 2019 (GMT + 9). There is a log in the forwarder but I don't know why not forwarding it.

[gcusello]

Hi @lifekis,
ok, let start dubugging:
at first check if you have internal logs from that server (index=_internal host=your_host).
If you haven't them, the problem is on the connection, so test the route (from the UF telnet ip_splunk_indexer 9997).
if Nok verify firewall route between servers, if Ok check in outputs.conf on UF if the address of the Indexers is correct.

If instead you have Splunk internal logs but not the logs you want, check in inputs.conf if the path and the filename of the monitor stanza is still correct.

Ciao.
Giuseppe

[lifekis]

Thank you for answer! I checked the disconnection log in splunkd.log.