Getting Data In

Why did a Splunk forwarder stop sending to log packet?

lifekis
Explorer

It was working fine until 1 month ago.
There was no Splunk forwarder and network configuration change.
No packets from the forwarder to the indexer.
How can I solve the problem?

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

View solution in original post

0 Karma

gnangia
Explorer

Here are the steps I would take to troubleshoot -
(1) Login to the server to make sure splunk process is running
(2) Check that your outputs.conf is still on the system
(3) Try to do a network connectivity test to your HF/Indexer
(4) Check splunkd.log to see if there are any errors

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

0 Karma

lifekis
Explorer

I haven't received it since December 13, 2019 (GMT + 9). There is a log in the forwarder but I don't know why not forwarding it.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
ok, let start dubugging:
at first check if you have internal logs from that server (index=_internal host=your_host).
If you haven't them, the problem is on the connection, so test the route (from the UF telnet ip_splunk_indexer 9997).
if Nok verify firewall route between servers, if Ok check in outputs.conf on UF if the address of the Indexers is correct.

If instead you have Splunk internal logs but not the logs you want, check in inputs.conf if the path and the filename of the monitor stanza is still correct.

Ciao.
Giuseppe

0 Karma

lifekis
Explorer

Thank you for answer! I checked the disconnection log in splunkd.log.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...