Why is one device out of others are receiving logs by syslog-ng but not by splunk cloud?

[cbwillh]

We have an Ubuntu syslog-ng server with the splunk forwarder.

It is configured with 18 device types (over 100 devices in those device type classes) sending logs to the syslog-ng server and then those logs are forwarded to splunk cloud by the universal forwarder.

Everything has been working perfectly for months and all data from all logs have been confirmed as arriving in splunk cloud through daily searches for ALL syslog indexes and sourcetypes configured.

As of yesterday, 1 of 18 devices is returning "0" results found when running a search for its index and sourcetype even though the syslog-ng server contains current up to date logs for the device in the location specified in the syslog-ng.conf configuration file.

So the issue is that the logs are making it to the syslog-ng server and being processed into the folder specified in the syslog-ng.conf file, but they are not making it from that location to the splunk cloud servers.

Nothing has changed on the syslog-ng server and I have already restarted BOTH the syslog-ng and the splunk services and even tried a reboot of the server with no change to the issue.

Anyone else ever seen this?

[cbwillh · Accepted solution]

thanks for the help nittala_surya, I appreciate the time you took to respond.
I believe I nutted this out on my own though and will post what I found for anyone else who might run across this.

I compared the last entry for the sourcetype in splunk cloud with that sourcetypes physical log on the syslog-ng server

I noted that the hour was different but the minutes and seconds matched perfectly. this led me to suspect the timezone was off somewhere.

So I checked the splunk cloud -> Settings -> Source Types settings for the sourcetype I was having the issues with...

and I found that the timezone set for the Source Type that was configured in Splunk Clouds Settings had the incorrect Time Zone set. I corrected this and I also added the following to my syslog-ng.conf files line for the udp connection settings:
time_zone(PST8PDT) keep_timestamp(no)

I then saved my changes and restarted the syslog-ng service and did a service syslog-ng status to confirm it came good and was started.
I monitored over the next 15 minutes and noted that my logs were starting to show with the correct times in splunk cloud again.

I am guessing that somehow this misconfigured Source Type was there since it was created and perhaps a dip in traffic to that log yesterday helped it to become obvious for the first time.
at any rate she seems to be working again now.

I noted that the timezone was incorrect.

[sudosplunk]

Glad you were able to resolve this. Happy Splunking!

PS: You can accept your answer to mark this question complete.

[sudosplunk]

I've seen this in the past but a restart would fix it. If 1 of 18 devices is having this issue, then there is a chance of latency or corrupted files. See how your thruput looks like for that sourcetype.

kbps by sourcetype: index="_internal" source=*metrics.log group="per_sourcetype_thruput" | timechart avg(kbps) by series

eps by sourcetype: index="_internal" source=*metrics.log group="per_sourcetype_thruput" | timechart avg(eps) by series

If you wanted host or source, use group="per_source_thruput" or group="per_host_thruput" instead.