Mismatch between client system time zone with splunk agent- Will it cause logs to stop onboarding?

[Dinesh1]

Hello Experts,

In my client environment, we have set of AWS EC2 instances have Splunk agent installed and sending logs to deployment server. But recently I'm facing issue for few newly build UNIX AWS EC2 instances are not sending logs to deployment server (Via Unix TA). But its reporting to Deployment server forwarder management.

On further troubleshooting found that Unix AWS EC2 instance local system time is on UTC and my Deployment server is on MYT, Will it cause the issue and stop logs onboarding?

If, I change/add the particular EC2 instance Splunk_UNIX_TA apps/ props.conf either local or default stanza will resolve the issue? (We have option to change that machine local time settings but, if client does not accept to change time settings what is next?)

Any advice?

Thanks in advance.

[richgalloway]

It is very common for the Splunk server and agent to have different time zone settings so I doubt that has any effect on logging.

First, however, agents should be sending logs to indexers, not to a deployment server.  The DS should only be used to send configurations (apps) to agents and not to collect data for indexing.

Each agent should have an outputs.conf file that specifies the indexer names and ports to use for indexing data.  If that file is incorrect then the agent will be unable to send logs to the server.  Start the troubleshooting there.  If you can see the agent's internal logs on the server then the outputs.conf settings are working.

Next, verify the inputs are enabled and that the agent has read access to them.  To help ensure the data is onboarded properly, ensure there is a  props.conf file including the TZ setting for each sourcetype onboarded.

Finally, if there is a time zone difference between the server and agent then it's possible the logs were received, but have been indexed with a timestamp in the future, making it difficult to find them.  Try searching with the "latest=+1y" option to see if that helps.