Why is Splunk service using Off-site (external) NTP queries?

[venkateshparank]

splunk service is using offsite (external) NTP queries.

ntp.domain.com is our internal NTP domain and we have configured the same on our server under /etc/ntp.conf

We could see that ntp configuration is being used in Splunk_TA_nix app under time.sh script and further verified I see it is configured with NTP internal domain as “ntp­­­­­­­­­­.domain.com” under /etc/ntp.conf.

But still splunk is using Offsite NTP queries. Any idea why splunk is using offsite NTP queries ?

[splunk@hostname bin]$ ls -lrt | grep -ir ntp
time.sh:if [ -f    /etc/ntp.conf ] ; then   

============================================

[splunk@hostname bin]$ cat /etc/ntp.conf | grep  ntp.domain.com

restrict -6 ::1 pool ntp.domain.com iburst 

[Tom_Lundie · Accepted solution]

Okay, I've re-read your question and I know what's going on here.

The time.sh script that you referenced is designed to echo a manual NTP query and then the server date. The ntp query is determined via the OS's NTP client's config files and you're right that it does attempt to use /etc/ntp.conf.

This is where the issue comes in: on line 31 we're attempting to parse the ntp.conf server directive. You have presumably commented out the original server and are now using a pool directive. This pool will not match the awk parameters so the script will fall-back to using the $DEFAULT_SERVER (defined on line 26) as per line 32.

This default server variable corresponds with 0.pool.ntp.org, which explains those fqdn names that you've been observing via tcpdump.

To address this, you'll need to modify time.sh to suit your needs. A quick fix could be to change line 31 to support the pool directive as follows:

		SERVER=$($AWK '^(server|pool) / {print $2; exit}' "$CONFIG")

To be fair, this is a Splunk supported add-on, and your ntp.conf is perfectly valid so you could also try and raise a support case to get this supported upstream.

[venkateshparank]

That's great @Tom_Lundie , Thank you for sharing details. This is helpful.

We have many servers in our environment but only few are having this issue. I identified that Splunk_TA_nix version which we have on these servers is very old, the serves which are having latest version doesn't have any issue.

There was slight difference in time.sh between 6.x and 8.x version. So will try to upgrade and validate. if still issue exists, will update the script as you mentioned.

Thanks again for the wonderful suggestion and accepted it as solution.

[venkateshparank]

We have RHEL 7

Linux  3.10.0-1160.83.1.el7.x86_64

We do not have chrony.conf on server:

cat /etc/chrony.conf
cat: /etc/chrony.conf: No such file or directory

[Tom_Lundie]

Hmm, it does seem like you are using ntpd then. Just to double-check, can you share the output of these commands:
systemctl is-active ntpd
systemctl is-active chronyd

If you definitely are using ntpd then make sure there are no ntpd configs that could be calling out to those external domain. Trace through any includefile references within /etc/ntp.conf.

If you are confident that your OS's NTP Client is not calling out to those external servers. Then the next step is to make sure you're only observing these call-outs when Splunk is running. If you determine this to be the case, then I think this could be down to an app on your Splunk instance.

Try the following command:
grep -re "gac\.edu" $SPLUNK_HOME/etc
grep -re "versadns\.com" $SPLUNK_HOME/etc

Where $SPLUNK_HOME is your /opt/splunk path or /opt/splunkforwarder path.

[venkateshparank]

@Tom_Lundie Here is the output

[root@server ~]# systemctl is-active ntpd
active
[root@server ~]# systemctl is-active chronyd
unknown
[root@server ~]# grep -re "gac\.edu" $SPLUNK_HOME/etc
[root@server ~]# grep -re "gac\.edu" $SPLUNK_HOME
[root@server ~]# grep -re "versadns\.com" $SPLUNK_HOME/etc
[root@server ~]# grep -re "versadns\.com" $SPLUNK_HOME
[root@server ~]#

[Tom_Lundie]

Just to make sure:

grep -re "versadns\.com" $SPLUNK_HOME/etc

Will only work if your environment has the SPLUNK_HOME variable set. I doubt your root user will in this context.

Is this Splunk Enterprise or Splunk UF?

Try:

grep -re "versadns\.com" /opt/splunk/etc
grep -re "gac\.edu" /opt/splunk/etc

(or wherever your Splunk instance is located).

If that doesn't have any hits, then I'm certain this won't be triggered by the Splunk service directly and will be more generally related to your OSs NTP client. Let me know how you get on!

[venkateshparank]

Yea @Tom_Lundie ,

We have environment variable set for SPLUNK_HOME and it is UF.

I have also tried by giving complete path instead of SPLUNK_HOME but still no results.

[Tom_Lundie]

AFAIK, Splunk does not have a built in NTP client and what you're seeing is a red herring.

What OS are you using? The reason I ask is that chrony is the default ntp client as of RHEL/CentOS 7 and uses a completely different config (/etc/chrony.conf).

I would check if you're using chrony (and if you are, then update your chrony config as appropriate).