splunk uf takes 30-40 mins generating auth certificate on first run

masterofnone — Wed, 07 Feb 2024 16:58:18 GMT

A snippet from strace output seems to indicate that the 30-40 mins may be taken by the ssl certificate generating steps:

<<<snipped>>>>>
wait4(9855, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 9855

stat("/opt/splunkforwarder/etc/auth/server.pem", 0x7ffdec4c4580) = -1 ENOENT (No such file or directory)

clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f143df47e50) = 9857

wait4(9857,

< < < stuck here for 30-40 mins > > > >

0x7ffdec4c45f4, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---

wait4(9857, New certs have been generated in '/opt/splunkforwarder/etc/auth'.

[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 9857

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9857, si_uid=0, si_status=0, si_utime=11, si_stime=5} ---

Strangely, this is only happening on Linux on Azure. Using openssl, I am able to generate self-sign cert within seconds on the same machine.

Our Linux on premises (on vmware) does not experience this performance issue.

Any thoughts on what the issue may be? How to troubleshoot?

Thank you

topic splunk uf takes 30-40 mins generating auth certificate on first run in Getting Data In