AIX splunk starts successfully but doesn't actually start. This is new installation. Looks like it starts "nodaemon" ... and stale PID file at the beginning of start???
[cb-nimapp1:/opt/splunkforwarder/bin]./splunk start
splunkd 16187888 was not running.
Removing stale pid file... done.
Splunk> Winning the War on Error
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.1.1-08187535c166-AIX-powerpc-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
0513-059 The splunkd Subsystem has been started. Subsystem PID is 19071306.
Done
[cb-nimapp1:/opt/splunkforwarder/bin]ps -ef |grep splunk
root 15991280 19071306 0 11:05:20 - 0:00 /bin/sh /opt/splunkforwarder/bin/pid_check.sh conf-mutator 16187888
root 19071306 6554088 3 11:05:18 - 0:00 splunkd --nodaemon -p 8089 _internal_exec_splunkd
[cb-nimapp1:/opt/splunkforwarder/bin]ps -ef |grep splunk
root 15008220 19071308 0 11:05:25 - 0:00 [splunkd pid=19071308] splunkd --nodaemon -p 8089 _internal_exec_splunkd [process-runner]
root 19071308 6554088 15 11:05:23 - 0:00 splunkd --nodaemon -p 8089 _internal_exec_splunkd
[cb-nimapp1:/opt/splunkforwarder/bin]ps -ef |grep splunk
[cb-nimapp1:/opt/splunkforwarder/bin]
from log I just found, but ulmits are adequate:
01-13-2021 12:05:40.548 -0500 WARN ulimit - A system resource limit on this machine is below the minimum recommended value: system_resource = Data segment size (ulimit -d); current_limit = 134217728; recommended_minimum_value = 536870912. Change the operating system resource limits to meet the minimum recommended values for Splunk Enterprise.
01-13-2021 12:05:40.548 -0500 WARN ulimit - Splunk might not work because a small resident memory size limit has been set.
[cb-nimapp1:/opt/splunkforwarder/var/log/splunk]ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 636870912
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 8192
threads(per process) unlimited
processes(per user) unlimited