Made a new splunk searchhead, all searches are failing.

[morinb]

My environment consists of 1 search head, 1 manager, and 3 indexers. I added another search head so that I can put enterprise security on it but when I run any search i get this error.  (only reason i did index=* was to show that ALL indexes are like this and no matter what I search this happens. What I'm the most confused about is why is the bottom portion (where the search results are) greyed out and I cant interact with it. 

Here's the last few lines from the search.log if more is required i can send more of the log. The log is just really long.

04-03-2024 18:00:38.937 INFO  SearchStatusEnforcer [11858 StatusEnforcerThread] - sid=1712181568.6, newState=BAD_INPUT_CANCEL, message=Search auto-canceled
04-03-2024 18:00:38.937 ERROR SearchStatusEnforcer [11858 StatusEnforcerThread] - SearchMessage orig_component=SearchStatusEnforcer sid=1712181568.6 message_key= message=Search auto-canceled
04-03-2024 18:00:38.937 INFO  SearchStatusEnforcer [11858 StatusEnforcerThread] - State changed to BAD_INPUT_CANCEL: Search auto-canceled
04-03-2024 18:00:38.945 INFO  TimelineCreator [11862 phase_1] - Commit timeline at cursor=1712168952.000000
04-03-2024 18:00:38.945 WARN  DispatchExecutor [11862 phase_1] - Execution status=CANCELLED: Search has been cancelled
04-03-2024 18:00:38.945 INFO  ReducePhaseExecutor [11862 phase_1] - Ending phase_1
04-03-2024 18:00:38.945 INFO  UserManager [11862 phase_1] - Unwound user context: b.morin -> NULL
04-03-2024 18:00:38.948 INFO  UserManager [11858 StatusEnforcerThread] - Unwound user context: b.morin -> NULL
04-03-2024 18:00:38.950 INFO  DispatchManager [11855 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='1712181568.6', username='b.morin')
04-03-2024 18:00:38.950 INFO  UserManager [11855 searchOrchestrator] - Unwound user context: b.morin -> NULL
04-03-2024 18:00:38.950 ERROR ScopedAliveProcessToken [11855 searchOrchestrator] - Failed to remove alive token file='/opt/splunk/var/run/splunk/dispatch/1712181568.6/alive.token'. No such file or directory
04-03-2024 18:00:38.950 INFO  SearchOrchestrator [11852 RunDispatch] - SearchOrchestrator is destructed.  sid=1712181568.6, eval_only=0
04-03-2024 18:00:38.952 INFO  UserManager [11861 SearchResultExecutorThread] - Unwound user context: b.morin -> NULL
04-03-2024 18:00:38.961 INFO  SearchStatusEnforcer [11852 RunDispatch] - SearchStatusEnforcer is already terminated
04-03-2024 18:00:38.961 INFO  UserManager [11852 RunDispatch] - Unwound user context: b.morin -> NULL
04-03-2024 18:00:38.961 INFO  LookupDataProvider [11852 RunDispatch] - Clearing out lookup shared provider map
04-03-2024 18:00:38.962 INFO  dispatchRunner [10908 MainThread] - RunDispatch is done: sid=1712181568.6, exit=0

[morinb · Accepted solution]

I changed ulimits to 64000
ulimit -n 64000

and I realized I had THP still enabled on the CentOS 7 VM it is on so i disabled it and rebooted the VM.
vim /etc/default/grub 
added transparent_hugepage = never
echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

I also enabled auto start for splunk.
/opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1

I then rebooted.

reboot

After doing that and the reboot the searches started to work correctly and stopped erroring out. Hopefully this thread can help someone else who has this weird problem!

[morinb]

New discovery if i refresh the page while the search is running you can see the search working but its still grey until the search finishes.

[morinb]

After some more poking around it seems like the searches are NOT failing at all. They are running and completing but it just instantly times out when loading the search. If I go to Activity > Jobs and click any search I ran it gives me the results and everything works as expected. Its just the initial search that is causing this error. If I click a job that is not finished and still running it gives the same error but shows some results with the greyed out bottom portion (see screenshot above). It also says the job has failed in the activity while its running but once it finishes it changes to done.