×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question
morinb
Explorer
Member since: ‎04-03-2024
‎04-05-2024
Community Statistics
Posts 4
Solutions 1
Karma Given 0
Karma Received 0
Member Since ‎04-03-2024
View All

Re: Made a new splunk searchhead, all searches are...

by in Splunk Search
‎04-04-2024 02:06 PM
‎04-04-2024 02:06 PM
I changed ulimits to 64000 ulimit -n 64000 and I realized I had THP still enabled on the CentOS 7 VM it is on so i disabled it and rebooted the VM. vim /etc/default/grub  added transparent_hugepage = never echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag I also enabled auto start for splunk. /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1 I then rebooted. reboot After doing that and the reboot the searches started to work correctly and stopped erroring out. Hopefully this thread can help someone else who has this weird problem! ... View more

Re: Made a new splunk searchhead, all searches are...

by in Splunk Search
‎04-04-2024 10:12 AM
‎04-04-2024 10:12 AM
New discovery if i refresh the page while the search is running you can see the search working but its still grey until the search finishes. ... View more

Re: Made a new splunk searchhead, all searches are...

by in Splunk Search
‎04-04-2024 09:55 AM
‎04-04-2024 09:55 AM
After some more poking around it seems like the searches are NOT failing at all. They are running and completing but it just instantly times out when loading the search. If I go to Activity > Jobs and click any search I ran it gives me the results and everything works as expected. Its just the initial search that is causing this error. If I click a job that is not finished and still running it gives the same error but shows some results with the greyed out bottom portion (see screenshot above). It also says the job has failed in the activity while its running but once it finishes it changes to done.    ... View more

Made a new splunk searchhead, all searches are fai...

by in Splunk Search
‎04-03-2024 03:08 PM
‎04-03-2024 03:08 PM
My environment consists of 1 search head, 1 manager, and 3 indexers. I added another search head so that I can put enterprise security on it but when I run any search i get this error.  (only reason i did index=* was to show that ALL indexes are like this and no matter what I search this happens. What I'm the most confused about is why is the bottom portion (where the search results are) greyed out and I cant interact with it.  Here's the last few lines from the search.log if more is required i can send more of the log. The log is just really long. 04-03-2024 18:00:38.937 INFO SearchStatusEnforcer [11858 StatusEnforcerThread] - sid=1712181568.6, newState=BAD_INPUT_CANCEL, message=Search auto-canceled 04-03-2024 18:00:38.937 ERROR SearchStatusEnforcer [11858 StatusEnforcerThread] - SearchMessage orig_component=SearchStatusEnforcer sid=1712181568.6 message_key= message=Search auto-canceled 04-03-2024 18:00:38.937 INFO SearchStatusEnforcer [11858 StatusEnforcerThread] - State changed to BAD_INPUT_CANCEL: Search auto-canceled 04-03-2024 18:00:38.945 INFO TimelineCreator [11862 phase_1] - Commit timeline at cursor=1712168952.000000 04-03-2024 18:00:38.945 WARN DispatchExecutor [11862 phase_1] - Execution status=CANCELLED: Search has been cancelled 04-03-2024 18:00:38.945 INFO ReducePhaseExecutor [11862 phase_1] - Ending phase_1 04-03-2024 18:00:38.945 INFO UserManager [11862 phase_1] - Unwound user context: b.morin -> NULL 04-03-2024 18:00:38.948 INFO UserManager [11858 StatusEnforcerThread] - Unwound user context: b.morin -> NULL 04-03-2024 18:00:38.950 INFO DispatchManager [11855 searchOrchestrator] - DispatchManager::dispatchHasFinished(id='1712181568.6', username='b.morin') 04-03-2024 18:00:38.950 INFO UserManager [11855 searchOrchestrator] - Unwound user context: b.morin -> NULL 04-03-2024 18:00:38.950 ERROR ScopedAliveProcessToken [11855 searchOrchestrator] - Failed to remove alive token file='/opt/splunk/var/run/splunk/dispatch/1712181568.6/alive.token'. No such file or directory 04-03-2024 18:00:38.950 INFO SearchOrchestrator [11852 RunDispatch] - SearchOrchestrator is destructed. sid=1712181568.6, eval_only=0 04-03-2024 18:00:38.952 INFO UserManager [11861 SearchResultExecutorThread] - Unwound user context: b.morin -> NULL 04-03-2024 18:00:38.961 INFO SearchStatusEnforcer [11852 RunDispatch] - SearchStatusEnforcer is already terminated 04-03-2024 18:00:38.961 INFO UserManager [11852 RunDispatch] - Unwound user context: b.morin -> NULL 04-03-2024 18:00:38.961 INFO LookupDataProvider [11852 RunDispatch] - Clearing out lookup shared provider map 04-03-2024 18:00:38.962 INFO dispatchRunner [10908 MainThread] - RunDispatch is done: sid=1712181568.6, exit=0   ... View more
Labels
Contact Me
Online Status
Offline
Date Last Visited
‎04-05-2024 02:14 AM