Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Why is Splunk crashing every 5 seconds on a single host with error "WARN Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable"?

ericmck2000
Explorer
‎11-01-2016 02:08 PM

So, I pushed Splunk out to our entire non-production AIX farm. It is working on 50 hosts just fine, but, fails on a single host.

From splunkd.log, the only help I found was 

11-01-2016 15:45:04.925 -0500 WARN  Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 35 threads active

I see this line too, but, I assume it is because the forward has not checked in with the deployment server yet..

11-01-2016 15:44:52.210 -0500 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Here is a crash dump.

[build 59c8927def0f] 2016-11-01 15:53:20
Received fatal signal 6 (IOT/Abort trap).
 Cause:
   Unknown signal origin (si_code=9).
 Crashing thread: HTTPDispatch
 Registers:
    IAR:  [0x09000000004A0F14] ?
    MSR:  [0xA00000000200D032]
    R0:  [0xFFFFFFFFFFFFFFFF]
    R1:  [0x0000000111ABF120]
    R2:  [0xFFFFFFFFFFFFFFFF]
    R3:  [0x0000000000000000]
    R4:  [0xFFFFFFFFFFFFFFFF]
    R5:  [0xFFFFFFFFFFFFFFFF]
    R6:  [0xFFFFFFFFFFFFFFFF]
    R7:  [0xFFFFFFFFFFFFFFFF]
    R8:  [0xFFFFFFFFFFFFFFFF]
    R9:  [0xFFFFFFFFFFFFFFFF]
    R10:  [0xFFFFFFFFFFFFFFFF]
    R11:  [0xFFFFFFFFFFFFFFFF]
    R12:  [0xFFFFFFFFFFFFFFFF]
    R13:  [0x0000000111AC8800]
    R14:  [0x0000000111ABF5E8]
    R15:  [0x0000000111ABF5E8]
    R16:  [0x0000000000000008]
    R17:  [0x00000001101311B8]
    R18:  [0x0000000000000000]
    R19:  [0x0000000000000001]
    R20:  [0x0900000000488E48]
    R21:  [0x0000000111AC0A78]
    R22:  [0x0900000000488E4C]
    R23:  [0xFFFFFFFFE8410028]
    R24:  [0x00000000FC000003]
    R25:  [0x00000000BEEEDEAD]
    R26:  [0x00000000BEEFDEAD]
    R27:  [0x09001000A0091938]
    R28:  [0x0000000000000006]
    R29:  [0x0000000000000006]
    R30:  [0x0000000000000000]
    R31:  [0x0000000000000203]
    CR:  [0x0000000000146000]
    XER:  [0x00000000FFFFFFFF]
    LR:  [0xFFFFFFFFFFFFFFFF]
    CTR:  [0xFFFFFFFF00000000]

 OS: AIX
 Arch: PowerPC

 Backtrace:
+++PARALLEL TOOLS CONSORTIUM LIGHTWEIGHT COREFILE FORMAT version 1.0
+++LCB 1.0 Tue Nov  1 15:53:20 2016 Generated by IBM AIX 7.1
#
+++ID Node 0 Process 34996350 Thread 3
***FAULT "SIGABRT - Abort"
+++STACK
pthread_kill : 0x000000d4
_p_raise : 0x00000044
raise : 0x00000048
abort : 0x000000c4
myabort__3stdFv : 0x0000000c
terminate__3stdFv : 0x0000006c
terminate__Fv : 0x0000000c
# At location 0x09000000006ed7a8 but procedure information unavailable.
thread_exception__FPCci : 0x00000084
checkPtResThrow__FPCci : 0x00000044
__ct__6ThreadFPCce : 0x000001d0
__ct__18WebuiStartupThreadFP20SingleRestHttpServerb : 0x00000030
afterStarted__20SingleRestHttpServerFv : 0x00000094
afterStarted__Q2_20SingleRestHttpServer7hserverFv : 0x00000018
callback__Q2_17SplunkdHttpServer11set_startedFv : 0x00000044
when_expired__14OneShotTimeoutFRUL : 0x00000040
runExpiredTimeouts__11TimeoutHeapFR7Timeval : 0x000001ac
run__9EventLoopFv : 0x000000a8
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__14TcpChannelLoopFv : 0x0000015c
go__17SplunkdHttpServerFv : 0x00000050
go__20SingleRestHttpServerFv : 0x00000018
main__18HTTPDispatchThreadFv : 0x00000210
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 3
#
+++ID Node 0 Process 34996350 Thread 1
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__10MainThreadFv : 0x00000114
run__10MainThreadFv : 0x00000030
main : 0x00003f84
---STACK
---ID Node 0 Process 34996350 Thread 1
#
+++ID Node 0 Process 34996350 Thread 2
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__19ProcessRunnerThreadFv : 0x00000058
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 2
#
+++ID Node 0 Process 34996350 Thread 4
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
main__23HttpClientPollingThreadFv : 0x000000fc
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 4
#
+++ID Node 0 Process 34996350 Thread 5
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
main__15PhonehomeThreadFv : 0x000004dc
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 5
#
+++ID Node 0 Process 34996350 Thread 6
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__15TcpOutboundLoopFv : 0x0000001c
main__25DRC_EventLoopRunnerThreadFv : 0x00000038
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 6
#
+++ID Node 0 Process 34996350 Thread 7
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x000004dc
_cond_wait : 0x000000c8
pthread_cond_wait : 0x0000019c
wait__16PthreadConditionFR14ConditionMutex : 0x0000007c
wait__16PthreadConditionFR20ScopedConditionMutex : 0x00000020
main__20CallbackRunnerThreadFv : 0x000000d8
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 7
#
+++ID Node 0 Process 34996350 Thread 8
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__14ShutdownThreadFv : 0x00000040
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 8
#
+++ID Node 0 Process 34996350 Thread 27
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__Q2_13ExecProcessor6threadFv : 0x00000064
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 27
#
+++ID Node 0 Process 34996350 Thread 10
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__15TcpOutboundLoopFv : 0x0000001c
main__28DistributedPeerMonitorThreadFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 10
#
+++ID Node 0 Process 34996350 Thread 11
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
sleep__20ConditionWaitTimeoutCFv : 0x0000002c
main__14DispatchReaperFv : 0x0000028c
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 11
#
+++ID Node 0 Process 34996350 Thread 12
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__17IdataDO_CollectorFv : 0x00000184
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 12
#
+++ID Node 0 Process 34996350 Thread 16
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 16
#
+++ID Node 0 Process 34996350 Thread 14
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 14
#
+++ID Node 0 Process 34996350 Thread 18
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__21FwdDataReceiverThreadFR15CowPipelineData : 0x00000084
createInputPipelineData__17TcpInputProcessorFR15CowPipelineData : 0x00000020
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 18
#
+++ID Node 0 Process 34996350 Thread 19
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__25SubordinateTcpChannelLoopFv : 0x00000064
main__22TcpInputReceiverThreadFv : 0x00000024
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 19
#
+++ID Node 0 Process 34996350 Thread 20
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__14TcpChannelLoopFv : 0x0000015c
main__17TcpListenerThreadFv : 0x00000044
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 20
#
+++ID Node 0 Process 34996350 Thread 21
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
main__14PQReaderThreadFv : 0x00000088
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 21
#
+++ID Node 0 Process 34996350 Thread 23
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__17UDPInputProcessorFR15CowPipelineData : 0x0000008c
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 23
#
+++ID Node 0 Process 34996350 Thread 24
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__17UDPInputProcessorFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 24
#
+++ID Node 0 Process 34996350 Thread 26
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__13ExecProcessorFR15CowPipelineData : 0x00000068
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 26
#
+++ID Node 0 Process 34996350 Thread 39
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
main__18QueueServiceThreadFv : 0x00000074
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 39
#
+++ID Node 0 Process 34996350 Thread 29
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
createInputPipelineData__24FSChangeManagerProcessorFR15CowPipelineData : 0x00000048
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 29
#
+++ID Node 0 Process 34996350 Thread 30
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
main__15FSUpdaterThreadFv : 0x00000034
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 30
#
+++ID Node 0 Process 34996350 Thread 32
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 32
#
+++ID Node 0 Process 34996350 Thread 34
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
createInputPipelineData__16ArchiveProcessorFR15CowPipelineData : 0x00000048
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 34
#
+++ID Node 0 Process 34996350 Thread 35
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
main__16ArchiveProcessorFv : 0x000000d0
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 35
#
+++ID Node 0 Process 34996350 Thread 37
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
handleMisconfiguredLWF__18TcpOutputProcessorFv : 0x000000a4
execute__18TcpOutputProcessorFR15CowPipelineData : 0x00000120
executeMulti__9ProcessorFR18PipelineDataVectorP18PipelineDataVector : 0x000000b8
main__8PipelineFv : 0x00000634
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 37
#
+++ID Node 0 Process 34996350 Thread 38
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__Q2_11Distributed11EloopRunnerFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 38
#
+++ID Node 0 Process 34996350 Thread 40
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__11TailWatcherFv : 0x00000110
main__11TailManagerFv : 0x000001a8
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 40
#
+++ID Node 0 Process 34996350 Thread 41
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__31HttpInputServerManagementThreadFv : 0x00000184
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 41
#
+++ID Node 0 Process 34996350 Thread 42
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
main__Q2_14HttpInputQueue15queuePumpThreadFv : 0x00000078
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 42
#
+++ID Node 0 Process 34996350 Thread 43
+++STACK
malloc_y : 0x0000079c
malloc_common@AF104_87 : 0x0000004c
_Fancy_malloc__FUl : 0x0000000c
init_pres__FUl : 0x00000024
_Allocate__3stdH12StrReference_UlP12StrReference_P12StrReference : 0x0000002c
allocate__Q2_3std9allocatorXT12StrReference_FUlPCv : 0x00000020
insert__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FQ2_3std6_PtritXT12StrReferenceTlTP12StrReferenceTR12StrReferenceTP12StrReferenceTR12StrReference_UlRC12StrReference : 0x0000019c
insert__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FQ2_3std6_PtritXT12StrReferenceTlTP12StrReferenceTR12StrReferenceTP12StrReferenceTR12StrReference_RC12StrReference : 0x00000098
push_back__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FRC12StrReference : 0x0000007c
__adv__12ConfPathBaseFRC12StrReference : 0x00000020
shouldPruneTrie__19ConfPathBasedPrunerFRCQ2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__RC12ConfPathTrie : 0x00000348
shouldPruneTrieEx__24SimpleConfPathTriePrunerFRC9PruneInfo : 0x0000003c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x000000f8
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
prune__12ConfPathTrieFR18ConfPathTriePruner : 0x00000064
updateConfCacheWithTrie__13BaseConfCacheFR12ConfPathTrieb : 0x00000068
updateConfCache__15LegacyConfCacheFv : 0x0000019c
updateIfDirty__13BaseConfCacheFv : 0x0000007c
__ct__20ScopedIConfCacheLockFR10IConfCache : 0x000000a4
fillLayersImpl__13PropertyPagesFR13PropertiesMapR9StrVectorb : 0x0000006c
deferredWrite__13PropertyPagesFRC3StrT1RC13PropertiesMapUiRC9AtomTyper : 0x000003d4
saveStanzaToApp__13PropertyPagesFRC3StrT1RC13PropertiesMapbRC9AtomTyper : 0x000000cc
putLocalNS__13PropertyPagesFRC3StrPCcPC13PropertiesMapb : 0x000000b8
putLearned__13PropertyPagesFPCcPC13PropertiesMap : 0x00000034
Locked_createLearnedSourcetype__21FileClassifierManagerFRC3StrPC13PropertiesMap : 0x00000050
_getFileType__21FileClassifierManagerFP13PropertiesMapRC8PathnameR9StrVectorRbT4PC3StrUl : 0x0000121c
getFileType__21FileClassifierManagerFP13PropertiesMapRC8PathnamebPC3StrUl : 0x0000009c
classifySource__10TailReaderCFR15CowPipelineDataRC8PathnameR3StrN23b : 0x00000188
setupSourcetype__10TailReaderFR15WatchedTailFileRQ2_7Tailing10FileStatus : 0x00000240
readFile__10TailReaderFR15WatchedTailFile : 0x00000140
read__10TailReaderFP15WatchedTailFileP11TailWatcher : 0x00000264
handleFile__10TailReaderFP15WatchedTailFileP11TailWatcher : 0x00000d38
runTailingJob__10TailReaderFP10TailingJob : 0x000000b8
main__12ReaderThreadFv : 0x00000440
main__10TailReaderFv : 0x00000024
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 43
#
+++ID Node 0 Process 34996350 Thread 44
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x000004dc
_cond_wait : 0x000000c8
pthread_cond_wait : 0x0000019c
wait__16PthreadConditionFR14ConditionMutex : 0x0000007c
wait__16PthreadConditionFR20ScopedConditionMutex : 0x00000020
main__12ReaderThreadFv : 0x0000025c
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 44
#There are more threads but pthread_getthrds_np returns <40>.
 AIX / axdevl5 / 1 / 7 / 00F90AFB4C00
Last errno: 12
Threads running: 34
Runtime: 5.749406s
argv: [splunkd --nodaemon -p 8089 _internal_exec_splunkd]
Thread: "HTTPDispatch", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @1117d14fc:
00000000  00 00 02 03                                       |....|
00000004

terminating...
0 Karma
Reply

kranthik
Explorer
‎07-10-2018 02:31 PM

Hi Ericmck2000,

What was the reason for the error that you faced? Because we are facing same error with UF.

Thank you,
Kranthi

0 Karma
Reply

ddrillic
Ultra Champion
‎11-01-2016 02:55 PM

I assume we are talking about the forwarders ...

A similar case at - Splunk Forwarder Crashes - pthread_create: Resource temporarily unavailable

@dwaddle speaks there about the number of threads but in your case we see - Threads running: 34.
Can you please post the output of ulimit -a? as quite often the servers don't have the proper ulimit configurations.

Reply

ericmck2000
Explorer
‎11-02-2016 06:34 AM 
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: virtual address space size: unlimited
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: data segment size: 134217728 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.744 -0500 WARN  ulimit - Splunk may not work due to small data segment limit!
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: resident memory size: 33554432 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.745 -0500 WARN  ulimit - Splunk may not work due to small resident memory size limit!
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: stack size: 33554432 bytes [hard maximum: 4294967296 bytes]
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: data file size: unlimited
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: open files: unlimited
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: cpu time: unlimited
0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 06:53 AM

A good discussion about forwarders' crashes on AIX at Why does Splunk 6.1.2 forwarder on an AIX 7.1 machine keep crashing?

Can you please post the output of ulimit -a?

0 Karma
Reply

ericmck2000
Explorer
‎11-02-2016 07:25 AM 
axdevl5 # ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     2097151
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited
0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 07:31 AM

Looks great!!! I would open a ticket with Support especially since you have the crash dump.

0 Karma
Reply

ericmck2000
Explorer
‎11-02-2016 07:32 AM

Well, ticket incoming.

0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 07:35 AM

Good luck - please keep us posted ; -)

0 Karma
Reply
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...