Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Why is Splunk crashing every 5 seconds on a single host with error "WARN Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable"?

ericmck2000
Explorer
‎11-01-2016 02:08 PM

So, I pushed Splunk out to our entire non-production AIX farm. It is working on 50 hosts just fine, but, fails on a single host.

From splunkd.log, the only help I found was 

11-01-2016 15:45:04.925 -0500 WARN  Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 35 threads active

I see this line too, but, I assume it is because the forward has not checked in with the deployment server yet..

11-01-2016 15:44:52.210 -0500 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Here is a crash dump.

[build 59c8927def0f] 2016-11-01 15:53:20
Received fatal signal 6 (IOT/Abort trap).
 Cause:
   Unknown signal origin (si_code=9).
 Crashing thread: HTTPDispatch
 Registers:
    IAR:  [0x09000000004A0F14] ?
    MSR:  [0xA00000000200D032]
    R0:  [0xFFFFFFFFFFFFFFFF]
    R1:  [0x0000000111ABF120]
    R2:  [0xFFFFFFFFFFFFFFFF]
    R3:  [0x0000000000000000]
    R4:  [0xFFFFFFFFFFFFFFFF]
    R5:  [0xFFFFFFFFFFFFFFFF]
    R6:  [0xFFFFFFFFFFFFFFFF]
    R7:  [0xFFFFFFFFFFFFFFFF]
    R8:  [0xFFFFFFFFFFFFFFFF]
    R9:  [0xFFFFFFFFFFFFFFFF]
    R10:  [0xFFFFFFFFFFFFFFFF]
    R11:  [0xFFFFFFFFFFFFFFFF]
    R12:  [0xFFFFFFFFFFFFFFFF]
    R13:  [0x0000000111AC8800]
    R14:  [0x0000000111ABF5E8]
    R15:  [0x0000000111ABF5E8]
    R16:  [0x0000000000000008]
    R17:  [0x00000001101311B8]
    R18:  [0x0000000000000000]
    R19:  [0x0000000000000001]
    R20:  [0x0900000000488E48]
    R21:  [0x0000000111AC0A78]
    R22:  [0x0900000000488E4C]
    R23:  [0xFFFFFFFFE8410028]
    R24:  [0x00000000FC000003]
    R25:  [0x00000000BEEEDEAD]
    R26:  [0x00000000BEEFDEAD]
    R27:  [0x09001000A0091938]
    R28:  [0x0000000000000006]
    R29:  [0x0000000000000006]
    R30:  [0x0000000000000000]
    R31:  [0x0000000000000203]
    CR:  [0x0000000000146000]
    XER:  [0x00000000FFFFFFFF]
    LR:  [0xFFFFFFFFFFFFFFFF]
    CTR:  [0xFFFFFFFF00000000]

 OS: AIX
 Arch: PowerPC

 Backtrace:
+++PARALLEL TOOLS CONSORTIUM LIGHTWEIGHT COREFILE FORMAT version 1.0
+++LCB 1.0 Tue Nov  1 15:53:20 2016 Generated by IBM AIX 7.1
#
+++ID Node 0 Process 34996350 Thread 3
***FAULT "SIGABRT - Abort"
+++STACK
pthread_kill : 0x000000d4
_p_raise : 0x00000044
raise : 0x00000048
abort : 0x000000c4
myabort__3stdFv : 0x0000000c
terminate__3stdFv : 0x0000006c
terminate__Fv : 0x0000000c
# At location 0x09000000006ed7a8 but procedure information unavailable.
thread_exception__FPCci : 0x00000084
checkPtResThrow__FPCci : 0x00000044
__ct__6ThreadFPCce : 0x000001d0
__ct__18WebuiStartupThreadFP20SingleRestHttpServerb : 0x00000030
afterStarted__20SingleRestHttpServerFv : 0x00000094
afterStarted__Q2_20SingleRestHttpServer7hserverFv : 0x00000018
callback__Q2_17SplunkdHttpServer11set_startedFv : 0x00000044
when_expired__14OneShotTimeoutFRUL : 0x00000040
runExpiredTimeouts__11TimeoutHeapFR7Timeval : 0x000001ac
run__9EventLoopFv : 0x000000a8
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__14TcpChannelLoopFv : 0x0000015c
go__17SplunkdHttpServerFv : 0x00000050
go__20SingleRestHttpServerFv : 0x00000018
main__18HTTPDispatchThreadFv : 0x00000210
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 3
#
+++ID Node 0 Process 34996350 Thread 1
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__10MainThreadFv : 0x00000114
run__10MainThreadFv : 0x00000030
main : 0x00003f84
---STACK
---ID Node 0 Process 34996350 Thread 1
#
+++ID Node 0 Process 34996350 Thread 2
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__19ProcessRunnerThreadFv : 0x00000058
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 2
#
+++ID Node 0 Process 34996350 Thread 4
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
main__23HttpClientPollingThreadFv : 0x000000fc
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 4
#
+++ID Node 0 Process 34996350 Thread 5
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
main__15PhonehomeThreadFv : 0x000004dc
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 5
#
+++ID Node 0 Process 34996350 Thread 6
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__15TcpOutboundLoopFv : 0x0000001c
main__25DRC_EventLoopRunnerThreadFv : 0x00000038
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 6
#
+++ID Node 0 Process 34996350 Thread 7
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x000004dc
_cond_wait : 0x000000c8
pthread_cond_wait : 0x0000019c
wait__16PthreadConditionFR14ConditionMutex : 0x0000007c
wait__16PthreadConditionFR20ScopedConditionMutex : 0x00000020
main__20CallbackRunnerThreadFv : 0x000000d8
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 7
#
+++ID Node 0 Process 34996350 Thread 8
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__14ShutdownThreadFv : 0x00000040
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 8
#
+++ID Node 0 Process 34996350 Thread 27
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__Q2_13ExecProcessor6threadFv : 0x00000064
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 27
#
+++ID Node 0 Process 34996350 Thread 10
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__15TcpOutboundLoopFv : 0x0000001c
main__28DistributedPeerMonitorThreadFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 10
#
+++ID Node 0 Process 34996350 Thread 11
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
sleep__20ConditionWaitTimeoutCFv : 0x0000002c
main__14DispatchReaperFv : 0x0000028c
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 11
#
+++ID Node 0 Process 34996350 Thread 12
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__17IdataDO_CollectorFv : 0x00000184
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 12
#
+++ID Node 0 Process 34996350 Thread 16
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 16
#
+++ID Node 0 Process 34996350 Thread 14
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 14
#
+++ID Node 0 Process 34996350 Thread 18
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__21FwdDataReceiverThreadFR15CowPipelineData : 0x00000084
createInputPipelineData__17TcpInputProcessorFR15CowPipelineData : 0x00000020
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 18
#
+++ID Node 0 Process 34996350 Thread 19
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__25SubordinateTcpChannelLoopFv : 0x00000064
main__22TcpInputReceiverThreadFv : 0x00000024
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 19
#
+++ID Node 0 Process 34996350 Thread 20
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
_do_run__19Base_TcpChannelLoopFv : 0x00000024
run__14TcpChannelLoopFv : 0x0000015c
main__17TcpListenerThreadFv : 0x00000044
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 20
#
+++ID Node 0 Process 34996350 Thread 21
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
main__14PQReaderThreadFv : 0x00000088
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 21
#
+++ID Node 0 Process 34996350 Thread 23
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__17UDPInputProcessorFR15CowPipelineData : 0x0000008c
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 23
#
+++ID Node 0 Process 34996350 Thread 24
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__17UDPInputProcessorFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 24
#
+++ID Node 0 Process 34996350 Thread 26
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
createInputPipelineData__13ExecProcessorFR15CowPipelineData : 0x00000068
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 26
#
+++ID Node 0 Process 34996350 Thread 39
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
main__18QueueServiceThreadFv : 0x00000074
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 39
#
+++ID Node 0 Process 34996350 Thread 29
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
createInputPipelineData__24FSChangeManagerProcessorFR15CowPipelineData : 0x00000048
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 29
#
+++ID Node 0 Process 34996350 Thread 30
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
main__15FSUpdaterThreadFv : 0x00000034
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 30
#
+++ID Node 0 Process 34996350 Thread 32
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x00000098
remove__21ProducerConsumerQueueFR17PipelineDataQueueUiUlRC20ConditionWaitTimeout : 0x0000005c
createPipelineData__19QueueInputProcessorFR15CowPipelineData : 0x00000058
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 32
#
+++ID Node 0 Process 34996350 Thread 34
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
createInputPipelineData__16ArchiveProcessorFR15CowPipelineData : 0x00000048
createPipelineData__14InputProcessorFR15CowPipelineData : 0x00000038
createMultiPipelineData__9ProcessorFP18PipelineDataVector : 0x00000070
main__8PipelineFv : 0x00000570
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 34
#
+++ID Node 0 Process 34996350 Thread 35
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
remove__21ProducerConsumerQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x00000044
main__16ArchiveProcessorFv : 0x000000d0
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 35
#
+++ID Node 0 Process 34996350 Thread 37
+++STACK
nsleep : 0x000000e4
sleep : 0x00000088
handleMisconfiguredLWF__18TcpOutputProcessorFv : 0x000000a4
execute__18TcpOutputProcessorFR15CowPipelineData : 0x00000120
executeMulti__9ProcessorFR18PipelineDataVectorP18PipelineDataVector : 0x000000b8
main__8PipelineFv : 0x00000634
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 37
#
+++ID Node 0 Process 34996350 Thread 38
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__Q2_11Distributed11EloopRunnerFv : 0x000000a4
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 38
#
+++ID Node 0 Process 34996350 Thread 40
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
run__11TailWatcherFv : 0x00000110
main__11TailManagerFv : 0x000001a8
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 40
#
+++ID Node 0 Process 34996350 Thread 41
+++STACK
poll__FPvUll : 0x00000024
run__9EventLoopFv : 0x0000019c
main__31HttpInputServerManagementThreadFv : 0x00000184
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 41
#
+++ID Node 0 Process 34996350 Thread 42
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x0000035c
_cond_wait : 0x000000c8
pthread_cond_timedwait : 0x00000200
wait__16PthreadConditionFR14ConditionMutexRC20ConditionWaitTimeout : 0x00000114
wait__16PthreadConditionFR20ScopedConditionMutexRC20ConditionWaitTimeout : 0x00000028
remove__15PersistentQueueFR15CowPipelineDataRC20ConditionWaitTimeout : 0x000000a4
main__Q2_14HttpInputQueue15queuePumpThreadFv : 0x00000078
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 42
#
+++ID Node 0 Process 34996350 Thread 43
+++STACK
malloc_y : 0x0000079c
malloc_common@AF104_87 : 0x0000004c
_Fancy_malloc__FUl : 0x0000000c
init_pres__FUl : 0x00000024
_Allocate__3stdH12StrReference_UlP12StrReference_P12StrReference : 0x0000002c
allocate__Q2_3std9allocatorXT12StrReference_FUlPCv : 0x00000020
insert__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FQ2_3std6_PtritXT12StrReferenceTlTP12StrReferenceTR12StrReferenceTP12StrReferenceTR12StrReference_UlRC12StrReference : 0x0000019c
insert__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FQ2_3std6_PtritXT12StrReferenceTlTP12StrReferenceTR12StrReferenceTP12StrReferenceTR12StrReference_RC12StrReference : 0x00000098
push_back__Q2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__FRC12StrReference : 0x0000007c
__adv__12ConfPathBaseFRC12StrReference : 0x00000020
shouldPruneTrie__19ConfPathBasedPrunerFRCQ2_3std6vectorXT12StrReferenceTQ2_3std9allocatorXT12StrReference__RC12ConfPathTrie : 0x00000348
shouldPruneTrieEx__24SimpleConfPathTriePrunerFRC9PruneInfo : 0x0000003c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x000000f8
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
pruneBase__12ConfPathTrieFR18ConfPathTriePrunerR9PruneInfo : 0x0000014c
prune__12ConfPathTrieFR18ConfPathTriePruner : 0x00000064
updateConfCacheWithTrie__13BaseConfCacheFR12ConfPathTrieb : 0x00000068
updateConfCache__15LegacyConfCacheFv : 0x0000019c
updateIfDirty__13BaseConfCacheFv : 0x0000007c
__ct__20ScopedIConfCacheLockFR10IConfCache : 0x000000a4
fillLayersImpl__13PropertyPagesFR13PropertiesMapR9StrVectorb : 0x0000006c
deferredWrite__13PropertyPagesFRC3StrT1RC13PropertiesMapUiRC9AtomTyper : 0x000003d4
saveStanzaToApp__13PropertyPagesFRC3StrT1RC13PropertiesMapbRC9AtomTyper : 0x000000cc
putLocalNS__13PropertyPagesFRC3StrPCcPC13PropertiesMapb : 0x000000b8
putLearned__13PropertyPagesFPCcPC13PropertiesMap : 0x00000034
Locked_createLearnedSourcetype__21FileClassifierManagerFRC3StrPC13PropertiesMap : 0x00000050
_getFileType__21FileClassifierManagerFP13PropertiesMapRC8PathnameR9StrVectorRbT4PC3StrUl : 0x0000121c
getFileType__21FileClassifierManagerFP13PropertiesMapRC8PathnamebPC3StrUl : 0x0000009c
classifySource__10TailReaderCFR15CowPipelineDataRC8PathnameR3StrN23b : 0x00000188
setupSourcetype__10TailReaderFR15WatchedTailFileRQ2_7Tailing10FileStatus : 0x00000240
readFile__10TailReaderFR15WatchedTailFile : 0x00000140
read__10TailReaderFP15WatchedTailFileP11TailWatcher : 0x00000264
handleFile__10TailReaderFP15WatchedTailFileP11TailWatcher : 0x00000d38
runTailingJob__10TailReaderFP10TailingJob : 0x000000b8
main__12ReaderThreadFv : 0x00000440
main__10TailReaderFv : 0x00000024
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 43
#
+++ID Node 0 Process 34996350 Thread 44
+++STACK
_event_wait : 0x00000344
_cond_wait_local : 0x000004dc
_cond_wait : 0x000000c8
pthread_cond_wait : 0x0000019c
wait__16PthreadConditionFR14ConditionMutex : 0x0000007c
wait__16PthreadConditionFR20ScopedConditionMutex : 0x00000020
main__12ReaderThreadFv : 0x0000025c
callMain__6ThreadFPv : 0x000000b4
_pthread_body : 0x000000f0
---STACK
---ID Node 0 Process 34996350 Thread 44
#There are more threads but pthread_getthrds_np returns <40>.
 AIX / axdevl5 / 1 / 7 / 00F90AFB4C00
Last errno: 12
Threads running: 34
Runtime: 5.749406s
argv: [splunkd --nodaemon -p 8089 _internal_exec_splunkd]
Thread: "HTTPDispatch", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @1117d14fc:
00000000  00 00 02 03                                       |....|
00000004

terminating...
0 Karma
Reply

kranthik
Explorer
‎07-10-2018 02:31 PM

Hi Ericmck2000,

What was the reason for the error that you faced? Because we are facing same error with UF.

Thank you,
Kranthi

0 Karma
Reply

ddrillic
Ultra Champion
‎11-01-2016 02:55 PM

I assume we are talking about the forwarders ...

A similar case at - Splunk Forwarder Crashes - pthread_create: Resource temporarily unavailable

@dwaddle speaks there about the number of threads but in your case we see - Threads running: 34.
Can you please post the output of ulimit -a? as quite often the servers don't have the proper ulimit configurations.

Reply

ericmck2000
Explorer
‎11-02-2016 06:34 AM 
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: virtual address space size: unlimited
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: data segment size: 134217728 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.744 -0500 WARN  ulimit - Splunk may not work due to small data segment limit!
11-01-2016 15:44:45.744 -0500 INFO  ulimit - Limit: resident memory size: 33554432 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.745 -0500 WARN  ulimit - Splunk may not work due to small resident memory size limit!
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: stack size: 33554432 bytes [hard maximum: 4294967296 bytes]
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: data file size: unlimited
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: open files: unlimited
11-01-2016 15:44:45.745 -0500 INFO  ulimit - Limit: cpu time: unlimited
0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 06:53 AM

A good discussion about forwarders' crashes on AIX at Why does Splunk 6.1.2 forwarder on an AIX 7.1 machine keep crashing?

Can you please post the output of ulimit -a?

0 Karma
Reply

ericmck2000
Explorer
‎11-02-2016 07:25 AM 
axdevl5 # ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     2097151
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user)  unlimited
0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 07:31 AM

Looks great!!! I would open a ticket with Support especially since you have the crash dump.

0 Karma
Reply

ericmck2000
Explorer
‎11-02-2016 07:32 AM

Well, ticket incoming.

0 Karma
Reply

ddrillic
Ultra Champion
‎11-02-2016 07:35 AM

Good luck - please keep us posted ; -)

0 Karma
Reply
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...