Installation

Why does Splunk keeps crashing after upgrade to Splunk 7.1?

Explorer

-bash-4.1$ cat crash-2018-05-22-13:02:27.log
(Out of file descriptors!)
[build 2e75b3406c5b] 2018-05-22 13:02:27

This is the error that I am seeing

Search peer (server_name) has the following message: Bundle Replication: Problem replicating config (bundle) to search peer ' (Server IP):8089 ', HTTP response code 409 (HTTP/1.1 409 Conflict). Failed to untar the bundle="/opt/splunk/var/run/searchpeers/(server_name)-1527015758.bundle". This could be due Search Head attempting to upload the same bundle again after a timeout. Check for sendRcvTimeout message in splund.log, consider increasing it.

And this is the crash log:
File descriptors open:

0: /opt/splunk/var/log/splunk/crash-2018-05-22-13:02:27.log
1: /opt/splunk/var/log/splunk/splunkd_stdout.log
2: /opt/splunk/var/log/splunk/splunkd_stderr.log
3: /opt/splunk/var/log/splunk/splunkd.log
4: socket:[27180755]
5: socket:[27180756]
6: socket:[27180757]
7: socket:[27180761]
8: [eventpoll]
9: socket:[27180763]
10: [eventfd]
11: pipe:[27180765]
12: /opt/splunk/var/log/splunk/audit.log
13: /opt/splunk/var/log/splunk/license_usage.log
14: [eventfd]
15: [eventpoll]
16: /opt/splunk/share/splunk/mbtiles/splunk-tiles.mbtiles
17: [eventfd]
18: socket:[27191160]
19: [eventpoll]
20: [eventpoll]
21: [eventfd]
22: [eventpoll]
23: [eventfd]
24: /opt/splunk/var/log/splunk/conf.log
25: [eventfd]
26: [eventpoll]
27: [eventfd]
28: [eventpoll]
29: [eventfd]
30: pipe:[27180848]
31: [eventpoll]
32: [eventfd]
33: /opt/splunk/var/log/splunk/mongod.log
34: /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/btree_index.dat
35: [eventfd]
36: socket:[27180852]
37: [eventpoll]
38: /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/btree_records.dat
39: [eventfd]
40: [eventfd]
41: [eventpoll]
42: [eventfd]
43: [eventpoll]
44: [eventfd]
45: [eventfd]
46: [eventpoll]
47: [eventpoll]
48: [eventpoll]
49: [eventfd]
50: [eventfd]
51: [eventpoll]
52: [eventpoll]
53: [eventfd]
54: [eventpoll]
55: [eventfd]
56: pipe:[27180877]
57: socket:[27196966]
58: pipe:[27180878]
59: socket:[27183669]
60: socket:[27180902]
61: /opt/splunk/var/log/splunk/metrics.log
62: /opt/splunk/var/lib/splunk/persistentstorage/fschangemanager_state
63: [eventpoll]
64: [eventfd]
65: [eventpoll]
66: [eventfd]
67: [eventpoll]
68: [eventfd]
69: [eventpoll]
70: [eventfd]
71: socket:[27191201]
72: [eventpoll]
73: [eventpoll]
74: socket:[27197043]
75: pipe:[27181879]
76: pipe:[27181910]
77: /opt/splunk/share/splunk/mbtiles/splunk-tiles.mbtiles
78: socket:[27183769]
79: socket:[27183773]
80: socket:[27183625]
81: socket:[27210557]
82: [eventfd]
83: [eventfd]
84: [eventpoll]
85: [eventfd]
86: /opt/splunk/var/log/introspection/disk_objects.log
87: [eventfd]
88: [eventpoll]
89: pipe:[27181021]
90: [eventfd]
91: [eventpoll]
92: /opt/splunk/var/log/splunk/health.log
93: /opt/splunk/var/log/splunk/splunkd_access.log
94: [eventpoll]
95: pipe:[27181880]
96: [eventpoll]
97: socket:[27205883]
98: [eventfd]
99: pipe:[27181911]
100: /opt/splunk/var/log/introspection/http_event_collector_metrics.log
101: socket:[27187915]
102: socket:[27183631]
103: [eventfd]
104: socket:[27183627]
105: socket:[27191162]
106: /opt/splunk/var/log/splunk/splunkd_ui_access.log
107: [eventfd]
108: [eventpoll]
109: socket:[27191244]
110: socket:[27212042]
111: [eventfd]
112: socket:[27183629]
113: [eventpoll]
114: socket:[27200480]
115: socket:[27216864]
116: /opt/splunk/var/log/introspection/kvstore.log
117: /opt/splunk/var/log/splunk/splunkd_ui_access.log
118: socket:[27191203]
119: /opt/splunk/var/log/introspection/resource_usage.log
120: /opt/splunk/var/log/splunk/scheduler.log
121: pipe:[27183400]
122: pipe:[27183400]
123: [eventpoll]
124: [eventfd]
125: /opt/splunk/var/log/splunk/remote_searches.log
126: socket:[27191246]
127: [eventpoll]
128: socket:[27204214]
129: socket:[27187283]
130: socket:[27196968]
131: socket:[27200482]
132: socket:[27187917]
133: socket:[27189987]
134: socket:[27189988]
135: socket:[27189989]
136: socket:[27189990]
137: /opt/splunk/var/log/splunk/splunkd_access.log
138: /etc/cma.d/lpc.conf
139: [eventfd]
140: socket:[27216866]
141: [eventpoll]
142: socket:[27204385]
143: socket:[27204216]
144: socket:[27222271]
145: /opt/splunk/var/log/splunk/audit.log
146: socket:[27222191]
147: socket:[27217115]
148: socket:[27197045]
149: socket:[27211889]
150: /opt/splunk/var/log/splunk/scheduler.log
151: socket:[27204387]
153: socket:[27222273]
157: socket:[27205754]
158: socket:[27205972]
159: socket:[27205756]
160: socket:[27217117]
163: socket:[27205448]
166: socket:[27205510]
167: socket:[27205450]
169: socket:[27205555]
170: socket:[27205512]
171: socket:[27205618]
172: socket:[27205557]
173: socket:[27205885]
174: socket:[27205620]
175: socket:[27206050]
176: socket:[27205974]
178: socket:[27206027]
179: socket:[27206112]
180: socket:[27206052]
181: socket:[27210559]
182: socket:[27206114]
188: socket:[27211891]
189: socket:[27212044]
(Total 175)
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 10152 running under UID 18002.
 Crashing thread: GenerationGrabberThread
 Registers:
    RIP:  [0x00007F8BF2848495] gsignal + 53 (libc.so.6 + 0x32495)
    RDI:  [0x00000000000027A8]
    RSI:  [0x0000000000004390]
    RBP:  [0x00007F8BF5F21D80]
    RSP:  [0x00007F8BD37FD538]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F8BF3DE0000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0x00000000000003F8]
    R9:  [0xFEFEFEFEFEFEFEFF]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x00007F8BF5E5DFAF]
    R13:  [0x00007F8BF6013600]
    R14:  [0x00007F8BD37FDDA0]
    R15:  [0x00007F8BD37FD840]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
Linux / (servernamehere) / 2.6.32-696.28.1.el6.x86_64 / #1 SMP Thu Apr 26 04:27:41 EDT 2018 / x86_64
glibc version: 2.12
glibc release: stable
Last errno: 23
Threads running: 76
Runtime: 471.198094s
argv: [splunkd -p 8089 start]
Regex JIT disabled due to SELinux

using CLOCK_MONOTONIC
Thread: "GenerationGrabberThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f8be9613090:
00000000 00 f7 7f d3 8b 7f 00 00 |........|
00000008

x86 CPUID registers:
         0: 0000000D 756E6547 6C65746E 49656E69
         1: 000206D2 09040800 9E982203 1F8BFBFF
         2: 76036301 00F0B5FF 00000000 00C10000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000000 00000000 00000000 00000000
         6: 00000077 00000002 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300401 0000007F 00000000 00000000
         B: 00000000 00000000 000000CD 00000009
         C: 00000000 00000000 00000000 00000000
         😧 00000000 00000000 00000000 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 65746E49 2952286C 6F655820 2952286E
  80000003: 55504320 2D354520 37383632 33762057
  80000004: 33204020 4730312E 00007A48 00000000
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...
-bash-4.1$ 
Labels (1)
1 Solution

Explorer

This was resolved when we upgraded Splunk 7.1.1.
They (Splunk) patched that in the next or latest version.

Hope that helps.

View solution in original post

0 Karma

Explorer

Btw, I got 2 of these crashes (Thread: "GenerationGrabberThread") on Splunk 7.2.7...

0 Karma

Explorer

This was resolved when we upgraded Splunk 7.1.1.
They (Splunk) patched that in the next or latest version.

Hope that helps.

View solution in original post

0 Karma

Path Finder

Was this resolved? I am getting the same message "Bundle Replication: Problem replicating config (bundle) to search peer ".

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!