Search Head Clustering - Search heads crash when trying to schedule PDF from simple xml dashboard

[dolivasoh]

Good evening all,

I just noticed an issue on my new search head cluster (6.2.1) where if I try to schedule a simple xml dashboard to deliver via PDF, the search head I'm on will crash. No errors thrown other than the one in the UI saying there may be a network issue or Splunk may be down. This was also killing other members of the cluster but after resynchronizing the replicated config, it only seems to happen to the search head I'm on at the time. Has anyone else experienced this or can test theirs for me?

[bpaul_splunk]

There was a bug (SPL-93913) in the early 6.2 versions of Splunk which prevented the sending of PDF reports in a search head cluster. This was fixed in version 6.2.2.

[dandaily]

I also have the same issue. I would also add that if it tries to replicate the change, it can cause all of the servers in the cluster to crash. I have updated the savedsearches.conf from the command line with the settings to send the pdf, which doesn't crash the cluster. I even used the deployer to push the setting out to all of search heads with no issue.

The moment I click save, from the GUI, that server's Splunk instance crashes and often all the members of the cluster.

[jayannah]

1. Did you check splunkd.log in $SPLUNK_HOME/var/log/splunk directory? please splunkd.log events here at the time of crash happened.
2. If you are using linux, please issue the shell command 'dmesg' and see output if you find the information (put the splunk related info here)
3. Did you check the CPU and Memory usage of the system?
4. whats your system configuration?

[dolivasoh]

@jayannah

Replying here because it says I cannot post comments on my own question.

No errors are thrown in splunkd.log, here's the last 5 events and then the service dies.

01-01-2015 12:56:02.334 -0500 WARN SearchOperator:kv - Missing FORMAT for: transform_name='Security_ID_as_src_nt_domain'
01-01-2015 12:56:02.334 -0500 WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='Security_ID_as_src_nt_domain'
01-01-2015 12:56:09.079 -0500 WARN CronScheduler - No suitable time found in the next 1051200 minutes
01-01-2015 12:56:09.084 -0500 WARN CronScheduler - No suitable time found in the next 1051200 minutes
01-01-2015 12:56:35.868 -0500 INFO TcpOutputProc - Connected to idx=10.96.4.32:9998

No splunk related info in dmesg either.

CPU load average less than .5 over ~30 gigs free memory on the system.

Running RHEL 6.5 on Supermicro hardware. 40 CPU cores 32 GB ram. 5 search heads, 3 in site1 2 in site 2, 4 indexers on same hardware 2 in each site + deployer, license master, cluster master, etc. Issue occurs on any of the clustered search heads.

I read in the release notes that PDF reporting of advanced xml dashboards is no longer supported but these are simple xml generated by the dashboard editor.