Good evening all,
I just noticed an issue on my new search head cluster (6.2.1) where if I try to schedule a simple xml dashboard to deliver via PDF, the search head I'm on will crash. No errors thrown other than the one in the UI saying there may be a network issue or Splunk may be down. This was also killing other members of the cluster but after resynchronizing the replicated config, it only seems to happen to the search head I'm on at the time. Has anyone else experienced this or can test theirs for me?
I also have the same issue. I would also add that if it tries to replicate the change, it can cause all of the servers in the cluster to crash. I have updated the savedsearches.conf from the command line with the settings to send the pdf, which doesn't crash the cluster. I even used the deployer to push the setting out to all of search heads with no issue.
The moment I click save, from the GUI, that server's Splunk instance crashes and often all the members of the cluster.
Replying here because it says I cannot post comments on my own question.
No errors are thrown in splunkd.log, here's the last 5 events and then the service dies.
01-01-2015 12:56:02.334 -0500 WARN SearchOperator:kv - Missing FORMAT for: transform_name='Security_ID_as_src_nt_domain'
01-01-2015 12:56:02.334 -0500 WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='Security_ID_as_src_nt_domain'
01-01-2015 12:56:09.079 -0500 WARN CronScheduler - No suitable time found in the next 1051200 minutes
01-01-2015 12:56:09.084 -0500 WARN CronScheduler - No suitable time found in the next 1051200 minutes
01-01-2015 12:56:35.868 -0500 INFO TcpOutputProc - Connected to idx=10.96.4.32:9998
No splunk related info in dmesg either.
CPU load average less than .5 over ~30 gigs free memory on the system.
Running RHEL 6.5 on Supermicro hardware. 40 CPU cores 32 GB ram. 5 search heads, 3 in site1 2 in site 2, 4 indexers on same hardware 2 in each site + deployer, license master, cluster master, etc. Issue occurs on any of the clustered search heads.
I read in the release notes that PDF reporting of advanced xml dashboards is no longer supported but these are simple xml generated by the dashboard editor.