We did rebuild existing server that hosted LM and DMC. I did install latest splunk on the rebuilt server. Copied configuration settings from old build. Now when I try to turn  on splunk service, the service does start up fine but when I try accessing it from UI, I see below mentioned errors logged in splunkd.log: Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:dr_indexer_cluster_group] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://splunk-idx-cm.trgtm.ostravam.corp.tels tra.com:8089/services/indexer_discovery http_code=502 http_response="Error connecting: SSL not configured on client"] I did decrypt and verified that the pass4symmkey is same on CM and the new server.  > SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 err or:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt SSL context could not be created - error in cert or password is wrong  HTTPServer - SSL will not be enabled Not sure what this error is all about.   ... View more

I have created a metrics dashboard in which I have configured column chart. By default scale used is "Linear", this had issues as lower values were not plotted. As a fix configured the metrics to use "Log" scale, which did meet requirements. But the issue now is that when the count value spans around 1,3,7,14 all these values are maxed at 1. It's weird because expectation is that the plotting would be 0,1,10,100 and the values should have been plotted accordingly. Not sharing query as this is purely based on the count value. Would someone know if this is expected behavior or is there some config that should be tweaked which i might have missed. Metrics dashboard config mentioned below if it helps.: <row> <panel> <title>Metrics Panel</title> <chart> <search> <query>index=abcx "logData.logType"=INFO | timechart span=$slice$ count(feedName) </query> <earliest>$time_range$</earliest> <latest>now</latest> </search> <option name="charting.chart">column</option> <option name="charting.axisTitleX.text">Timestamp</option> <option name="charting.axisY.scale">log</option> <option name="charting.chart.showDataLabels">all</option> </chart> </panel> </row> ... View more

Need some help in understanding how the _time, timestamp default fields are extracted. Raw event as mentioned below and the field values extracted for respective event is as mentioned below. As can clearly be seen  I dont see anything that could relate to the value extracted in _time field. Any pointer related to this would be much helpful. Fields extracted: @timestamp                                                    |      _time                             |  timestamp 2020-06-22T15:17:34.892576+00:00 | 2020-06-17 17:54:50 | 2020-06-23 01:17:34.888 Raw event: ========= {"docker":{"container_id":"c0cb3bd3563f5f01133bcc496479b77b6c72bf898f24612ad7634b50a1749301"},"test":{"container_name":"anything","namespace_name":"test10-project","pod_name":"anything-1-w44fj","pod_id":"9289218b-b1cc-11ea-abcd-005056a44ead","labels":{"app":"anything","deployment":"anything-1","deploymentconfig":"anything"},"host":"ost-clb-osp-app-c02.linux.ostravam.corp.telstra.com","master_url":"https://test.default.svc.cluster.local","namespace_id":"0fbe0d11-cade-11e9-a562-005056a44ead"},"message":"2020-06-23 01:17:34.888 DEBUG --- [nio-8090-exec-5] o.s.web.servlet.DispatcherServlet : GET \"/healthcheck\", parameters={}\n","level":"info","hostname":"xxxxxxxxxxxxx","pipeline_metadata":{"collector":{"ipaddr4":"10.130.5.172","ipaddr6":"fe80::823:d3ff:fe3f:bf2d","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2020-06-22T15:17:35.076698+00:00","version":"0.12.43 1.6.0"}},"@timestamp":"2020-06-22T15:17:34.892576+00:00","viaq_index_name":"project.test10-project.0fbe0d11-cade-11e9-a562-005056a44ead.2020.06.22","viaq_msg_id":"YzY0NWI1ZGItMjc5Ni00YWI2LWI4OWUtMWZkODU1NTRlNjdj","forwarded_by":"standalone-fluentd-splunk.openshift-logging.svc.cluster.local","source_component":"testsource"} ... View more