I meant to log in and update this topic with the solution we found.
In the app configuration/conf file we had multiple ldap servers listed in the hostname field separated by a semi colon. It seems at some point that became an invalid configuration and stopped working. Once we removed the 2nd server to only list a single hostname it started working again.
... View more
The Deployment Server represents 'home' to ONLY those nodes running a forwarder client (who can then 'phone home' to the deployment server). The deployment server should not be running a universal or heavy forwarder because you are running the core enterprise software, wherein, you are directing output (_internal index goodies) directly to the indexer (forwarding and indexing configuration). Both the forwarder and the core installation utilize the splunkd process, so cannot physically both be running on the same node. The deployment server is 'home' and the source of your forwarder inputs sent to your forwarding nodes. I hope this helps.
In Splunk search for the log entry above and take note of the 'source' and 'host' values. This will tell you which node is responsible for generating the event.
... View more
I've answered my own question I think. The solution basically involves modification of the user interface such that the user is required to input ALL possible options first, then pass this subset (via cascading search, as per 6.x dashboard examples) to another input which is then coded as the 'overlay' value. Here is the functioning panel
<panel>
<input type="time" token="time_token3">
<label>Select a Time Range</label>
<default>
<earliestTime>-30d@d</earliestTime>
<latestTime>now</latestTime>
</default>
</input>
<input type="multiselect" token="lab_param_token2" searchWhenChanged="true">
<label>Select Analyte(s)</label>
<choice value="*">All</choice>
<populatingSearch earliest="-6mon" latest="now" fieldForLabel="lab_param" fieldForValue="lab_param">`wtms_labdata` | fields lab_param | stats count by lab_param</populatingSearch>
<default>Alkalinity</default>
<valuePrefix>lab_param="</valuePrefix>
<valueSuffix>"</valueSuffix>
<delimiter> OR </delimiter>
</input>
<input type="multiselect" token="lab_param_token3" searchWhenChanged="true">
<label>Select Analyte - Right Axis</label>
<populatingSearch fieldForLabel="lab_param" fieldForValue="lab_param">`wtms_labdata` $lab_param_token2$ | fields lab_param | stats count by lab_param</populatingSearch>
</input>
<chart>
<title>Multiple Analytes by $lab_location_token$</title>
<searchString>`wtms_labdata` treatment_plant="$treatment_plant_token$" $lab_location_token$ $lab_param_token2$ | timechart span=1h values(result) by lab_param</searchString>
<earliestTime>$time_token3.earliest$</earliestTime>
<latestTime>$time_token3.latest$</latestTime>
<option name="wrap">true</option>
<option name="rowNumbers">false</option>
<option name="dataOverlayMode">none</option>
<option name="count">10</option>
<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
<option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
<option name="charting.axisTitleX.visibility">visible</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisTitleY2.visibility">visible</option>
<option name="charting.axisX.scale">linear</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.axisY2.enabled">true</option>
<option name="charting.axisY2.scale">inherit</option>
<option name="charting.chart">line</option>
<option name="charting.chart.nullValueMode">connect</option>
<option name="charting.chart.sliceCollapsingThreshold">0.01</option>
<option name="charting.chart.stackMode">default</option>
<option name="charting.chart.style">shiny</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">0</option>
<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
<option name="charting.legend.placement">right</option>
<option name="charting.chart.overlayFields">$lab_param_token3$</option>
</chart>
</panel>
... View more
Thank you for your help and suggestions, guys. The problem has finally been solved. I used chmod 777 to that file and everything worked smoothly. I've checked the files' permission and splunk user was able to read that file. I didn't know how it ended up like an error when the forwarder tried to send the data though.
Best Regards,
Vincent
... View more
You may want to interrogate the splunk indexer's contributions to the _internal index as a timechart by SOURCE. The difference in log events by time should correspond to to your hourly CPU temper tantrum. Hopefully you can see a periodic difference in the number of events by source, which may help you identify events that only occur in this span.
Do you have any batch operations indexing data every hour....maybe being directed to only one indexer instead of being load-balanced?
... View more
In my circumstance, within the /var/run/splunk dir, I had NO .pid file. Otherwise my error and circumstance were identical.
To fix the issue, previous 'session' files were copied to another folder I called 'old_sessions'. Upon restart, with the session data cleared, the pid file regenerated properly and everything looks fine.
... View more