Thanks for the reply. What you said makes sense. I have a concern though. I looked at one of our typical UF installs and I verified that there already is a ../etc/system/local/server.conf. Since I'm the admin and normally do all UF deployments, I know that this file was automatically generated when the forwarder was installed. As you suspected, it contains the hostname of the server.  Interestingly, the ../etc/system/default/server.conf contains serverName = $HOSTNAME. So the serverName field is populated when the UF is installed and a local/server.conf is created. The issue I have is that this would have to be overridden after a UF is installed. This is possible, but seems like it shouldn't be necessary. Thoughts? ... View more

I read thru the Splunk docs and it seems like a UF with customized inputs.conf and output.conf file should work. If the two enterprise servers are defined in the output.conf file, than we can use the inputs conf stanza to customize the destination where various log files are sent. Just wanted to confirm before accepting this solution. _TCP_ROUTING = <comma-separated list> * A comma-separated list of tcpout group names. * This setting lets you selectively forward data to one or more specific indexers. * Specify the tcpout group that the forwarder uses when forwarding the data. The tcpout group names are defined in outputs.conf with [tcpout:<tcpout_group_name>]. * To forward data to all tcpout group names that have been defined in outputs.conf, set to '*' (asterisk). * To forward data from the "_internal" index, you must explicitly set '_TCP_ROUTING' to either "*" or a specific splunktcp target group. * Default: The groups specified in 'defaultGroup' in [tcpout] stanza in the outputs.conf file    ... View more

Thanks for the reply. Regarding forwarding to two output groups, is there some doc that describes this in detail? In any case, given that we have multiple app log files and script output already going to one of the enterprise servers, how difficult will it be to segregate which log files go to which Enterprise server?   ... View more

Thanks for the info. I tried both solutions and they are functionally equivalent although the "untable" approach only includes events where there are data points, i.e., if the time range is > than events within the time range, the ultimate timechart only includes the time range for the events included. I inspected each search within splunk and for a relatively short time range, I see the following, so the foreach seems to be more efficient.  foreach approach : This search has completed and has returned 30 results by scanning 174 events in 1.185 seconds untable approach: This search has completed and has returned 11 results by scanning 528 events in 1.674 seconds ... View more

Sorry I don't understand this. What is the intent of the appendpipe and xyseries? The end result should be a timechart containing average of some measurement and a count of disctinct "nodes".  ... View more

Thanks for all the replies. Looks like there are 2 approaches to explore.  The foreach approach seems to work fine. I'd like to explore the other as well. Sorry for the response delay. I had PTO and some other things to do.  ... View more

Thanks for the solution. However, there is an additional case to be handled. Your solution handles the 1 day timeframe. However, we also want to see totals for timeframes > 1 day by day, i.e., license usage by environment by day. Is there a way to make the display of the stats dashboard conditional depending on the selected time frame?  ... View more

I accepted the solution. However, I'm surprised that the cloning process does not generate that automatically. ... View more

Here's how the visualization and corresponding data source are defined. I don't see a "default" in the code.  "visualizations": {         "viz_chart_1": {             "type": "splunk.line",             "dataSources": {                 "primary": "ds_search_1"             },             "showProgressBar": true,             "title": "nodesim instances ram usage (ps)",             "options": {                 "yAxisAbbreviation": "auto",                 "y2AxisAbbreviation": "off",                 "showRoundedY2AxisLabels": false,                 "legendTruncation": "ellipsisEnd",                 "showY2MajorGridLines": true,                 "yAxisMajorTickInterval": 1,                 "yAxisTitleText": "Megabytes",                 "nullValueDisplay": "connect",                 "dataValuesDisplay": "off",                 "legendDisplay": "bottom"             },             "context": {}         },   "dataSources": {         "ds_search_1": {             "type": "ds.search",             "options": {                 "query": "index=os (sourcetype=ps OR sourcetype=vmstat) (host=scale-sysperf-95x-install-18* ) (COMMAND=CURRENT* OR COMMAND=nodesim)\n| rename memTotalMB as physicalMB\n| appendpipe [ where sourcetype==\"vmstat\" | rename memUsedMB as physicalMB | eval App=\"Physical-Used\"   | fields  physicalMB]\n| stats count sum(RSZ_KB) as RSZ_KB_sum  max(physicalMB) as physicalMB  by _time host COMMAND\n| timechart span=1m avg(eval(if(isnull(physicalMB),RSZ_KB_sum/1000,physicalMB))) as AppRSZ_MB by host",                 "queryParameters": {                     "earliest": 0,                     "latest": ""                 }             }         }, ... View more

Here's the code from the converted and unmodified dashboard. It's not clear to me exactly what to change per your advise. Can you please clarify? Thanks. "inputs": { "input_1": { "type": "input.timerange", "title": "Select a time:", "options": { "token": "global_time", "defaultValue": "-60m@m,now" } }, ... View more

I'm seeing similar messages in my local Splunk install which is version 9.0.3. Additionally, the Splunk UI message looks like:  Missing or malformed messages.conf stanza for MOD_INPUT:INIT_FAILURE__modinput_eventgen_the app "SA-Eventgen"_Introspecting scheme=modinput_eventgen: script running failed (exited with code 1).2/1/2023, 12:58:23 PM ... View more

I answered this previously. Using the splunk cmd with the script and path works fine. Running the script automatically by configuring the input.conf file in ../etc/apps/SplunkUniversalForwarder/local/ results in the error 08-22-2022 16:14:15.264 +0000 ERROR ExecProcessor [3793782 ExecProcessor] - message from "python3.7 /opt/splunk/sjcinf8469vmw15/splunkforwarder-8.2.3/splunkforwarder/bin/scripts/scrapeGmrPage.py" /bin/sh: python3.7: command not found   ... View more

Is there a way to simulate that, i.e., hack. Say set up symlinks to python in Splunk's bin and lib directories? ... View more

The script runs fine when I run as you suggested.   ./splunk cmd ~/sjcinf8469vmw15/splunkforwarder/bin/scripts/scrapeGmrPage.py BTW, I do not see any python interpreters in the splunk bin directory. Not sure if that is relevant: pwd /opt/splunk/sjcinf8469vmw15/splunkforwarder/bin ls 2to3-3.7 copyright.txt genRootCA.sh openssl pip3 prigreypng priweavepng slim wheel btool easy_install genSignedServerCert.sh pid_check.sh pip3.7 pripalpng pydoc3 splunk btprobe easy_install-2.7 genWebCert.sh pip pip3.7x pripamtopng pydoc3.7 splunkd bzip2 easy_install-3.7 idle3 pip2 prichunkpng pripnglsch scripts splunkmon classify fill_test idle3.7 pip2.7 priforgepng pripngtopam setSplunkEnv srm ... View more

I understand your point. However, the curl command with the edited dashboard XML allows some automation. Since I can download the XML using scp from the dashboard directory, edit the dashboard's XML, and run the .../pdfgen/render, I suppose I have a hack work around. ... View more

Thanks for that bit of insight. I didn't know you could do that... ... View more