I've got a scripted input running on a universal forwarder that generates json output to the tune of 18,000+ lines.  However, when I query for its events in the Splunk search, it shows only 13 lines per event.   { "apiVersion": "v1", "items": [ { "apiVersion": "apps/v1", "kind": "Deployment", "metadata": { "annotations": { "deployment.kubernetes.io/revision": "1", "field.cattle.io/publicEndpoints": "myPublicEndpoint1", "meta.helm.sh/release-name": "myrelease1", "meta.helm.sh/release-namespace": "mynamespace1" },    I've tried setting TRUNCATE to both 0 and 1000000 in props.conf for the scripted input's sourcetype ("scriptedinput1") on both the universal forwarder and the search instances and restarted the services, but the truncating remains the same.   [scriptedinput1] KV_MODE=json TRUNCATE=1000000   I should also note that I'm not seeing "truncating" anywhere in my splunkd.log on the universal forwarder and search instances.  Any assistance with configuring Splunk to not truncate my scripted input running on the universal forwarder would be greatly appreciated. ... View more

I have created a scripted input (/opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh) to run against my kubetools instances that simply runs a kubectl command and outputs json:     #!/bin/sh /usr/bin/kubectl -n mynamespace1 get deployments,statefulsets -o json     However, after I go to set up the scripted input in the Data inputs section of the Splunk console and run a search, I'm seeing this error in splunkd.log:     ERROR ExecProcessor - message from "/opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh" /opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh: line 3: /usr/bin/kubectl: No such file or directory     I suspect it's throwing this error because kubetools is only installed on the kubetools instances and not the splunksearch instance.  Is there any way to run scripted inputs with commands in them that aren't installed on the splunksearch instance?  If not, what alternative solution would be recommended?  Any assistance with solving this would be greatly appreciated. ... View more

I have Splunk_TA_nix installed and ps.sh enabled on my Apache storm nimbus instances.  I can run a general ps sourcetype query on a service I know should always be running like rhnsd and get events back just fine ...     index=os host="my-stormn-1" sourcetype=ps rhnsd      However, when I do the same for the "stormnimbus" service I get zero events back ...     index=os host="my-stormn-1" sourcetype=ps stormnimbus     Meanwhile, a "sudo systemctl status stormnimbus" on the my-stormn-1 instance itself shows that it is active and running.  I'm having the same problem also with the stormui service as well as the stormsupervisor service running on my storm supervisor instances.  I should note that I do have Splunk_TA_nix installed on my splunk indexers.  Any advice as to why these services are not returning events with ps and how to fix it would be greatly appreciated. ... View more

I'm trying to create a column chart (bar graph) in my Splunk (v8.1.3) dashboard that shows the availabilities of a given service for various instances, whereby the bars showing percent availability change color depending on their value.  For any availability greater than or equal to 0 and less than 90, I want the bar to be red.  For any availability greater than or equal to 90 and less than or equal to 100, I want the bar to be green.  For any availability outside of those ranges, the default color is fine.  Seems simple enough and this question has been asked in several different forms several times over the years, but I just can't seem to get mine to work.  The availability bars just keep showing in blue, and I've tried with both the rangemap and eval methods.  See the XML of my chart based on the rangemap solution below.  Any help is greatly appreciated.   <chart> <title>rhnsd Availability - my-db-*</title> <search> <query>index=om host="my-db-*" sourcetype=ps rhnsd | stats count by host | addinfo | eval availability=if(count&gt;=(info_max_time-info_min_time)/1800,100,count/floor((info_max_time-info_min_time)/1800)*100) | rangemap field=availability red=0-90 green=90-100 | fields host availability</query> <earliest>$query_time.earliest$</earliest> <latest>$query_time.latest$</latest> </search> <option name="charting.axisY.maximumNumber">100</option> <option name="charting.axisY.minimumNumber">0</option> <option name="charting.chart">column</option> <option name="charting.chart.stackMode">stacked</option> <option name="charting.fieldColors">{"red":0xdc4e41,"green":0x53a051}</option> <option name="charting.layout.splitSeries">0</option> <option name="charting.legend.placement">none</option> <option name="refresh.display">progressbar</option> </chart>     ... View more

I'm trying to create a statistics table for whether or not a given Linux service is running on a set of hosts.  For example, for service "rhnsd" running on hosts "my-*" ... Host State my-db-1 Running my-db-2 Stopped my-web-1 Running my-web-2 Stopped   I have the ps module enabled, so I can use that as a source/sourcetype, but not sure how to eloquently display all hosts and the state of the given service like I've illustrated above.  Any help is greatly appreciated. ... View more