Depending on where and when in the search process the conversion from kilobytes to gigabytes is done, rounding errors may be coming into play. Try perhaps not doing the conversion till after the sum. That should help minimise the error, especially if you have lots of small bits of data coming in. ... View more

There are lots of ways to skin a cat. I suggest you start by looking here: https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/Forwarding/Routeandfilterdatad Come back if you have any further questions or want further explanations. ... View more

If the HF has the appropriate inputs file to gather the data, you can write a search to format the data as a table, and then use the outputcsv search command which will dump a csv of the results on disk. You could then use a script to check for new csvs and scp them to where you need them as and when required. You could also ask the HF to send uncooked data to the third party system but it would send each file line by line so you'd need to reconstruct at the other end. ... View more

Is this Linux or Windows? If Linux and the permissions were not set correctly you may have only partially upgraded and may be running a Frankenstein's ES at the moment. If it is as simple as that, a chown -R splunk:splunk /opt/splunk (assuming that's the account and location that match your environment) and a second attempt at install should fix things. If not... What does it say in Splunkd.log? Look for ERROR or WARN messages there. Also in $SPLUNK_HOME/var/log/splunk/ you should have a file called (if memory serves) ess2_installer.log or maybe ess_installer2.log, can you look there and post the last 100 lines or so? ... View more

Does your Splunk system account have permission to read other files in that directory? ... View more

Is the lookup actually present on the indexers? Most Apps/Some TAs are only installed on the search head, meaning if the lookup came that way it won't be on the indexers and you'll get a warn at search time. By default, Splunk will attempt to use lookups on the indexers, as that spreads the load out and it's one less thing for the search head to do when results come in. However, you can if you wish set local=true as part of the search which invokes the lookup so that it's not run on the peers. Take a look here: https://docs.splunk.com/Documentation/Splunk/8.0.1/SearchReference/Lookup#Optional_arguments ... View more

If they can't add, are the definitely the same version as the other indexers? Are they the same operating system? Do they have the required ports open for replication and communication to the master? Is there a network route from them to the master and to the other peers? Is there a firewall that needs configured (both on the network and on the host). Are the new indexers using the correct pass4SymmKey? Is there a typo in the name of the clustermaster in server.conf [clustering] stanza? As a troubleshooting measure, take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log for WARN or ERROR messages concerning clustering on the new indexers. The reason why they can't join will likely be explained there. ... View more

Is fileName a field that exists in your search results immediately before you | to your custom command? If not, I would have expected the script to error out (missing a required arg?) but perhaps it's carrying on with null/empty value and that's why you're getting a csv with no name? ... View more

Can you give me an example of the search you're currently using and I'll see if I can rework it to use stats instead of transaction. Transaction is a very costly search command, and you will almost always get better performance using stats instead. ... View more

Try this: | dbinspect where index=* state=hot | fields startEpoch sizeOnDiskMB path index | eval now=now(),ageInHours=round(((now-startEpoch)/3600),2),sizeOnDiskMB=round(sizeOnDiskMB,2) | fields - now startEpoch | table index path sizeOnDiskMB ageInHours ... View more

The issue you might have is that colouring on single values is for ranges. You're looking for deviation percentage, whilst also still retaining the actual result in the single value. You could perhaps have two single values next to each other - one is the literal value, and next to it is the percentage deviation. Then you can colour it on a range. ... View more

I'm not sure about using TERM for subnets. TERM instructs Splunk to not view the dot as a minor breaker, but instead to literally search for that IP, not for 5 5 0 0. ... View more

What version of Splunk is this on? ... View more

Hmm, if you're getting the Pony, there's usually something more fundamental at play. That normally occurs when there are incompatible apps installed, or an app requires another but that's not installed, or not the right version. I've seen that most commonly with Splunk App for Windows Infrastructure and the Windows TA/MSAD TA/DNS TA. ... View more

Hmm, have you tried navigating away from the dashboard and then going back to it? I have seen intermittent random errors like this before. ... View more

Are all of your tokens populated? You'll get that error if a dashboard panel expects token input and the token isn't set. ... View more

Without going into the specifics of Ansible, there are some other things you should be aware of when doing upgrades in an environment like that: Namely that the order of upgrades is important and you should ensure Ansible respects that (i.e. it doesn't attempt to do things in parallel). You should first update the Licence Master, then the Cluster Master, then the Search Heads, then the Indexers. You could create Ansible variables for LM_updated/CM_updated and assign them to be true if splunk --version matches the target version of an upgrade, so that phase two of an upgrade won't start until phase 1 has started. ... View more