Depending on where and when in the search process the conversion from kilobytes to gigabytes is done, rounding errors may be coming into play. Try perhaps not doing the conversion till after the sum. That should help minimise the error, especially if you have lots of small bits of data coming in. ... View more

There are lots of ways to skin a cat. I suggest you start by looking here: https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/Forwarding/Routeandfilterdatad Come back if you have any further questions or want further explanations. ... View more

If the HF has the appropriate inputs file to gather the data, you can write a search to format the data as a table, and then use the outputcsv search command which will dump a csv of the results on disk. You could then use a script to check for new csvs and scp them to where you need them as and when required. You could also ask the HF to send uncooked data to the third party system but it would send each file line by line so you'd need to reconstruct at the other end. ... View more

Is this Linux or Windows? If Linux and the permissions were not set correctly you may have only partially upgraded and may be running a Frankenstein's ES at the moment. If it is as simple as that, a chown -R splunk:splunk /opt/splunk (assuming that's the account and location that match your environment) and a second attempt at install should fix things. If not... What does it say in Splunkd.log? Look for ERROR or WARN messages there. Also in $SPLUNK_HOME/var/log/splunk/ you should have a file called (if memory serves) ess2_installer.log or maybe ess_installer2.log, can you look there and post the last 100 lines or so? ... View more

Does your Splunk system account have permission to read other files in that directory? ... View more

Is the lookup actually present on the indexers? Most Apps/Some TAs are only installed on the search head, meaning if the lookup came that way it won't be on the indexers and you'll get a warn at search time. By default, Splunk will attempt to use lookups on the indexers, as that spreads the load out and it's one less thing for the search head to do when results come in. However, you can if you wish set local=true as part of the search which invokes the lookup so that it's not run on the peers. Take a look here: https://docs.splunk.com/Documentation/Splunk/8.0.1/SearchReference/Lookup#Optional_arguments ... View more

If they can't add, are the definitely the same version as the other indexers? Are they the same operating system? Do they have the required ports open for replication and communication to the master? Is there a network route from them to the master and to the other peers? Is there a firewall that needs configured (both on the network and on the host). Are the new indexers using the correct pass4SymmKey? Is there a typo in the name of the clustermaster in server.conf [clustering] stanza? As a troubleshooting measure, take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log for WARN or ERROR messages concerning clustering on the new indexers. The reason why they can't join will likely be explained there. ... View more

Is fileName a field that exists in your search results immediately before you | to your custom command? If not, I would have expected the script to error out (missing a required arg?) but perhaps it's carrying on with null/empty value and that's why you're getting a csv with no name? ... View more

Can you give me an example of the search you're currently using and I'll see if I can rework it to use stats instead of transaction. Transaction is a very costly search command, and you will almost always get better performance using stats instead. ... View more

Try this: | dbinspect where index=* state=hot | fields startEpoch sizeOnDiskMB path index | eval now=now(),ageInHours=round(((now-startEpoch)/3600),2),sizeOnDiskMB=round(sizeOnDiskMB,2) | fields - now startEpoch | table index path sizeOnDiskMB ageInHours ... View more

The issue you might have is that colouring on single values is for ranges. You're looking for deviation percentage, whilst also still retaining the actual result in the single value. You could perhaps have two single values next to each other - one is the literal value, and next to it is the percentage deviation. Then you can colour it on a range. ... View more

I'm not sure about using TERM for subnets. TERM instructs Splunk to not view the dot as a minor breaker, but instead to literally search for that IP, not for 5 5 0 0. ... View more

If the usernames are important, then add them in as key=value pairs before the first pipe, and then as a second by clause for the stats(dc). Something like this: index=windows-logs (EventCode=4624 OR EventCode=4625) Source_Network_Address=* NOT src="10.1.2.23" (user=Administrator OR user=root) | stats dc(dest) AS dest by Source_Network_Address user | where dest > 20 ... View more

What version of Splunk is this on? ... View more

Hmm, if you're getting the Pony, there's usually something more fundamental at play. That normally occurs when there are incompatible apps installed, or an app requires another but that's not installed, or not the right version. I've seen that most commonly with Splunk App for Windows Infrastructure and the Windows TA/MSAD TA/DNS TA. ... View more

Hmm, have you tried navigating away from the dashboard and then going back to it? I have seen intermittent random errors like this before. ... View more

Are all of your tokens populated? You'll get that error if a dashboard panel expects token input and the token isn't set. ... View more

It looks like you're trying to do makecontinuous over _time. If you haven't already, take a look at the documentation here: https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/SearchReference/Makecontinuous#Examples You'll need to pass in a few extra settings to makecontinuous to tell it what size bin/span you want to use. ... View more

Without going into the specifics of Ansible, there are some other things you should be aware of when doing upgrades in an environment like that: Namely that the order of upgrades is important and you should ensure Ansible respects that (i.e. it doesn't attempt to do things in parallel). You should first update the Licence Master, then the Cluster Master, then the Search Heads, then the Indexers. You could create Ansible variables for LM_updated/CM_updated and assign them to be true if splunk --version matches the target version of an upgrade, so that phase two of an upgrade won't start until phase 1 has started. ... View more

Do you have the actual EVAL statement? If an EVAL is applied, but doesn't actually evaluate out because of a no match/the logic doesn't function as expected, then you wouldn't see anything in your results. If the EVAL is sensitive, can you post a sanitised version of it? ... View more

| tstats count where index=yourindex by yourindexedfield If it works, you're in business, if not... sadly no. You could also do a conventional search like index=foo fieldname::value If that works, it's an indexed field. ... View more

You will need to tell us about what your actual search is, what version of Splunk you're using, your architecture, your data ingest volumes etc etc before there's any way we can help with a query this generic. ... View more

index = _internal component=Metrics earliest=-60m@m latest=-30m@m | stats count as OldMetrics | appendcols [search index = _internal component=Metrics earliest=-30m@m latest=now | stats count as LatestMetrics] | eval deviation=round(OldMetrics/LatestMetrics,2),deviation=1-deviation | eval alert=case(deviation<-0.5,"Yes",deviation>-.05 AND deviation<0.5,"No",deviation>0.5,"Yes") That should give you an example of how you can compare two values across two time periods. For your use case you'd want to format the single value to be red if deviation is between -0.5 and 0.5 (hence you can use the alert field) - if you need to use numeric values cause formatting doesn't let you use Yes/No, then use replace those in the search ... View more