That command is for after making changes to serverclass.conf.  It won't help after installing an app on the DS. To reload configs, try this command http://<yoursplunkserver>:8000/en-US/debug/refresh?entity=admin/transforms-lookup See also https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Configurationfilechangesthatrequirerestart#Splunk_Enterprise_changes_that_do_not_require_a_restart ... View more

It's an unsupported configuration.  It was explicitly prohibited at one time, but I can't find that documented now. Note that it is possible for a DS to be a client of another DS.  This hierarchical structure has been used to manage multiple DSs when there are too many endpoints for a single DS to handle. ... View more

You've just stumbled across SmartStore (S2).  S2 keeps hot buckets local and copies warm buckets to S3.  A cache of roughly 30 days of data is retained locally for faster search performance. To implement S2 correctly, see https://docs.splunk.com/Documentation/Splunk/9.4.2/Indexer/AboutSmartStore ... View more

Since Splunk cannot be changed, you will have to change your email policy to allow messages from the specified email address. ... View more

The frozenTimePeriodInSecs setting does not apply to hot buckets.  You should, however, see warm buckets once a hot bucket fills up or becomes 90 days old.  I can't explain why you don't. ... View more

You say you found no warm buckets, but what about cold buckets?  Did you find any of those? If you're not running out of disk space then maxVolumeDataSizeMB is not too high. The current settings have buckets spanning 90 days.  Therefore, you should have 5 "generations" of buckets - 0-90 days (hot), 91-180 days, 181-270 days, 271-360 days, and 361-450 days.  That last one is because a bucket won't be frozen until *all* events in it exceed the retention time. Set maxHotSpanSecs to 86400 so each bucket only contains a single day of data and retention should improve. ... View more

https://www.splunk.com/en_us/about-splunk/splunk-data-security-and-privacy/compliance-at-splunk.html?locale=en_us ... View more

Having a lot of indexes can work against you.  It means the UI can take longer to load.  It also means indexers have to open and unzip more files.  It may also lead to more buckets for the Cluster Manager to track. Some are tempted to create a new index for each data source.  Resist that temptation.  A new index is needed if: 1) New access requirements are needed for some data 2) New retention requirements are needed for some data 3) Data volume is high enough that searches for low-volume data in the same index is affected ... View more

As @livehybrid noted, explicit type conversion does not make a difference here.  If you need numbers sorted as strings then you must use the str operator. ... View more

In the CMC, go to Forwarders->Forwarder monitoring setup and click the "Rebuild forwarder assets" button. ... View more

Avoid search filters.  While they can be useful at times, more often they complicate matters.  For instance, in your case the members of 6-8 AD groups (therefore, presumably, in 6-8 Splunk roles) will have 6-8 search filters combined with implicit AND operators to create a search that finds nothing. The only reliable way to control access to data is to put that data in an index with the proper RBAC settings.  Rather than have a single summary index, it would be better to create a separate summary index for each group of users with unique access requirements. ... View more

Ten seconds is far too often to refresh a dashboard.  Unless you have an automaton monitoring the dashboard and taking action on what it finds, 5 minutes is more reasonable.   It sounds like the dashboard is occasionally encountering periods when there are too many other searches running so it has to wait for resources.  There is little a dashboard can do about that (other than not contributing to the problem by refreshing too frequently).   If multiple users are accessing the dashboard at the same time, consider replacing in-line searches with scheduled searches that are invoked by the dashboard using the loadjob or savedsearch command.  That will replace multiple executions of the same query with a single execution and each user will have the same view of the data. ... View more

If neither the production logs nor the internal logs from the same UF are found then it's likely that the UF has lost its connection to the indexer(s). Verify the UF has the right settings in outputs.conf Verify the network still allows connections from the UF to the indexers. If you use TLS, confirm the certificate is still valid and that the UF has the correct password. ... View more

Have you tried using single quotes to tell eval you're referring to a field name? | eval older = mvfilter(timestamps < '_time')   ... View more

@livehybrid's advise is good.  I would add that the idea of copying protected data into an unprotected index is unsound. ... View more

When I add  | eval type=typeof(fruit) to the query the results say the numbers are indeed numbers rather than strings.  That would explain the sort. When I use  | sort str(fruit) the results are in the expected lexicographical order. FWIW, the docs do say "Numeric data is sorted as you would expect for numbers and the sort order is specified as ascending or descending." ... View more

Did you put <collection> in a collections.conf file, distribute it to all SHs, and restart Splunk?  Make sure the collections.conf file defines each field you want to use. ... View more

1. Yes.  If the UI cannot delete a KO then it must be removed by other means, including editing the .conf file.  Best Practice is to update the app that defines the KO and then re-install the app. 2. Yes, if disabling is available then that is a safe option. 3. Use btool.  It will apply proper config file precedence and show where each setting came from. splunk btool --debug <<config file base name>> list ... View more

The allowedDomainList setting can be in any alert_actions.conf file on your search head(s).  Precedence rules apply, however.  See https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Wheretofindtheconfigurationfiles ... View more

You're right.  I took the sm21.txt file in the OP to be sample data rather than a lookup table. ... View more

Try an eval element with a case function. <eval token="foo">case(schedule_dttm.latest="now", now(), is_num(schedule_dttm.latest), schedule_dttm.latest, 1==1, relative_time(now(), schedule_dttm.latest)</eval>   ... View more