This problem is similar to an already submitted question regarding dispatch filenames, however mine is different given that we don't have the option make all searches saved searches.
When an environment uses SSO with tokens and long user ids, some searches fail. Users will see a red warning symbol (⚠), but search results will still be returned. The warning indicates that some data is truncated or discarded during the search due to subsearch failures.
Splunk instances running searches create temporary files in the Splunk dispatch directory. To ensure these files are unique, they are named using a concatenation of many variables about the search, including the time the search was run, the name of the app it was run on, the hostname, the user id of the user running the search, and so on.
Filenames are almost universally limited to 255 characters in length, regardless of OS and filesystem. In this scenario, Splunk is generating filenames that exceed 255 characters, and the search job trying to create the file fails.
Our SSO IDAM solution exacerbates the issue because user ids in these environments are very long, in the form lastname.firstname.middlenames.edipi. Since SSO user ids contain non-alphanumeric characters, rather than concatenating the user id itself into the filename, Splunk first encodes them first using Base64 to eliminate any potential invalid characters. This increases the length of the already-long user ids by 33%. For example, alexandra.williamson.a.12345678 (31 characters) becomes YWxleGFuZHJhLnBhcmtlci5pLjEyMzQ1Njc4 (40 characters).
User ids are also included in the filename twice, increasing them in total by 166%, meaning for every character you add to a user id, 2.6 more characters end up in the filename. Longer user ids quickly exhaust this 255 character limit.
Has anyone found a solution to this problem? Seems like a quick/easy solution would be for Splunk to hash vs Base64.
@rayskycubed we had similar issue where we had got Enhancement Request
SPL-165509 created for Splunk to ensure that the Dispatch Director name does not grow beyond 255 characters.
There are several reasons because of which Dispatch Directory folder name uses base64 encoding and hence becomes two long. Refer to Splunk Documentation as well for details on how Dispatch Director name is created: https://docs.splunk.com/Documentation/Splunk/latest/Search/Dispatchdirectoryandsearchartifacts
workaround We have taken care of those in our Splunk environment in order to control dispatch directory names:
1. Splunk App name should not be too long (>20 characters and should not have special characters) which hashes the App name hence increases the number of characters in Directory name.
2. User role should shortened and should not have special characters.
3. Too many subsearches should be avoided, as besides it search performance/event truncation impacts, each subsearch adds a
subsearch_ to dispatch directory name.
Thanks, however all this could be avoided if Splunk just hashed vs encoded.
This impacts search head snapshots which fail with too long path names. Then if a shcluster member falls out of sync far enough where only a resync can recover, old snapshots will be invoked wiping out changes to the cluster since the last snapshot. Don't resync to recover an unhealthy member until you check /var/run/snapshot and ensure you have current bundles on at least the captain.
There's close to zero documentation about snapshots and how they work other than a little note in release notes and snapshots failing due to long pathnames. We of course will add to look for the errors regarding long pathnames and snapshots failing in splunkd.log in our monitoring solution, but pop up errors/messages in the gui, like we get for other cluster problems would have really helped here.