This problem is similar to an already submitted question regarding dispatch filenames, however mine is different given that we don't have the option make all searches saved searches.
Symptom
When an environment uses SSO with tokens and long user ids, some searches fail. Users will see a red warning symbol (⚠), but search results will still be returned. The warning indicates that some data is truncated or discarded during the search due to subsearch failures.
Splunk instances running searches create temporary files in the Splunk dispatch directory. To ensure these files are unique, they are named using a concatenation of many variables about the search, including the time the search was run, the name of the app it was run on, the hostname, the user id of the user running the search, and so on.
Filenames are almost universally limited to 255 characters in length, regardless of OS and filesystem. In this scenario, Splunk is generating filenames that exceed 255 characters, and the search job trying to create the file fails.
Our SSO IDAM solution exacerbates the issue because user ids in these environments are very long, in the form lastname.firstname.middlenames.edipi. Since SSO user ids contain non-alphanumeric characters, rather than concatenating the user id itself into the filename, Splunk first encodes them first using Base64 to eliminate any potential invalid characters. This increases the length of the already-long user ids by 33%. For example, alexandra.williamson.a.12345678 (31 characters) becomes YWxleGFuZHJhLnBhcmtlci5pLjEyMzQ1Njc4 (40 characters).
User ids are also included in the filename twice, increasing them in total by 166%, meaning for every character you add to a user id, 2.6 more characters end up in the filename. Longer user ids quickly exhaust this 255 character limit.
Has anyone found a solution to this problem? Seems like a quick/easy solution would be for Splunk to hash vs Base64.
... View more