How do I get Splunk to limit the Windows file path to under 260 characters?

[michael_schmidt]

Getting the following Error on one of our clustered indexers (and similar ones on the other indexers):

10-26-2016 16:20:03.362 -0500 ERROR SearchResultsWriter - Unable to open output file: path=C:\Program Files\Splunk\var\run\splunk\dispatch\remote_SplunkSH02_scheduler__admin_c3BsdW5rX2FwcF93aW5kb3dzX2luZnJhc3RydWN0dXJl__RMD5e93ff07c552f3ee0_at_1477516800_3187_F5AAE4E2-7A34-4327-8CDA-83913FB48502\index_buckets.csv.647C07D6-2813-4D98-AD2E-ED1FCACEB554.tmp error=The system cannot find the path specified.

Background: 3 Indexer Cluster, all running on Windows. 3 Search Head Cluster, also Windows.

The directories all exist, the permissions are set correctly, and the file itself does not exist. When these errors occur, the RAM usage goes through the roof and quite often it ends up crashing splunkd on the indexer.

Spoiler Alert:

I know why the error is occurring. It's because in all of M$'s glory, they still hard code the file path limit to 260 characters. This file path is 264 characters. Now, how do I get Splunk to limit the file paths to under 260 characters?

[jeremyhagand61]

If you are running Windows Server 2016 or above, you can simply enable long paths. We had this problem and the solution fixed it for us.

https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd

Note that while this documentation refers to Windows 10 it is just as valid on Windows Server

[juan_miller]

I would suggest to use Long Path Tool program. It resolves problem regarding source path too long.

[juan_miller]

Hi, for problems concerning path too long issues, I suggest you try the new long path tool. This can help you with all kinds of path too long cases,

[williamlaing]

Another way to solve the long path issue is to use Long Path Tool.
Simple and easy. Worked for me!

[Mark0Community]

260 get me ?

Yeah it can be a real headache, I tried LongPath Tool Program which helped a lot.

[jkat54]

This is a newer option that’s only available on select versions of Windows 10 iirc.

Great vendor supplied option if you’re OS is compatible.

[gavsdavs_GR]

Can I add a "me too" to this thread ?
SplunkEnterpriseSecuritySuite searches fall foul of this as well

[michael_schmidt]

Also just for the record, moving the default installation to a shorter path also fixed our problems with our ES search head as well, and now things are much better!

We also changed our ES search head to be installed on Linux, and got better performance in general, but in theory if you had ES running on a Windows based search head, changing the install path on that as well should alleviate any problems you'd have on the search head with temp files, and changing the install path on the indexers I can definitively say alleviates the errors that were occurring on the indexers.

[gavsdavs_GR]

If i remember correctly (we have now migrated off Windows indexers), the issue was most critical on the indexers not the search heads.

I moved the installs from c:\Program Files\Splunk to c:\S, in the process winning back 18 characters of pathname space. It sort of helped, but there were still dispatch directories exceeding 255 characters and truncating.

Do any of the more modern windoze releases permit >256 pathnames ?

[jkat54]

Looks like windows 10 supports “extended path names”.

NTFS supports super long paths but it’s MS code that does not.

[jkat54 · Accepted solution]

One option: Move Splunk to c:\splunk

[jkat54]

Good point but what if you symlinked c:\asdf to the Splunk dir, and then changed the service to execute c:\asdf\bin\splunk.exe

Might work

Did you rename those searches though?

[michael_schmidt]

Circling back to close this up... finally...
Nope, Turns out the best solution for us was to reinstall Splunk, and move the necessary Conf files from the original location to the new one, as was jkat's original solution.

[jkat54]

http://docs.splunk.com/Documentation/Splunk/6.5.0/Search/Dispatchdirectoryandsearchartifacts

If the name of the search is less than 20 characters and contains only ASCII alphanumeric characters, then the search-specific directory name includes the search name.

If the name of the search is 20 characters or longer, or contains non-alphanumeric characters, then a hash is used instead. This is to ensure a search-specific directory named by the search ID can be created on the filesystem.

So, is your search name longer than 20 characters and causing a hash (It appears so)?

If so, Option 2: shorten your search name

I think the more reliable option is Option 1 however, move splunk to c:\splunk. You'll never know when someone is going to create a search name longer than 20 chars.

Also someone should file an enhancement request / bug report here.

Enhancement = Let us specify the number of chars in the hashing algo
Bug = Causes issues on windows out of box

[juan_miller]

I would suggest to use Long Path Tool program. It resolves problems regarding the source path too long.

[michael_schmidt]

sourcetype=wineventlog =22 characters.
You're not going to get anywhere with a 20 character search, so I agree that's a bad option. I could move the Splunk install location, but even that's not a great option, and would take some time to rip it out, clean it up, and put it all back together, then repeat two more times to get all three indexers back up. This is definitely something that's in need of a bug/enhancement request.

Even better than the request to specify the number of chars in hashes, would be to allow us to specify the location where the hashed directories are created. Then I could map a drive to wherever I wanted, and just use the letter to specify where they go. For example if there was a conf that included:

[hashConfig]
hashPath = H:

then I could map H to C:\Program Files\Splunk\var\run\splunk\dispatch\
And then the files would be in the exact same location, but would be accessible because the file path for H:\ is only 3 characters.

[jkat54]

no no no... not a 20 character search... a 20 character search NAME. When you save the search... the NAME you give it is what they're talking about in the link.

what if you sym linked the dispatch directory to a lower directory?

start->run->cmd
mklink /?
mklink /J "C:\Program Files\Splunk\var\run\splunk\dispatch\"  H:\

Might want to stop splunk before doing this, and start it afterwards. Might even require a reboot. Honestly i've never used symlinks in windows but the mklink /? shows the syntax.

[michael_schmidt]

Just for the record...I didn't create the searches that are causing this problem. I finally figured out where they're coming from. It's the pre-defined searches (aka "Guided Setup") that the Splunk App for Windows Infrastructure runs to verify that the proper data is flowing into Splunk, and to pre determine what panels to setup in the App.

Also, Symbolic Linking it wouldn't alleviate the problem, unless you could tell Splunk to use the sym link instead of the default path. The problem isn't that there's no space, or that the space can't be accessed. I could in theory create an H:\ and then add 257 characters again for a grand total of 500 characters, but windows won't let you work past 260 in the NAME of the path, so unless splunk lets me change the file path to the dispatch directory, windows won't allow it.

[Mark0Community]

260 get me ?

Yeah it can be a real headache, I tried LongPath Tool Program which helped a lot.

[ddrillic]

A lovely notice -

Microsoft removes 260 character limit for NTFS Path in new Windows 10 Insider Preview