Repository navigation
Extremely poor performance with latest stable, I/O error, random crashes on folder creation #54
Description
Using:
- sshfs-win-2.7.17334-x64.msi
- winfsp-1.3.18160.msi
Connecting to an Android phone (running primitive ftpd) over wifi results in extremely poor performance.
Copying a folder with 10 files with explorer takes like a minute just to calculate the files to copy.
Using WinSCP the files start copying instantly and with over 5 MB/s.
Using:
- winfsp-1.4.18211.msi
- sshfs-win-3.2.18213-x64.msi
first resulted in an I/O error, then the drive got disconnected and stayed that way (there was no possibility to remove the faulty drive).
After mapping another drive, I again got I/O errors even just trying to create a new folder.
Reacted by Wiz
Activity
billziss-gh commentedon Aug 15, 2018
@xnoreq how are you launching the file system?
xnoreq commentedon Aug 15, 2018
Explorer - Connect network drive:
\\sshfs\user@ipv4!portxnoreq commentedon Aug 15, 2018
With the stable version it actually doesn't even copy the files. It's stuck in calculating... for a long time (the Explorer file copy dialog) and then when it finally starts copying the first file it fails the error element not found.
xnoreq commentedon Aug 16, 2018
Is there any way to debug/log what is going on?
billziss-gh commentedon Aug 17, 2018
@xnoreq you can start sshfs from the command line and give it the option
-d.xnoreq commentedon Aug 17, 2018
billziss-gh commentedon Aug 17, 2018
@xnoreq try to set the path before executing SSHFS:
Also when you start SSHFS from the command-line use the options
-o uid=-1,gid=-1to make the file system accessible to the user that starts the file system.xnoreq commentedon Aug 17, 2018
Still the same error.
It looks like it is doing some operation that is not supported by the server.
What am I doing wrong? Why does mounting work through Explorer?
xnoreq commentedon Aug 17, 2018
I switched the way primftpd accesses the filesystem on the android phone (the option is called "plain old filesystem"), and now it connects also in using sshfs.exe, but the resulting drive cannot be accessed.
The security tab in explorer shows my user, nobody and everyone, but none of them have any permissions.
Has anyone ever tried this with an Android phone?
billziss-gh commentedon Aug 18, 2018
@xnoreq did you try setting the
PATHenvironment variable before startings SSHFS? Do you have anotherssh.exein thePATHsomewhere?xnoreq commentedon Aug 18, 2018
@billziss-gh Of course I have adjusted PATH, and no, I don't have another ssh.exe in the PATH.
4O4 commentedon Aug 18, 2018
@xnoreq What about native sshfs under linux, have you maybe tested it before? If not, can you setup some minimal linux virtual machine and check if it works better with your android server?
xnoreq commentedon Nov 5, 2018
After some changes (using termux now on my phone) and testing using sshfs on a linux box (works fine) I get the following problems:
The folder is created however, because after closing the error and refreshing it appears in Explorer.
Additionally, when pressing retry in the error popup sshfs crashes.
Here is the log from creating a folder:
Retry:
billziss-gh commentedon Nov 5, 2018
@xnoreq sshfs-win does not currently support pubkey authentication without having to resort to the command line and/or batch files or special launchers (e.g. sirikali).
Error
0x800703E3is the "wrapped" Win32 ERROR_OPERATION_ABORTED. This happens in rare occasions, but having the file system fail with an access violation (as per your second log) is one of them.I am assuming this happens with the latest SSHFS beta and not the stable version?
xnoreq commentedon Nov 6, 2018
What if it happens every time?
Correct.
8 remaining items
xnoreq commentedon May 24, 2019
As I said before I've already tried upstream sshfs on cygwin. It crashes. It says it wrote a stackdump but it doesn't.
It crashes the same with gdb. No backtrace.
I tried again with
meson --buildtype debug. Same result.Either it didn't properly compile in debug mode or the crash is in winfsp.
Can you provide builds with debug symbols?
billziss-gh commentedon May 24, 2019
Using
addr2line -f -C -e ./sshfs.exe ADDRESSI was able to extract a useful stack frame with sshfs compiled with debug symbols.Setting a breakpoint at
sshfs.c:3163breaks before the failure, which happens insidebuf_add_buf. I do not have the time investigate this further right now, but there may be an incompatibility in how the WinFsp-FUSE3 layer passes information to sshfs.EDIT: line numbers are from the sshfs commit included as a git submodule to this project.
xnoreq commentedon May 25, 2019
@billziss-gh Thanks.
I've debugged a bit.
After creating the folder, the first call to sshfs_getattr returns -EIO sshfs.c#L3152 because the fuse_file_info->fh (which is a struct sshfs_file *) seems to have been corrupted.
The second time (e.g. when clicking retry after the I/O error dialog pops up in Explorer) sshfs_file_is_conn will succeed but sf is still corrupt.
So
buf_add_buf(&buf, &sf->handle)crashes becausesf->handleis corrupted:{p = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 6445641896, size = 32}The whole backtrace:
So unless sshfs randomly mangles its sshfs_file structs it looks like winfsp mangles struct fuse_file_info (at least the fh field) which it passes to
cache_getattr.xnoreq commentedon May 25, 2019
This line seems fishy: https://github.com/billziss-gh/winfsp/blob/c01402443ded2d0043a85fe514b4a234f0a6fd9a/src/dll/fuse/fuse_intf.c#L376
billziss-gh commentedon May 25, 2019
Unfortunately the backtraces that GDB prints for WinFsp appear bogus.
Thanks for tracking this. It might be that there is a subtle difference between the rules for FUSE2 vs FUSE3 in this case. FUSE2 had both
getattrandfgetattr, but in FUSE3 they have been merged into a singlegetattrcall.bes-internal commentedon May 27, 2019
I got STATUS_ACCESS_VIOLATION on create dir at windows 10 x86_64, WinFsp-2019.2 both with new sshfs-win-3.2.18213-x64 and sshfs-win-3.5.19106-x64. But not with sshfs-win-2.7.17334-x64
billziss-gh commentedon May 27, 2019
Yes, this looks like this is a problem with the FUSE3 layer that newer SSHFS uses, but not with the FUSE2 layer that older SSHFS uses.
temintyd commentedon Jun 29, 2019
Passing "-o workaround=fstat" fixes error 0x800703E3 #54 for me (WinFsp 2019.3 B1 + SSHFS-Win 3.5 BETA)
billziss-gh commentedon Jul 1, 2019
@xnoreq and @temintyd thanks for the investigation and workaround respectively.
I finally did some more digging into this using the Visual Studio debugger attached to
sshfs.exetogether with a debug version of the WinFsp DLL. I include full steps to recreate my debugging session at the end of this post.During directory creation a "CREATE DIRECTORY" call is being made (
fsp_fuse_intf_CreatewithFILE_DIRECTORY_FILE). When creating a directory in Windows the semantics are to create the directory and open it. Therefore this code callsmkdirand thenopendir. Thefhvalue returned byopendiris recorded infi.fh. Later the code callsfsp_fuse_intf_GetFileInfoFunnel, which sees a valid value forfi->fhand callsfgetattron it. This results in an Access Violation.There is a clear incompatibility here. WinFsp-FUSE treats values returned in
fheither byopenoropendir(orcreate) as valid values that can be passed tofgetattr. However SSHFS seems to not like that.Digging into the SSHFS code we find
sshfs_openandsshfs_opendir. Notice thatfi->fhis assigned different kinds of objects in the 2 functions:sshfs_opensshfs_opendirI believe the correct action here is to take this to the libfuse and sshfs maintainer @ nikratio and have a discussion with him. While I would like to argue that
openandopendirshould be returningfhvalues from the same "object space" and thereforefgetattrshould work on both kinds offhvalues (and I certainly design my file systems this way), the inconvenient truth is that what libfuse and sshfs does is correct by default and I will likely have to change WinFsp-FUSE to avoid callingfgetattron anfhreturned byopendir.build\VStudio\winfsp.slnand configurationDebug|x64.sshfs.exeinto the WinFsp build directory.winfsp.dllas startup project (the default) and openProject Properties > Debugging$(OutDir)sshfs.exe-f -ouid=-1,gid=-1 USER@SERVER: Y:$(OutDir)PATH=C:\Cygwin64\bin(or wherever your Cygwin lives)fsp_fuse_intf_Create.winfsp.dll.USER@SERVERpassword.The service sshfs has been started.fsp_fuse_intf_Createshould trigger.mkdircall and into theopendircall.fi.fhvalue returned byopendir.fsp_fuse_intf_GetFileInfoExcall. Then step into the function call.fsp_fuse_intf_GetFileInfoFunnel.fgetattr.fi->fh. Notice that the value is the same as the one returned byopendir.fgetattr.billziss-gh commentedon Jul 1, 2019
I have created experimental commit winfsp/winfsp@3a12d92 which seems to fix this problem.
xnoreq commentedon Jul 10, 2019
Nikratio is right in that the current behavior of SSHFS is correct because FUSE guarantees:
https://libfuse.github.io/doxygen/structfuse__operations.html#a1813889bc5e6e0087a936b7abe8b923f
So it doesn't need to be an actual handle to a file and it should never be passed to any other functions than
readdir,releasedirandfsyncdir.So here:
readdir:
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fuse/fuse_intf.c#L1577
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fuse/fuse_intf.c#L1899
releasedir:
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fuse/fuse_intf.c#L953
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fuse/fuse_intf.c#L1197
fsyncdir:
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fuse/fuse_intf.c#L1338
are the only places where the
fhset by opendir should be passed to.I've looked at your commit. It looks like you just pass the
fuse_file_info*asnullwhenever it's a directory.Wouldn't it have been easier to just not set the
FileHandleif we open a directory, i.e. just don't do this if its a directory:https://github.com/billziss-gh/winfsp/blob/c97f2cb66023eccd2f87b21f5d5297ed8f1a9cd6/src/dll/fuse/fuse_intf.c#L1056
And lastly, wouldn't it be more correct to add a "DirHandle" to
struct fsp_fuse_file_descand pass that asfhtoreaddir,releasedirandfsyncdirif we're dealing with a directory?billziss-gh commentedon Jul 11, 2019
@xnoreq
I agree. It looks like I misunderstood how the
fhfield is supposed to be used for directories.The existing code already had
FileHandleandIsDirectoryfields infsp_fuse_file_desc, so I found it easier to make the change (hack) that I made. I agree that introducing aDirHandlewould have worked as well, and might lead to better code. OTOH it might require code changes in more places and I am averse to making large code changes when I am fixing bugs. Still I may consider refactoring this code in the future.BTW, in the next day or two I will be releasing a new beta of WinFsp with this fix and other improvements, so this issue should be finally resolved. Thank you for your long patience.
xnoreq commentedon Jul 11, 2019
billziss-gh commentedon Jul 11, 2019
The SSHFS-Win Beta was actually crashing for me on
mkdir(but not on any other operations). The stable SSHFS-Win releases never crashed on me (or anyone else AFAIK).Unfortunately this (WinFsp) problem was not caught by the test suites, likely because my FUSE test file systems are always written in such a way that
openandopendirreturn file handles from the same "set / space" and can therefore be used withfgetattr.In any case I have released the latest WinFsp beta. Give it a try if you want and let me know.
https://github.com/billziss-gh/winfsp/releases
I am closing this, but please reopen if the problem persists.