451 "Broken pipe" when reading large directories with edtFTPj

Question

We've been using edtFTPj successfully for years. However, recently we ran into "451 Transfer aborted. Broken pipe" errors when performing "dir" on an FTP server. The reason was that the FTP directory contained an unusual amount of files (about 2000). We solved our problem on short term by temporarily moving away files from the FTP server and subsequently feeding them back in "digestable" chunks of 200.

We turned on tracing and reproduced the problem to find the cause of the "451 Transfer aborted. Broken pipe" errors. Client and server logs are provided here: http://pastebin.com/vL2YtDXb. According to our analysis, this is what happens:

1. edtFTPj successfully connects to the FTP server. It opens a control port (21) which will remain open for the entire FTP session. 
2. Our application wants to list the FTP directory, however at maximum 10 files. This is implemented like this:
   1. Application calls com.enterprisedt.net.ftp.FileTransferClient.directoryList() passing an own DirectoryListCallback implementation (DirCallback.java).
   2. edtFTPj sets off the DIR command and receives the directory contents over a data socket (cf. com.enterprisedt.net.ftp.FTPClient.dir())
   3. FTP server writes directory contents into the data socket, each entry is one line
   4. FTPClient.dir() reads the data socket line by line and uses the application's DirectoryListCallback implementation ("DirCallback") to process them (cf. while loop at FTPClient.java:3806)
      1. DirCallback calls DirectoryListArgument.abortListing() after the 10th entry; this makes FTPClient.dir() leave the while loop
      2. FTPClient.dir() closes the data socket and reads the FTP Reply code from the control socket, expecting one of 226 or 250 (FTPClient.java:3831 and 3837)
         ==> normally everything is fine and the server sends 226 or 250
         ==> but if there are "too many" files on the FTP server, it sends a 451 (Transfer aborted. Broken pipe). 

The 451 error seems to occur when ever edtFTPj closes the data socket before the server managed to write all the directory contents into the data stream. The larger the directory contents, the more likely the 451 error occurs. In fact, if we increase the maximum size of files to read from 10 to 1000, we can successfully read large directories.

==> A possible solution for this problem is to make FTPClient.dir() accept 451 (addtitionally to 226 and 250) as a valid code after closing the data socket (FTPClient:3835). Maybe this could be configurable. 

This feature is necessary, because the current design doesn't allow us to solve the problem ourselves: We could subclass FTPClient.java and override dir(), but we coundn't use it in FileTransferClient.java (private field "ftpClient" without setter).

 

Facts:

• FTP client: edtFTPj-2.4.0.jar (OS: AIX 6.1)
• FTP Server: ProFTPD 1.3.4b (maint) (OS: GNU/Linux x86_64))

Logs: http://pastebin.com/vL2YtDXb

 

Answer 1

So you only want 10 files from the listing? How do you know what files you want?

Comments

I was thinking a little more about this - why don't you just catch the exception and check if the error code is 451?

---

I only want to tolerate 451 if it occurs at FTPClient:3836, but not if it occurs elsewhere, e.g. at FTPClient:3790. So catching the FTPException won't really solve my problem.

I've managed to build edtFTPj. I edited it so that I can introduce an own, overridden FTPClient in FileTransferClient. However, subclassing and overriding FTPClient won't work, because the dir()-method accesses private members ("cancelTransfer" and "lister") and methods ("closeDataSocket()") that are not available to my subclassed FTPClient.
Even if if it were possible (e.g. by adding getters to FTPClient), the solution with subclassing and overriding has several drawbacks:
1. The solution duplicates a lot of code (dir() is about 90 line long!). Upgrading edtFTPj in future becomes error-prone, because we need to verify that the duplicated code in our FTPClient still matches the code in the new edtFTPj's FTPClient.
2. FTPClient.dir() is private, so it's necessary to overide all callers of dir() aswell (dir(String, boolean), dirDetails(String,DirectoryListCallback)) to call the overridden version of dir().
3. Every other user of edtFTPj potentially has the same problem slumbering when reading large directories with an aborting DirectoryListCallback.

Dealing with 451 in FTPClient directly would be a much simpler, sustainable solution.

---

You know when you've called abortListing, so if you get a 451 directly after this in the dir method, why can't you just ignore it?

---

Actually because we want separation of concerns in our design: On one side we have the application code that wants to list a directory by calling edtFTPj (FileTransferClient.directoryList()). On the other side wie have our DirectoryListCallback-implementation which is called by edtFTPj and deals with processing directory results. The first part should not need any knowledge about how directory processing occurs.

However, I admit that it is a possible solution to try/catch the FTPException in our application, check if it is a 451 and ask our DirectoryListCallback-implementation if it requested to abort. If both is true, the application can continue. Thanks for that idea. I'll try that.

---

The try/catch solution worked for us. Thank you.

I still think this is a workaround solution. Really this case should be handled by edtFTPj internally, because edtFTPj is prematurely closing the FTP data socket.