How to transfer using FTP streams

One of the advantages of integrating FTP functionality directly into a product rather than using stand-alone FTP applications is that data can be transferred directly to and from memory. This is particularly useful when transferring dynamic content needs, such as the results of database queries and other application data.

FTPConnection offers two alternatives for memory transfers: byte-arrays and streams. Byte- arrays are generally easier to deal with but do not facilitate streaming. In other words, the data must be fully generated and stored in a byte-array before being transferred. This is fine when memory usage is not an issue, but if it is necessary to limit memory usage then streams should be used.

Byte-Array Transfers are performed using the DownloadByteArray and UploadByteArray methods. They simply take the byte-array and the remote file-name as a parameter.

Stream Transfers allow the programmer to delay creation of the data to be transferred until it is ready to be transferred. For example, the data to be transferred might be the result of a database query that could return millions of rows. In such a case, storing it all in memory at once would be inefficient. To circumvent this problem a custom stream may be written which uses a cursor to iterate through the query result as the file is being transferred.

To transfer streams, the methods DownloadStream and UploadStream should be used. If a suitable Stream is not available from the standard libraries a new one may be created by extending the System.IO.Stream class and overriding the appropriate read and/or write methods.

The following code-snippet shows code and pseudo-code that illustrates the use of a custom stream to upload the results of a query to an FTP server:

SqlDataStream dbStr = new SqlDataStream(sqlDataReader);
ftpConnection.UploadStream(dbStr, remoteFileName);
.
.
.
public class SqlDataStream : Stream
{
   private MemoryStream buffer = new MemoryStream();

   public override int Read(byte[] bytes, int offset, int count)
   {
      if (not enough bytes in buffer)
         ReadFromDatabase(count)
      int numBytes = buffer.Read(bytes, 0, count);
      remove read bytes from buffer
   }

   private void ReadFromDatabase(int count)
   {
      1. write remainder of fields from current record to the buffer
      2. read sufficient records from the database such that there are
         at least 'count' bytes in the buffer
   }