I'm using python's ftplib to transfer lots and lots of data (~100 files X 2GB) across a local network to an FTP server. This code is running on Ubuntu. Here is my call (self is my FtpClient object, which is a wrapper around ftplib client):
# Store file.
self.ftpClient.storbinary('STOR ' + destination, fileHandle, blocksize = self.blockSize, callback = self.__UpdateFileTransferProgress)
My question is, how do I choose an optimal block size? My understanding is that the optimal block size is dependent on a number of things, not the least of which are connection speed and latency. My code will be running on many different networks with different speeds and varying amounts of congestion throughout the day. Ideally, I would like to compute the optimal block size at run time.
Would the optimal FTP transfer block size be the same as the optimal TCP window size? If this is true, and TCP window scaling is turned on, is there a way to get the optimal TCP window size from the kernel? How/when does the linux kernel determine optimal window size? Ideally I could ask the linux kernel for the optimal block size, so as to avoid reinventing the wheel.
TCP_CORK
option.