User:P.A./Download networking improvements

This page contains observations related to the network access aspects of downloads. It was last updated in April 2013.

Identification of corrupt HTTP downloads

With regard to individual HTTP requests over the network, there are several cases that may result in truncated downloads.

1. At some point, no more TCP packets are received

   This may occur because of several causes, among which the easiest to replicate is disconnecting the network cable. When the network cable is connected again or a temporary network unavailability is resolved, the TCP connection might be able to resume automatically. The timeout for this automatic reconnection depends on the server and the network topology, but might be hours.

   Thus, in this case we may not want to terminate the download, but leave it in progress waiting for the connection to resume. The user may still cancel the download manually if required. Alternatively, we may also be able to detect the physical link state on the machine, and use this to interrupt downloads if the link is unavailable for enough time, and restart them when the link is available again.

2. A TCP RST packet is received, indicating an abruptly terminated connection

   This results in the channel being closed with the NS_ERROR_NET_RESET code. On mobile, data we already downloaded is immediately discarded.

   We are now able to have automated tests for this case, thanks to a feature added to the Necko socket code.

3. A TCP FIN packet is received, but not all the data has been received

   This is the recommended way for a server to terminate the connection, even if data is truncated, since an RST packet isn't guaranteed to be received by the client. Thus, we must check the actual response to understand if it is complete.

   1. The response has Transfer-Encoding: chunked, and we receive no termination chunk

      Incorrect server implementations of chunked encoding, or custom server-side scripts, may omit the zero-length last-chunk even if the download is complete.

      Thus, we may not want to treat this as an error condition. We already have code to detect this condition inside the channel, but we don't currently expose the condition.

   2. The response has Transfer-Encoding: chunked, and we receive only part of the data in the most recent chunk

      This generally indicates a truncated connection, but we currently ignore this condition and consider the partially received data to be vaild.

      We can add code to detect this condition, however it is unclear whether this should become an error for all requests, only for downloads, or never.

   3. The response has no Transfer-Encoding header, but a Content-Length header is specified

      Even if we have a Content-Length, incorrect server implementations might provide the wrong value for some file types, though usually the difference is small.

      In case the length is different, it could be possible to retry the request with a byte range, if the server supports it, to find out if the download is really finished.

   4. The response has no Transfer-Encoding and no Content-Length headers

      In this case, we must necessarily assume that the download is finished. We may add integrity checks based on content types, and if they indicate a potentially truncated download, try a byte range request to see if more data is available.

We may also want to add integrity checks based on content types to downloads we think succeeded, assuming the checks happen on a background thread, and retry or mark those downloads in the user interface as well.

Progress notifications

The Content-Length header indicates the length of the response body, and progress events dispatched by nsIProgressEventSink correctly indicate how may bytes of the response body were transferred.

However, when the request has a Content-Encoding header and we opt to decode the data we receive from the channel, the data received by the registered nsIStreamListener and written to the local file is already decoded, so the file size will not reflect the progress events.

Since decoding or decompression cannot resume from the middle of the compressed data, resuming this type of downloads should be disallowed. Currently, we may incorrectly use byte range requests when retrying this type of downloads, based on the size of the decoded file.

Network cache

The current network cache is able to store a partial response, in case the connection was terminated midway by the client or the server. This is not required for downloads, as our implementation handles this case using .part files.

Off-main-thread OnDataAvailable

The JavaScript API for downloads contains an intermediate object implementing nsIStreamListener, placed between the source channel and the BackgroundFileSaver object. This prevents OnDataAvailable to be dispatched to another thread. We may move the intermediate object to a C++ component to allow off-main-thread dispatching.

Progress events notified through nsIProgressEventSink will be dispatched to the main thread. We currently have code to ensure those events aren't dispached after OnStopRequest, however this could start to happen when OnDataAvailable moves to another thread. The Downloads API is already able to handle this case and ignore late progress events.

Note that if a new listener was injected by JavaScript code before OnStartRequest is called, using nsITraceableChannel during the http-on-examine-response notification, then OnDataAvailable cannot happen off-main-thread in any case.