Package org.elasticsearch.indices.recovery

Class MultiChunkTransfer<Source,Request extends MultiChunkTransfer.ChunkRequest>

java.lang.Object

org.elasticsearch.indices.recovery.MultiChunkTransfer<Source,Request>

All Implemented Interfaces:

Closeable, AutoCloseable

public abstract class MultiChunkTransfer<Source,Request extends MultiChunkTransfer.ChunkRequest>
extends Object
implements Closeable

File chunks are sent/requested sequentially by at most one thread at any time. However, the sender/requestor won't wait for the response before processing the next file chunk request to reduce the recovery time especially on secure/compressed or high latency communication.

The sender/requestor can send up to maxConcurrentFileChunks file chunk requests without waiting for responses. Since the recovery target can receive file chunks out of order, it has to buffer those file chunks in memory and only flush to disk when there's no gap. To ensure the recover target never buffers more than maxConcurrentFileChunks file chunks, we allow the sender/requestor to send only up to maxConcurrentFileChunks file chunk requests from the last flushed (and acknowledged) file chunk. We leverage the local checkpoint tracker for this purpose. We generate a new sequence number and assign it to each file chunk request before sending; then mark that sequence number as processed when we receive a response for the corresponding file chunk request. With the local checkpoint tracker, we know the last acknowledged-flushed file-chunk is a file chunk whose requestSeqId equals to the local checkpoint because the recover target can flush all file chunks up to the local checkpoint.

When the number of un-replied file chunk requests reaches the limit (i.e. the gap between the max_seq_no and the local checkpoint is greater than maxConcurrentFileChunks), the sending/requesting thread will abort its execution. That process will be resumed by one of the networking threads which receive/handle the responses of the current pending file chunk requests. This process will continue until all chunk requests are sent/responded.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	MultiChunkTransfer.ChunkRequest

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	MultiChunkTransfer(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<Void> listener, int maxConcurrentChunks, List<Source> sources)

Method Summary

Modifier and Type	Method	Description
protected boolean	assertOnSuccess()
protected abstract void	executeChunkRequest(Request request, ActionListener<Void> listener)
protected abstract void	handleError(Source resource, Exception e)
protected abstract Request	nextChunkRequest(Source resource)
protected void	onNewResource(Source resource)	This method is called when starting sending/requesting a new source.
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Constructor Details

MultiChunkTransfer

protected MultiChunkTransfer(org.apache.logging.log4j.Logger logger,
 ThreadContext threadContext,
 ActionListener<Void> listener,
 int maxConcurrentChunks,
 List<Source> sources)

Method Details

start

public final void start()

assertOnSuccess

protected boolean assertOnSuccess()

onNewResource

protected void onNewResource(Source resource)
                      throws IOException

This method is called when starting sending/requesting a new source. Subclasses should override this method to reset the file offset or close the previous file and open a new file if needed.

Throws:

IOException

nextChunkRequest

protected abstract Request nextChunkRequest(Source resource)
                                     throws IOException

Throws:

IOException

executeChunkRequest

protected abstract void executeChunkRequest(Request request,
 ActionListener<Void> listener)

handleError

protected abstract void handleError(Source resource,
 Exception e)
                             throws Exception

Throws:

Exception