Package org.elasticsearch.index.shard

Class IndexShard

java.lang.Object
org.elasticsearch.index.shard.AbstractIndexShardComponent
org.elasticsearch.index.shard.IndexShard
All Implemented Interfaces:
IndexShardComponent, IndicesClusterStateService.Shard
public class IndexShard extends AbstractIndexShardComponent implements IndicesClusterStateService.Shard

  • Field Details

  • Constructor Details

  • Method Details

    • getThreadPool

      public ThreadPool getThreadPool()

    • store

      public Store store()

    • getIndexSort

      public org.apache.lucene.search.Sort getIndexSort()
      Return the sort order of this index, or null if the index has no sort.

    • getService

      public ShardGetService getService()

    • shardBitsetFilterCache

      public ShardBitsetFilterCache shardBitsetFilterCache()

    • mapperService

      public MapperService mapperService()

    • getSearchOperationListener

      public SearchOperationListener getSearchOperationListener()

    • warmerService

      public ShardIndexWarmerService warmerService()

    • requestCache

      public ShardRequestCache requestCache()

    • fieldData

      public ShardFieldData fieldData()

    • isSystem

      public boolean isSystem()

    • getPendingPrimaryTerm

      public long getPendingPrimaryTerm()
      USE THIS METHOD WITH CARE! Returns the primary term the index shard is supposed to be on. In case of primary promotion or when a replica learns about a new term due to a new primary, the term that's exposed here will not be the term that the shard internally uses to assign to operations. The shard will auto-correct its internal operation term, but this might take time. See IndexMetadata.primaryTerm(int)

    • getOperationPrimaryTerm

      public long getOperationPrimaryTerm()
      Returns the primary term that is currently being used to assign to operations

    • routingEntry

      public ShardRouting routingEntry()
      Returns the latest cluster routing entry received with this shard.
      Specified by:
      routingEntry in interface IndicesClusterStateService.Shard

    • getQueryCachingPolicy

      public org.apache.lucene.search.QueryCachingPolicy getQueryCachingPolicy()

    • updateShardState

      public void updateShardState(ShardRouting newRouting, long newPrimaryTerm, BiConsumer<IndexShard,ActionListener<PrimaryReplicaSyncer.ResyncTask>> primaryReplicaSyncer, long applyingClusterStateVersion, Set<String> inSyncAllocationIds, IndexShardRoutingTable routingTable) throws IOException
      Description copied from interface: IndicesClusterStateService.Shard
      Updates the shard state based on an incoming cluster state: - Updates and persists the new routing value. - Updates the primary term if this shard is a primary. - Updates the allocation ids that are tracked by the shard if it is a primary. See ReplicationTracker.updateFromMaster(long, Set, IndexShardRoutingTable) for details.
      Specified by:
      updateShardState in interface IndicesClusterStateService.Shard
      Parameters:
      newRouting - the new routing entry
      newPrimaryTerm - the new primary term
      primaryReplicaSyncer - the primary-replica resync action to trigger when a term is increased on a primary
      applyingClusterStateVersion - the cluster state version being applied when updating the allocation IDs from the master
      inSyncAllocationIds - the allocation ids of the currently in-sync shard copies
      routingTable - the shard routing table
      Throws:
      IOException - if shard state could not be persisted

    • markAsRecovering

      public IndexShardState markAsRecovering(String reason, RecoveryState recoveryState) throws IndexShardStartedException, IndexShardRelocatedException, IndexShardRecoveringException, IndexShardClosedException
      Marks the shard as recovering based on a recovery state, fails with exception is recovering is not allowed to be set.
      Throws:
      IndexShardStartedException
      IndexShardRelocatedException
      IndexShardRecoveringException
      IndexShardClosedException

    • relocated

      public void relocated(String targetAllocationId, BiConsumer<ReplicationTracker.PrimaryContext,ActionListener<Void>> consumer, ActionListener<Void> listener) throws IllegalIndexShardStateException, IllegalStateException
      Completes the relocation. Operations are blocked and current operations are drained before changing state to relocated. The provided BiConsumer is executed after all operations are successfully blocked.
      Parameters:
      consumer - a BiConsumer that is executed after operations are blocked and that consumes the primary context as well as a listener to resolve once it finished
      listener - listener to resolve once this method actions including executing consumer in the non-failure case complete
      Throws:
      IllegalIndexShardStateException - if the shard is not relocating due to concurrent cancellation
      IllegalStateException - if the relocation target is no longer part of the replication group

    • state

      public IndexShardState state()
      Description copied from interface: IndicesClusterStateService.Shard
      Returns the latest internal shard state.
      Specified by:
      state in interface IndicesClusterStateService.Shard

    • applyIndexOperationOnPrimary

      public Engine.IndexResult applyIndexOperationOnPrimary(long version, VersionType versionType, SourceToParse sourceToParse, long ifSeqNo, long ifPrimaryTerm, long autoGeneratedTimestamp, boolean isRetry) throws IOException
      Throws:
      IOException

    • applyIndexOperationOnReplica

      public Engine.IndexResult applyIndexOperationOnReplica(long seqNo, long opPrimaryTerm, long version, long autoGeneratedTimeStamp, boolean isRetry, SourceToParse sourceToParse) throws IOException
      Throws:
      IOException

    • prepareIndex

      public static Engine.Index prepareIndex(MapperService mapperService, String type, SourceToParse source, long seqNo, long primaryTerm, long version, VersionType versionType, Engine.Operation.Origin origin, long autoGeneratedIdTimestamp, boolean isRetry, long ifSeqNo, long ifPrimaryTerm)

    • markSeqNoAsNoop

      public Engine.NoOpResult markSeqNoAsNoop(long seqNo, long opPrimaryTerm, String reason) throws IOException
      Throws:
      IOException

    • getFailedIndexResult

      public Engine.IndexResult getFailedIndexResult(Exception e, long version)

    • getFailedDeleteResult

      public Engine.DeleteResult getFailedDeleteResult(Exception e, long version)

    • applyDeleteOperationOnPrimary

      public Engine.DeleteResult applyDeleteOperationOnPrimary(long version, String type, String id, VersionType versionType, long ifSeqNo, long ifPrimaryTerm) throws IOException
      Throws:
      IOException

    • applyDeleteOperationOnReplica

      public Engine.DeleteResult applyDeleteOperationOnReplica(long seqNo, long opPrimaryTerm, long version, String type, String id) throws IOException
      Throws:
      IOException

    • get

      public Engine.GetResult get(Engine.Get get)

    • refresh

      public void refresh(String source)
      Writes all indexing changes to disk and opens a new searcher reflecting all changes. This can throw AlreadyClosedException.

    • getWritingBytes

      public long getWritingBytes()
      Returns how many bytes we are currently moving from heap to disk

    • refreshStats

      public RefreshStats refreshStats()

    • flushStats

      public FlushStats flushStats()

    • docStats

      public DocsStats docStats()

    • commitStats

      public CommitStats commitStats()
      Returns:
      CommitStats
      Throws:
      org.apache.lucene.store.AlreadyClosedException - if shard is closed

    • seqNoStats

      public SeqNoStats seqNoStats()
      Returns:
      SeqNoStats
      Throws:
      org.apache.lucene.store.AlreadyClosedException - if shard is closed

    • indexingStats

      public IndexingStats indexingStats(String... types)

    • searchStats

      public SearchStats searchStats(String... groups)

    • getStats

      public GetStats getStats()

    • storeStats

      public StoreStats storeStats()

    • mergeStats

      public MergeStats mergeStats()

    • segmentStats

      public SegmentsStats segmentStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments)

    • warmerStats

      public WarmerStats warmerStats()

    • fieldDataStats

      public FieldDataStats fieldDataStats(String... fields)

    • translogStats

      public TranslogStats translogStats()

    • completionStats

      public CompletionStats completionStats(String... fields)

    • syncFlush

      public Engine.SyncedFlushResult syncFlush(String syncId, Engine.CommitId expectedCommitId)

    • flush

      public Engine.CommitId flush(FlushRequest request)
      Executes the given flush request against the engine.
      Parameters:
      request - the flush request
      Returns:
      the commit ID

    • trimTranslog

      public void trimTranslog()
      checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details

    • rollTranslogGeneration

      public void rollTranslogGeneration()
      Rolls the tranlog generation and cleans unneeded.

    • forceMerge

      public void forceMerge(ForceMergeRequest forceMerge) throws IOException
      Throws:
      IOException

    • upgrade

      public org.apache.lucene.util.Version upgrade(UpgradeRequest upgrade) throws IOException
      Upgrades the shard to the current version of Lucene and returns the minimum segment version
      Throws:
      IOException

    • minimumCompatibleVersion

      public org.apache.lucene.util.Version minimumCompatibleVersion()

    • acquireLastIndexCommit

      public Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineException
      Creates a new IndexCommit snapshot from the currently running engine. All resources referenced by this commit won't be freed until the commit / snapshot is closed.
      Parameters:
      flushFirst - true if the index should first be flushed to disk / a low level lucene commit should be executed
      Throws:
      EngineException

    • acquireIndexCommitForSnapshot

      public Engine.IndexCommitRef acquireIndexCommitForSnapshot() throws EngineException
      Acquires the IndexCommit which should be included in a snapshot.
      Throws:
      EngineException

    • acquireSafeIndexCommit

      public Engine.IndexCommitRef acquireSafeIndexCommit() throws EngineException
      Snapshots the most recent safe index commit from the currently running engine. All index files referenced by this index commit won't be freed until the commit/snapshot is closed.
      Throws:
      EngineException

    • snapshotStoreMetadata

      public Store.MetadataSnapshot snapshotStoreMetadata() throws IOException
      gets a Store.MetadataSnapshot for the current directory. This method is safe to call in all lifecycle of the index shard, without having to worry about the current state of the engine and concurrent flushes.
      Throws:
      org.apache.lucene.index.IndexNotFoundException - if no index is found in the current directory
      org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.
      org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.
      org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.
      FileNotFoundException - if one or more files referenced by a commit are not present.
      NoSuchFileException - if one or more files referenced by a commit are not present.
      IOException

    • failShard

      public void failShard(String reason, @Nullable Exception e)
      Fails the shard and marks the shard store as corrupted if e is caused by index corruption

    • acquireSearcherSupplier

      public Engine.SearcherSupplier acquireSearcherSupplier()
      Acquires a point-in-time reader that can be used to create Engine.Searchers on demand.

    • acquireSearcherSupplier

      public Engine.SearcherSupplier acquireSearcherSupplier(Engine.SearcherScope scope)
      Acquires a point-in-time reader that can be used to create Engine.Searchers on demand.

    • acquireSearcher

      public Engine.Searcher acquireSearcher(String source)

    • close

      public void close(String reason, boolean flushEngine) throws IOException
      Throws:
      IOException

    • preRecovery

      public void preRecovery()

    • postRecovery

      public void postRecovery(String reason) throws IndexShardStartedException, IndexShardRelocatedException, IndexShardClosedException
      Throws:
      IndexShardStartedException
      IndexShardRelocatedException
      IndexShardClosedException

    • prepareForIndexRecovery

      public void prepareForIndexRecovery()
      called before starting to copy index files over

    • recoverLocallyUpToGlobalCheckpoint

      public long recoverLocallyUpToGlobalCheckpoint()
      A best effort to bring up this shard to the global checkpoint using the local translog before performing a peer recovery.
      Returns:
      a sequence number that an operation-based peer recovery can start with. This is the first operation after the local checkpoint of the safe commit if exists.

    • trimOperationOfPreviousPrimaryTerms

      public void trimOperationOfPreviousPrimaryTerms(long aboveSeqNo)

    • getMaxSeenAutoIdTimestamp

      public long getMaxSeenAutoIdTimestamp()
      Returns the maximum auto_id_timestamp of all append-only requests have been processed by this shard or the auto_id_timestamp received from the primary via updateMaxUnsafeAutoIdTimestamp(long) at the beginning of a peer-recovery or a primary-replica resync.
      See Also:
      updateMaxUnsafeAutoIdTimestamp(long)

    • updateMaxUnsafeAutoIdTimestamp

      public void updateMaxUnsafeAutoIdTimestamp(long maxSeenAutoIdTimestampFromPrimary)
      Since operations stored in soft-deletes do not have max_auto_id_timestamp, the primary has to propagate its max_auto_id_timestamp (via getMaxSeenAutoIdTimestamp() of all processed append-only requests to replicas at the beginning of a peer-recovery or a primary-replica resync to force a replica to disable optimization for all append-only requests which are replicated via replication while its retry variants are replicated via recovery without auto_id_timestamp.

      Without this force-update, a replica can generate duplicate documents (for the same id) if it first receives a retry append-only (without timestamp) via recovery, then an original append-only (with timestamp) via replication.

    • applyTranslogOperation

      public Engine.Result applyTranslogOperation(Translog.Operation operation, Engine.Operation.Origin origin) throws IOException
      Throws:
      IOException

    • openEngineAndRecoverFromTranslog

      public void openEngineAndRecoverFromTranslog() throws IOException
      opens the engine on top of the existing lucene engine and translog. Operations from the translog will be replayed to bring lucene up to date.
      Throws:
      IOException

    • openEngineAndSkipTranslogRecovery

      public void openEngineAndSkipTranslogRecovery() throws IOException
      Opens the engine on top of the existing lucene engine and translog. The translog is kept but its operations won't be replayed.
      Throws:
      IOException

    • performRecoveryRestart

      public void performRecoveryRestart() throws IOException
      called if recovery has to be restarted after network error / delay **
      Throws:
      IOException

    • resetRecoveryStage

      public void resetRecoveryStage()
      If a file-based recovery occurs, a recovery target calls this method to reset the recovery stage.

    • recoveryStats

      public RecoveryStats recoveryStats()
      returns stats about ongoing recoveries, both source and target

    • recoveryState

      public RecoveryState recoveryState()
      Returns the current RecoveryState if this shard is recovering or has been recovering. Returns null if the recovery has not yet started or shard was not recovered (created via an API).
      Specified by:
      recoveryState in interface IndicesClusterStateService.Shard

    • getTimestampRange

      public ShardLongFieldRange getTimestampRange()
      Specified by:
      getTimestampRange in interface IndicesClusterStateService.Shard
      Returns:
      the range of the @timestamp field for this shard, or ShardLongFieldRange.EMPTY if this field is not found, or ShardLongFieldRange.UNKNOWN if its range is not fixed.

    • finalizeRecovery

      public void finalizeRecovery()
      perform the last stages of recovery once all translog operations are done. note that you should still call postRecovery(String).

    • ignoreRecoveryAttempt

      public boolean ignoreRecoveryAttempt()
      Returns true if this shard can ignore a recovery attempt made to it (since the already doing/done it)

    • readAllowed

      public void readAllowed() throws IllegalIndexShardStateException
      Throws:
      IllegalIndexShardStateException

    • isReadAllowed

      public boolean isReadAllowed()
      returns true if the IndexShardState allows reading

    • verifyActive

      protected final void verifyActive() throws IllegalIndexShardStateException
      Throws:
      IllegalIndexShardStateException

    • getIndexBufferRAMBytesUsed

      public long getIndexBufferRAMBytesUsed()
      Returns number of heap bytes used by the indexing buffer for this shard, or 0 if the shard is closed

    • addShardFailureCallback

      public void addShardFailureCallback(Consumer<IndexShard.ShardFailure> onShardFailure)

    • checkIdle

      public void checkIdle(long inactiveTimeNS)
      Called by IndexingMemoryController to check whether more than inactiveTimeNS has passed since the last indexing operation, and notify listeners that we are now inactive so e.g. sync'd flush can happen.

    • isActive

      public boolean isActive()

    • shardPath

      public ShardPath shardPath()

    • recoverFromLocalShards

      public void recoverFromLocalShards(BiConsumer<String,MappingMetadata> mappingUpdateConsumer, List<IndexShard> localShards, ActionListener<Boolean> listener) throws IOException
      Throws:
      IOException

    • recoverFromStore

      public void recoverFromStore(ActionListener<Boolean> listener)

    • restoreFromRepository

      public void restoreFromRepository(Repository repository, ActionListener<Boolean> listener)

    • onSettingsChanged

      public void onSettingsChanged()

    • acquireHistoryRetentionLock

      public Closeable acquireHistoryRetentionLock(Engine.HistorySource source)
      Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed

    • estimateNumberOfHistoryOperations

      public int estimateNumberOfHistoryOperations(String reason, Engine.HistorySource source, long startingSeqNo) throws IOException
      Returns the estimated number of history operations whose seq# at least the provided seq# in this shard.
      Throws:
      IOException

    • getHistoryOperations

      public Translog.Snapshot getHistoryOperations(String reason, Engine.HistorySource source, long startingSeqNo) throws IOException
      Creates a new history snapshot for reading operations since the provided starting seqno (inclusive). The returned snapshot can be retrieved from either Lucene index or translog files.
      Throws:
      IOException

    • hasCompleteHistoryOperations

      public boolean hasCompleteHistoryOperations(String reason, Engine.HistorySource source, long startingSeqNo) throws IOException
      Checks if we have a completed history of operations since the given starting seqno (inclusive). This method should be called after acquiring the retention lock; See acquireHistoryRetentionLock(Engine.HistorySource)
      Throws:
      IOException

    • getMinRetainedSeqNo

      public long getMinRetainedSeqNo()
      Gets the minimum retained sequence number for this engine.
      Returns:
      the minimum retained sequence number

    • newChangesSnapshot

      public Translog.Snapshot newChangesSnapshot(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange) throws IOException
      Creates a new changes snapshot for reading operations whose seq_no are between fromSeqNo(inclusive) and toSeqNo(inclusive). The caller has to close the returned snapshot after finishing the reading.
      Parameters:
      source - the source of the request
      fromSeqNo - the from seq_no (inclusive) to read
      toSeqNo - the to seq_no (inclusive) to read
      requiredFullRange - if true then Translog.Snapshot.next() will throw IllegalStateException if any operation between fromSeqNo and toSeqNo is missing. This parameter should be only enabled when the entire requesting range is below the global checkpoint.
      Throws:
      IOException

    • segments

      public List<Segment> segments(boolean verbose)

    • getHistoryUUID

      public String getHistoryUUID()

    • getIndexEventListener

      public IndexEventListener getIndexEventListener()

    • activateThrottling

      public void activateThrottling()

    • deactivateThrottling

      public void deactivateThrottling()

    • writeIndexingBuffer

      public void writeIndexingBuffer()
      Called when our shard is using too much heap and should move buffered indexed/deleted documents to disk.

    • updateLocalCheckpointForShard

      public void updateLocalCheckpointForShard(String allocationId, long checkpoint)
      Notifies the service to update the local checkpoint for the shard with the provided allocation ID. See ReplicationTracker.updateLocalCheckpoint(String, long) for details.
      Parameters:
      allocationId - the allocation ID of the shard to update the local checkpoint for
      checkpoint - the local checkpoint for the shard

    • updateGlobalCheckpointForShard

      public void updateGlobalCheckpointForShard(String allocationId, long globalCheckpoint)
      Update the local knowledge of the persisted global checkpoint for the specified allocation ID.
      Parameters:
      allocationId - the allocation ID to update the global checkpoint for
      globalCheckpoint - the global checkpoint

    • addGlobalCheckpointListener

      public void addGlobalCheckpointListener(long waitingForGlobalCheckpoint, GlobalCheckpointListeners.GlobalCheckpointListener listener, org.elasticsearch.core.TimeValue timeout)
      Add a global checkpoint listener. If the global checkpoint is equal to or above the global checkpoint the listener is waiting for, then the listener will be notified immediately via an executor (so possibly not on the current thread). If the specified timeout elapses before the listener is notified, the listener will be notified with an TimeoutException. A caller may pass null to specify no timeout.
      Parameters:
      waitingForGlobalCheckpoint - the global checkpoint the listener is waiting for
      listener - the listener
      timeout - the timeout

    • getRetentionLeases

      public RetentionLeases getRetentionLeases()
      Get all retention leases tracked on this shard.
      Returns:
      the retention leases

    • getRetentionLeases

      public RetentionLeases getRetentionLeases(boolean expireLeases)
      If the expire leases parameter is false, gets all retention leases tracked on this shard and otherwise first calculates expiration of existing retention leases, and then gets all non-expired retention leases tracked on this shard. Note that only the primary shard calculates which leases are expired, and if any have expired, syncs the retention leases to any replicas. If the expire leases parameter is true, this replication tracker must be in primary mode.
      Returns:
      the non-expired retention leases

    • getRetentionLeaseStats

      public RetentionLeaseStats getRetentionLeaseStats()

    • addRetentionLease

      public RetentionLease addRetentionLease(String id, long retainingSequenceNumber, String source, ActionListener<ReplicationResponse> listener)
      Adds a new retention lease.
      Parameters:
      id - the identifier of the retention lease
      retainingSequenceNumber - the retaining sequence number
      source - the source of the retention lease
      listener - the callback when the retention lease is successfully added and synced to replicas
      Returns:
      the new retention lease
      Throws:
      IllegalArgumentException - if the specified retention lease already exists

    • renewRetentionLease

      public RetentionLease renewRetentionLease(String id, long retainingSequenceNumber, String source)
      Renews an existing retention lease.
      Parameters:
      id - the identifier of the retention lease
      retainingSequenceNumber - the retaining sequence number
      source - the source of the retention lease
      Returns:
      the renewed retention lease
      Throws:
      IllegalArgumentException - if the specified retention lease does not exist

    • removeRetentionLease

      public void removeRetentionLease(String id, ActionListener<ReplicationResponse> listener)
      Removes an existing retention lease.
      Parameters:
      id - the identifier of the retention lease
      listener - the callback when the retention lease is successfully removed and synced to replicas

    • updateRetentionLeasesOnReplica

      public void updateRetentionLeasesOnReplica(RetentionLeases retentionLeases)
      Updates retention leases on a replica.
      Parameters:
      retentionLeases - the retention leases

    • loadRetentionLeases

      public RetentionLeases loadRetentionLeases() throws IOException
      Loads the latest retention leases from their dedicated state file.
      Returns:
      the retention leases
      Throws:
      IOException - if an I/O exception occurs reading the retention leases

    • persistRetentionLeases

      public void persistRetentionLeases() throws WriteStateException
      Persists the current retention leases to their dedicated state file.
      Throws:
      WriteStateException - if an exception occurs writing the state file

    • assertRetentionLeasesPersisted

      public boolean assertRetentionLeasesPersisted() throws IOException
      Throws:
      IOException

    • syncRetentionLeases

      public void syncRetentionLeases()
      Syncs the current retention leases to all replicas.

    • initiateTracking

      public void initiateTracking(String allocationId)
      Called when the recovery process for a shard has opened the engine on the target shard. Ensures that the right data structures have been set up locally to track local checkpoint information for the shard and that the shard is added to the replication group.
      Parameters:
      allocationId - the allocation ID of the shard for which recovery was initiated

    • markAllocationIdAsInSync

      public void markAllocationIdAsInSync(String allocationId, long localCheckpoint) throws InterruptedException
      Marks the shard with the provided allocation ID as in-sync with the primary shard. See ReplicationTracker.markAllocationIdAsInSync(String, long) for additional details.
      Parameters:
      allocationId - the allocation ID of the shard to mark as in-sync
      localCheckpoint - the current local checkpoint on the shard
      Throws:
      InterruptedException

    • getLocalCheckpoint

      public long getLocalCheckpoint()
      Returns the persisted local checkpoint for the shard.
      Returns:
      the local checkpoint

    • getLastKnownGlobalCheckpoint

      public long getLastKnownGlobalCheckpoint()
      Returns the global checkpoint for the shard.
      Returns:
      the global checkpoint

    • getLastSyncedGlobalCheckpoint

      public long getLastSyncedGlobalCheckpoint()
      Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)

    • getInSyncGlobalCheckpoints

      public com.carrotsearch.hppc.ObjectLongMap<String> getInSyncGlobalCheckpoints()
      Get the local knowledge of the global checkpoints for all in-sync allocation IDs.
      Returns:
      a map from allocation ID to the local knowledge of the global checkpoint for that allocation ID

    • maybeSyncGlobalCheckpoint

      public void maybeSyncGlobalCheckpoint(String reason)
      Syncs the global checkpoint to the replicas if the global checkpoint on at least one replica is behind the global checkpoint on the primary.

    • getReplicationGroup

      public ReplicationGroup getReplicationGroup()
      Returns the current replication group for the shard.
      Returns:
      the replication group

    • getPendingReplicationActions

      public PendingReplicationActions getPendingReplicationActions()
      Returns the pending replication actions for the shard.
      Returns:
      the pending replication actions

    • updateGlobalCheckpointOnReplica

      public void updateGlobalCheckpointOnReplica(long globalCheckpoint, String reason)
      Updates the global checkpoint on a replica shard after it has been updated by the primary.
      Parameters:
      globalCheckpoint - the global checkpoint
      reason - the reason the global checkpoint was updated

    • addCleanFilesDependency

      public Runnable addCleanFilesDependency()
      Creates a Runnable that must be executed before the clean files step in peer recovery can complete.
      Returns:
      runnable that must be executed during the clean files step in peer recovery

    • afterCleanFiles

      public void afterCleanFiles(Runnable runnable)
      Execute a Runnable on the generic pool once all dependencies added via addCleanFilesDependency() have finished. If there are no dependencies to wait for then the Runnable will be executed on the calling thread.

    • outstandingCleanFilesConditions

      public int outstandingCleanFilesConditions()

    • activateWithPrimaryContext

      public void activateWithPrimaryContext(ReplicationTracker.PrimaryContext primaryContext)
      Updates the known allocation IDs and the local checkpoints for the corresponding allocations from a primary relocation source.
      Parameters:
      primaryContext - the sequence number context

    • pendingInSync

      public boolean pendingInSync()
      Check if there are any recoveries pending in-sync.
      Returns:
      true if there is at least one shard pending in-sync, otherwise false

    • noopUpdate

      public void noopUpdate(String type)
      Should be called for each no-op update operation to increment relevant statistics.
      Parameters:
      type - the doc type of the update

    • maybeCheckIndex

      public void maybeCheckIndex()

    • getEngineOrNull

      protected Engine getEngineOrNull()
      NOTE: returns null if engine is not yet started (e.g. recovery phase 1, copying over index files, is still running), or if engine is closed.

    • startRecovery

      public void startRecovery(RecoveryState recoveryState, PeerRecoveryTargetService recoveryTargetService, PeerRecoveryTargetService.RecoveryListener recoveryListener, RepositoriesService repositoriesService, BiConsumer<String,MappingMetadata> mappingUpdateConsumer, IndicesService indicesService)

    • isRelocatedPrimary

      public boolean isRelocatedPrimary()
      Returns whether the shard is a relocated primary, i.e. not in charge anymore of replicating changes (see ReplicationTracker).

    • addPeerRecoveryRetentionLease

      public RetentionLease addPeerRecoveryRetentionLease(String nodeId, long globalCheckpoint, ActionListener<ReplicationResponse> listener)

    • cloneLocalPeerRecoveryRetentionLease

      public RetentionLease cloneLocalPeerRecoveryRetentionLease(String nodeId, ActionListener<ReplicationResponse> listener)

    • removePeerRecoveryRetentionLease

      public void removePeerRecoveryRetentionLease(String nodeId, ActionListener<ReplicationResponse> listener)

    • getPeerRecoveryRetentionLeases

      public List<RetentionLease> getPeerRecoveryRetentionLeases()
      Returns a list of retention leases for peer recovery installed in this shard copy.

    • useRetentionLeasesInPeerRecovery

      public boolean useRetentionLeasesInPeerRecovery()

    • buildIndexAnalyzer

      public static org.apache.lucene.analysis.Analyzer buildIndexAnalyzer(MapperService mapperService)

    • acquirePrimaryOperationPermit

      public void acquirePrimaryOperationPermit(ActionListener<org.elasticsearch.core.Releasable> onPermitAcquired, String executorOnDelay, Object debugInfo)
      Acquire a primary operation permit whenever the shard is ready for indexing. If a permit is directly available, the provided ActionListener will be called on the calling thread. During relocation hand-off, permit acquisition can be delayed. The provided ActionListener will then be called using the provided executor.
      Parameters:
      debugInfo - an extra information that can be useful when tracing an unreleased permit. When assertions are enabled the tracing will capture the supplied object's Object.toString() value. Otherwise the object isn't used

    • acquirePrimaryOperationPermit

      public void acquirePrimaryOperationPermit(ActionListener<org.elasticsearch.core.Releasable> onPermitAcquired, String executorOnDelay, Object debugInfo, boolean forceExecution)

    • acquireAllPrimaryOperationsPermits

      public void acquireAllPrimaryOperationsPermits(ActionListener<org.elasticsearch.core.Releasable> onPermitAcquired, org.elasticsearch.core.TimeValue timeout)
      Acquire all primary operation permits. Once all permits are acquired, the provided ActionListener is called. It is the responsibility of the caller to close the Releasable.

    • runUnderPrimaryPermit

      public void runUnderPrimaryPermit(Runnable runnable, Consumer<Exception> onFailure, String executorOnDelay, Object debugInfo)
      Runs the specified runnable under a permit and otherwise calling back the specified failure callback. This method is really a convenience for acquirePrimaryOperationPermit(ActionListener, String, Object) where the listener equates to try-with-resources closing the releasable after executing the runnable on successfully acquiring the permit, an otherwise calling back the failure callback.
      Parameters:
      runnable - the runnable to execute under permit
      onFailure - the callback on failure
      executorOnDelay - the executor to execute the runnable on if permit acquisition is blocked
      debugInfo - debug info

    • acquireReplicaOperationPermit

      public void acquireReplicaOperationPermit(long opPrimaryTerm, long globalCheckpoint, long maxSeqNoOfUpdatesOrDeletes, ActionListener<org.elasticsearch.core.Releasable> onPermitAcquired, String executorOnDelay, Object debugInfo)
      Acquire a replica operation permit whenever the shard is ready for indexing (see acquirePrimaryOperationPermit(ActionListener, String, Object)). If the given primary term is lower than then one in shardRouting, the ActionListener.onFailure(Exception) method of the provided listener is invoked with an IllegalStateException. If permit acquisition is delayed, the listener will be invoked on the executor with the specified name.
      Parameters:
      opPrimaryTerm - the operation primary term
      globalCheckpoint - the global checkpoint associated with the request
      maxSeqNoOfUpdatesOrDeletes - the max seq_no of updates (index operations overwrite Lucene) or deletes captured on the primary after this replication request was executed on it (see getMaxSeqNoOfUpdatesOrDeletes()
      onPermitAcquired - the listener for permit acquisition
      executorOnDelay - the name of the executor to invoke the listener on if permit acquisition is delayed
      debugInfo - an extra information that can be useful when tracing an unreleased permit. When assertions are enabled the tracing will capture the supplied object's Object.toString() value. Otherwise the object isn't used

    • acquireAllReplicaOperationsPermits

      public void acquireAllReplicaOperationsPermits(long opPrimaryTerm, long globalCheckpoint, long maxSeqNoOfUpdatesOrDeletes, ActionListener<org.elasticsearch.core.Releasable> onPermitAcquired, org.elasticsearch.core.TimeValue timeout)
      Acquire all replica operation permits whenever the shard is ready for indexing (see acquireAllPrimaryOperationsPermits(ActionListener, TimeValue). If the given primary term is lower than then one in shardRouting, the ActionListener.onFailure(Exception) method of the provided listener is invoked with an IllegalStateException.
      Parameters:
      opPrimaryTerm - the operation primary term
      globalCheckpoint - the global checkpoint associated with the request
      maxSeqNoOfUpdatesOrDeletes - the max seq_no of updates (index operations overwrite Lucene) or deletes captured on the primary after this replication request was executed on it (see getMaxSeqNoOfUpdatesOrDeletes()
      onPermitAcquired - the listener for permit acquisition
      timeout - the maximum time to wait for the in-flight operations block

    • getActiveOperationsCount

      public int getActiveOperationsCount()
      Obtain the active operation count, or OPERATIONS_BLOCKED if all permits are held (even if there are outstanding operations in flight).
      Returns:
      the active operation count, or OPERATIONS_BLOCKED when all permits are held.

    • getActiveOperations

      public List<String> getActiveOperations()
      Returns:
      a list of describing each permit that wasn't released yet. The description consist of the debugInfo supplied when the permit was acquired plus a stack traces that was captured when the permit was request.

    • sync

      public final void sync(Translog.Location location, Consumer<Exception> syncListener)
      Syncs the given location with the underlying storage unless already synced. This method might return immediately without actually fsyncing the location until the sync listener is called. Yet, unless there is already another thread fsyncing the transaction log the caller thread will be hijacked to run the fsync for all pending fsync operations. This method allows indexing threads to continue indexing without blocking on fsync calls. We ensure that there is only one thread blocking on the sync an all others can continue indexing. NOTE: if the syncListener throws an exception when it's processed the exception will only be logged. Users should make sure that the listener handles all exception cases internally.

    • sync

      public void sync() throws IOException
      Throws:
      IOException

    • isSyncNeeded

      public boolean isSyncNeeded()
      Checks if the underlying storage sync is required.

    • getTranslogDurability

      public Translog.Durability getTranslogDurability()
      Returns the current translog durability mode

    • afterWriteOperation

      public void afterWriteOperation()
      Schedules a flush or translog generation roll if needed but will not schedule more than one concurrently. The operation will be executed asynchronously on the flush thread pool.

    • scheduledRefresh

      public boolean scheduledRefresh()
      Executes a scheduled refresh if necessary.
      Returns:
      true iff the engine got refreshed otherwise false

    • isSearchIdle

      public final boolean isSearchIdle()
      Returns true if this shards is search idle

    • hasRefreshPending

      public final boolean hasRefreshPending()
      Returns true if this shard has some scheduled refresh that is pending because of search-idle.

    • awaitShardSearchActive

      public final void awaitShardSearchActive(Consumer<Boolean> listener)
      Registers the given listener and invokes it once the shard is active again and all pending refresh translog location has been refreshed. If there is no pending refresh location registered the listener will be invoked immediately.
      Parameters:
      listener - the listener to invoke once the pending refresh location is visible. The listener will be called with true if the listener was registered to wait for a refresh.

    • addRefreshListener

      public void addRefreshListener(Translog.Location location, Consumer<Boolean> listener)
      Add a listener for refreshes.
      Parameters:
      location - the location to listen for
      listener - for the refresh. Called with true if registering the listener ran it out of slots and forced a refresh. Called with false otherwise.

    • getMaxSeqNoOfUpdatesOrDeletes

      public long getMaxSeqNoOfUpdatesOrDeletes()
      Returns the maximum sequence number of either update or delete operations have been processed in this shard or the sequence number from advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.

      The primary captures this value after executes a replication request, then transfers it to a replica before executing that replication request on a replica.

    • advanceMaxSeqNoOfUpdatesOrDeletes

      public void advanceMaxSeqNoOfUpdatesOrDeletes(long seqNo)
      A replica calls this method to advance the max_seq_no_of_updates marker of its engine to at least the max_seq_no_of_updates value (piggybacked in a replication request) that it receives from its primary before executing that replication request. The receiving value is at least as high as the max_seq_no_of_updates on the primary was when any of the operations of that replication request were processed on it.

      A replica shard also calls this method to bootstrap the max_seq_no_of_updates marker with the value that it received from the primary in peer-recovery, before it replays remote translog operations from the primary. The receiving value is at least as high as the max_seq_no_of_updates on the primary was when any of these operations were processed on it.

      These transfers guarantee that every index/delete operation when executing on a replica engine will observe this marker a value which is at least the value of the max_seq_no_of_updates marker on the primary after that operation was executed on the primary.

      See Also:
      acquireReplicaOperationPermit(long, long, long, ActionListener, String, Object), RecoveryTarget.indexTranslogOperations(List, int, long, long, RetentionLeases, long, ActionListener)

    • verifyShardBeforeIndexClosing

      public void verifyShardBeforeIndexClosing() throws IllegalStateException
      Performs the pre-closing checks on the IndexShard.
      Throws:
      IllegalStateException - if the sanity checks failed