Package org.elasticsearch.index.translog

Class Translog

java.lang.Object
org.elasticsearch.index.shard.AbstractIndexShardComponent
org.elasticsearch.index.translog.Translog
All Implemented Interfaces:
Closeable, AutoCloseable, IndexShardComponent
public class Translog extends AbstractIndexShardComponent implements IndexShardComponent, Closeable
A Translog is a per index shard component that records all non-committed index operations in a durable manner. In Elasticsearch there is one Translog instance per InternalEngine. Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY with each commit to ensure a strong association between the lucene index an the transaction log file. This UUID is used to prevent accidental recovery from a transaction log that belongs to a different engine.

Each Translog has only one translog file open for writes at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current translog's file generation, its fsynced offset in bytes, and other important statistics.

When the current translog file reaches a certain size (IndexSettings.INDEX_TRANSLOG_GENERATION_THRESHOLD_SIZE_SETTING, or when a clear separation between old and new operations (upon change in primary term), the current file is reopened for read only and a new write only file is created. Any non-current, read only translog file always has a translog-${gen}.ckp associated with it which is an fsynced copy of its last translog.ckp such that in disaster recovery last fsynced offsets, number of operation etc. are still preserved.

  • Field Details

  • Constructor Details

    • Translog

      public Translog(TranslogConfig config, String translogUUID, TranslogDeletionPolicy deletionPolicy, LongSupplier globalCheckpointSupplier, LongSupplier primaryTermSupplier, LongConsumer persistedSequenceNumberConsumer) throws IOException
      Creates a new Translog instance. This method will create a new transaction log unless the given Translog.TranslogGeneration is null. If the generation is null this method is destructive and will delete all files in the translog path given. If the generation is not null, this method tries to open the given translog generation. The generation is treated as the last generation referenced from already committed data. This means all operations that have not yet been committed should be in the translog file referenced by this generation. The translog creation will fail if this generation can't be opened.
      Parameters:
      config - the configuration of this translog
      translogUUID - the translog uuid to open, null for a new translog
      deletionPolicy - an instance of TranslogDeletionPolicy that controls when a translog file can be safely deleted
      globalCheckpointSupplier - a supplier for the global checkpoint
      primaryTermSupplier - a supplier for the latest value of primary term of the owning index shard. The latest term value is examined and stored in the header whenever a new generation is rolled. It's guaranteed from outside that a new generation is rolled when the term is increased. This guarantee allows to us to validate and reject operation whose term is higher than the primary term stored in the translog header.
      persistedSequenceNumberConsumer - a callback that's called whenever an operation with a given sequence number is successfully persisted.
      Throws:
      IOException

  • Method Details

    • parseIdFromFileName

      public static long parseIdFromFileName(Path translogFile)
      Extracts the translog generation from a file name.
      Throws:
      IllegalArgumentException - if the path doesn't match the expected pattern.

    • isOpen

      public boolean isOpen()
      Returns true if this Translog is still open.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • location

      public Path location()
      Returns all translog locations as absolute paths. These paths don't contain actual translog files they are directories holding the transaction logs.

    • currentFileGeneration

      public long currentFileGeneration()
      Returns the generation of the current transaction log.

    • getMinFileGeneration

      public long getMinFileGeneration()
      Returns the minimum file generation referenced by the translog

    • totalOperations

      public int totalOperations()
      Returns the number of operations in the translog files

    • sizeInBytes

      public long sizeInBytes()
      Returns the size in bytes of the v files

    • totalOperationsByMinGen

      public int totalOperationsByMinGen(long minGeneration)
      Returns the number of operations in the translog files at least the given generation

    • estimateTotalOperationsFromMinSeq

      public int estimateTotalOperationsFromMinSeq(long minSeqNo)
      Returns the number of operations in the transaction files that contain operations with seq# above the given number.

    • sizeInBytesByMinGen

      public long sizeInBytesByMinGen(long minGeneration)
      Returns the size in bytes of the translog files at least the given generation

    • add

      public Translog.Location add(Translog.Operation operation) throws IOException
      Adds an operation to the transaction log.
      Parameters:
      operation - the operation to add
      Returns:
      the location of the operation in the translog
      Throws:
      IOException - if adding the operation to the translog resulted in an I/O exception

    • shouldRollGeneration

      public boolean shouldRollGeneration()
      Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
      Returns:
      true if the current generation should be rolled to a new generation

    • getLastWriteLocation

      public Translog.Location getLastWriteLocation()
      The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write.

    • getLastSyncedGlobalCheckpoint

      public long getLastSyncedGlobalCheckpoint()
      The last synced checkpoint for this translog.
      Returns:
      the last synced checkpoint

    • newSnapshot

      public Translog.Snapshot newSnapshot() throws IOException
      Throws:
      IOException

    • newSnapshot

      public Translog.Snapshot newSnapshot(long fromSeqNo, long toSeqNo) throws IOException
      Creates a new translog snapshot containing operations from the given range.
      Parameters:
      fromSeqNo - the lower bound of the range (inclusive)
      toSeqNo - the upper bound of the range (inclusive)
      Returns:
      the new snapshot
      Throws:
      IOException

    • readOperation

      public Translog.Operation readOperation(Translog.Location location) throws IOException
      Reads and returns the operation from the given location if the generation it references is still available. Otherwise this method will return null.
      Throws:
      IOException

    • acquireRetentionLock

      public Closeable acquireRetentionLock()
      Acquires a lock on the translog files, preventing them from being trimmed

    • sync

      public void sync() throws IOException
      Sync's the translog.
      Throws:
      IOException

    • syncNeeded

      public boolean syncNeeded()
      Returns true if an fsync is required to ensure durability of the translogs operations or it's metadata.

    • getFilename

      public static String getFilename(long generation)
      package private for testing

    • trimOperations

      public void trimOperations(long belowTerm, long aboveSeqNo) throws IOException
      Trims translog for terms of files below belowTerm and seq# above aboveSeqNo. Effectively it moves max visible seq# Checkpoint.trimmedAboveSeqNo therefore TranslogSnapshot skips those operations.
      Throws:
      IOException

    • ensureSynced

      public boolean ensureSynced(Translog.Location location) throws IOException
      Ensures that the given location has be synced / written to the underlying storage.
      Returns:
      Returns true iff this call caused an actual sync operation otherwise false
      Throws:
      IOException

    • ensureSynced

      public boolean ensureSynced(Stream<Translog.Location> locations) throws IOException
      Ensures that all locations in the given stream have been synced / written to the underlying storage. This method allows for internal optimization to minimize the amount of fsync operations if multiple locations must be synced.
      Returns:
      Returns true iff this call caused an actual sync operation otherwise false
      Throws:
      IOException

    • closeOnTragicEvent

      protected void closeOnTragicEvent(Exception ex)
      Closes the translog if the current translog writer experienced a tragic exception. Note that in case this thread closes the translog it must not already be holding a read lock on the translog as it will acquire a write lock in the course of closing the translog
      Parameters:
      ex - if an exception occurs closing the translog, it will be suppressed into the provided exception

    • stats

      public TranslogStats stats()
      return stats

    • getConfig

      public TranslogConfig getConfig()

    • getDeletionPolicy

      public TranslogDeletionPolicy getDeletionPolicy()

    • readOperations

      public static List<Translog.Operation> readOperations(StreamInput input, String source) throws IOException
      Reads a list of operations written with writeOperations(StreamOutput, List)
      Throws:
      IOException

    • writeOperations

      public static void writeOperations(StreamOutput outStream, List<Translog.Operation> toWrite) throws IOException
      Writes all operations in the given iterable to the given output stream including the size of the array use readOperations(StreamInput, String) to read it back.
      Throws:
      IOException

    • writeOperationNoSize

      public static void writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op) throws IOException
      Throws:
      IOException

    • getMinGenerationForSeqNo

      public Translog.TranslogGeneration getMinGenerationForSeqNo(long seqNo)
      Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if there is no generation that could any such sequence number.
      Parameters:
      seqNo - the sequence number
      Returns:
      the minimum generation for the sequence number

    • rollGeneration

      public void rollGeneration() throws IOException
      Roll the current translog generation into a new generation if it's not empty. This does not commit the translog.
      Throws:
      IOException - if an I/O exception occurred during any file operations

    • trimUnreferencedReaders

      public void trimUnreferencedReaders() throws IOException
      Trims unreferenced translog generations by asking TranslogDeletionPolicy for the minimum required generation
      Throws:
      IOException

    • getGeneration

      public Translog.TranslogGeneration getGeneration()
      Returns the current generation of this translog. This corresponds to the latest uncommitted translog generation

    • getTragicException

      public Exception getTragicException()
      If this Translog was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception. Otherwise (no tragic exception has occurred) it returns null.

    • readGlobalCheckpoint

      public static long readGlobalCheckpoint(Path location, String expectedTranslogUUID) throws IOException
      Reads the sequence numbers global checkpoint from the translog checkpoint. This ensures that the translogUUID from this translog matches with the provided translogUUID.
      Parameters:
      location - the location of the translog
      Returns:
      the global checkpoint
      Throws:
      IOException - if an I/O exception occurred reading the checkpoint
      TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid

    • readMinTranslogGeneration

      public static long readMinTranslogGeneration(Path location, String expectedTranslogUUID) throws IOException
      Returns the minimum translog generation retained by the translog at the given location. This ensures that the translogUUID from this translog matches with the provided translogUUID.
      Parameters:
      location - the location of the translog
      Returns:
      the minimum translog generation
      Throws:
      IOException - if an I/O exception occurred reading the checkpoint
      TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid

    • getTranslogUUID

      public String getTranslogUUID()
      Returns the translog uuid used to associate a lucene index with a translog.

    • getMaxSeqNo

      public long getMaxSeqNo()
      Returns the max seq_no of translog operations found in this translog. Since this value is calculated based on the current existing readers, this value is not necessary to be the max seq_no of all operations have been stored in this translog.

    • createEmptyTranslog

      public static String createEmptyTranslog(Path location, long initialGlobalCheckpoint, ShardId shardId, long primaryTerm) throws IOException
      Throws:
      IOException

    • createEmptyTranslog

      public static String createEmptyTranslog(Path location, ShardId shardId, long initialGlobalCheckpoint, long primaryTerm, @Nullable String translogUUID, @Nullable ChannelFactory factory) throws IOException
      Creates a new empty translog within the specified location that contains the given initialGlobalCheckpoint, primaryTerm and translogUUID. This method should be used directly under specific circumstances like for shards that will see no indexing. Specifying a non-unique translog UUID could cause a lot of issues and that's why in all (but one) cases the method createEmptyTranslog(Path, long, ShardId, long) should be used instead.
      Parameters:
      location - a Path to the directory that will contains the translog files (translog + translog checkpoint)
      shardId - the ShardId
      initialGlobalCheckpoint - the global checkpoint to initialize the translog with
      primaryTerm - the shard's primary term to initialize the translog with
      translogUUID - the unique identifier to initialize the translog with
      factory - a ChannelFactory used to open translog files
      Returns:
      the translog's unique identifier
      Throws:
      IOException - if something went wrong during translog creation