org.apache.lucene.index

Class LiveIndexWriterConfig

java.lang.Object

org.apache.lucene.index.LiveIndexWriterConfig

Direct Known Subclasses:

IndexWriterConfig

public class LiveIndexWriterConfig
extends Object

Holds all the configuration used by IndexWriter with few setters for settings that can be changed on an IndexWriter instance "live".

Since:

4.0

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	checkPendingFlushOnUpdate if an indexing thread should check for pending flushes on update in order to help out on a full flush
protected Codec	codec Codec used to write new segments.
protected IndexCommit	commit IndexCommit that IndexWriter is opened on.
protected boolean	commitOnClose True if calls to IndexWriter.close() should first do a commit.
protected int	createdVersionMajor Compatibility version to use for this index.
protected IndexDeletionPolicy	delPolicy IndexDeletionPolicy controlling when commit points are deleted.
protected org.apache.lucene.index.FlushPolicy	flushPolicy FlushPolicy to control when segments are flushed.
protected org.apache.lucene.index.DocumentsWriterPerThread.IndexingChain	indexingChain DocumentsWriterPerThread.IndexingChain that determines how documents are indexed.
protected Sort	indexSort The sort order to use to write merged segments.
protected Set<String>	indexSortFields The field names involved in the index sort
protected InfoStream	infoStream InfoStream for debugging messages.
protected Comparator<LeafReader>	leafSorter The comparator for sorting leaf readers.
protected long	maxFullFlushMergeWaitMillis Amount of time to wait for merges returned by MergePolicy.findFullFlushMerges(...)
protected MergePolicy	mergePolicy MergePolicy for selecting merges.
protected MergeScheduler	mergeScheduler MergeScheduler to use for running merges.
protected IndexWriterConfig.OpenMode	openMode IndexWriterConfig.OpenMode that IndexWriter is opened with.
protected int	perThreadHardLimitMB Sets the hard upper bound on RAM usage for a single segment, after which the segment is forced to flush.
protected boolean	readerPooling True if readers should be pooled.
protected Similarity	similarity Similarity to use when encoding norms.
protected String	softDeletesField soft deletes field
protected boolean	useCompoundFile True if segment flushes should use compound file format

Method Summary

Modifier and Type	Method and Description
Analyzer	getAnalyzer() Returns the default analyzer to use for indexing documents.
Codec	getCodec() Returns the current Codec.
boolean	getCommitOnClose() Returns true if IndexWriter.close() should first commit before closing.
IndexCommit	getIndexCommit() Returns the IndexCommit as specified in IndexWriterConfig.setIndexCommit(IndexCommit) or the default, null which specifies to open the latest index commit point.
int	getIndexCreatedVersionMajor() Return the compatibility version to use for this index.
IndexDeletionPolicy	getIndexDeletionPolicy() Returns the IndexDeletionPolicy specified in IndexWriterConfig.setIndexDeletionPolicy(IndexDeletionPolicy) or the default KeepOnlyLastCommitDeletionPolicy/
Sort	getIndexSort() Get the index-time Sort order, applied to all (flushed and merged) segments.
Set<String>	getIndexSortFields() Returns the field names involved in the index sort
InfoStream	getInfoStream() Returns InfoStream used for debugging.
Comparator<LeafReader>	getLeafSorter() Returns a comparator for sorting leaf readers.
int	getMaxBufferedDocs() Returns the number of buffered added documents that will trigger a flush if enabled.
long	getMaxFullFlushMergeWaitMillis() Expert: return the amount of time to wait for merges returned by by MergePolicy.findFullFlushMerges(...).
IndexWriter.IndexReaderWarmer	getMergedSegmentWarmer() Returns the current merged segment warmer.
MergePolicy	getMergePolicy() Returns the current MergePolicy in use by this writer.
MergeScheduler	getMergeScheduler() Returns the MergeScheduler that was set by IndexWriterConfig.setMergeScheduler(MergeScheduler).
IndexWriterConfig.OpenMode	getOpenMode() Returns the IndexWriterConfig.OpenMode set by IndexWriterConfig.setOpenMode(OpenMode).
double	getRAMBufferSizeMB() Returns the value set by setRAMBufferSizeMB(double) if enabled.
int	getRAMPerThreadHardLimitMB() Returns the max amount of memory each DocumentsWriterPerThread can consume until forcefully flushed.
boolean	getReaderPooling() Returns true if IndexWriter should pool readers even if DirectoryReader.open(IndexWriter) has not been called.
Similarity	getSimilarity() Expert: returns the Similarity implementation used by this IndexWriter.
String	getSoftDeletesField() Returns the soft deletes field or null if soft-deletes are disabled.
boolean	getUseCompoundFile() Returns true iff the IndexWriter packs newly written segments in a compound file.
boolean	isCheckPendingFlushOnUpdate() Expert: Returns if indexing threads check for pending flushes on update in order to help our flushing indexing buffers to disk
LiveIndexWriterConfig	setCheckPendingFlushUpdate(boolean checkPendingFlushOnUpdate) Expert: sets if indexing threads check for pending flushes on update in order to help our flushing indexing buffers to disk.
LiveIndexWriterConfig	setMaxBufferedDocs(int maxBufferedDocs) Determines the minimal number of documents required before the buffered in-memory documents are flushed as a new Segment.
LiveIndexWriterConfig	setMergedSegmentWarmer(IndexWriter.IndexReaderWarmer mergeSegmentWarmer) Set the merged segment warmer.
LiveIndexWriterConfig	setMergePolicy(MergePolicy mergePolicy) Expert: MergePolicy is invoked whenever there are changes to the segments in the index.
LiveIndexWriterConfig	setRAMBufferSizeMB(double ramBufferSizeMB) Determines the amount of RAM that may be used for buffering added documents and deletions before they are flushed to the Directory.
LiveIndexWriterConfig	setUseCompoundFile(boolean useCompoundFile) Sets if the IndexWriter should pack newly written segments in a compound file.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

delPolicy

protected volatile IndexDeletionPolicy delPolicy

IndexDeletionPolicy controlling when commit points are deleted.

commit

protected volatile IndexCommit commit

IndexCommit that IndexWriter is opened on.

openMode

protected volatile IndexWriterConfig.OpenMode openMode

IndexWriterConfig.OpenMode that IndexWriter is opened with.

createdVersionMajor

protected int createdVersionMajor

Compatibility version to use for this index.

similarity

protected volatile Similarity similarity

Similarity to use when encoding norms.

mergeScheduler

protected volatile MergeScheduler mergeScheduler

MergeScheduler to use for running merges.

indexingChain

protected volatile org.apache.lucene.index.DocumentsWriterPerThread.IndexingChain indexingChain

DocumentsWriterPerThread.IndexingChain that determines how documents are indexed.

codec

protected volatile Codec codec

Codec used to write new segments.

infoStream

protected volatile InfoStream infoStream

InfoStream for debugging messages.

mergePolicy

protected volatile MergePolicy mergePolicy

MergePolicy for selecting merges.

readerPooling

protected volatile boolean readerPooling

True if readers should be pooled.

flushPolicy

protected volatile org.apache.lucene.index.FlushPolicy flushPolicy

FlushPolicy to control when segments are flushed.

perThreadHardLimitMB

protected volatile int perThreadHardLimitMB

Sets the hard upper bound on RAM usage for a single segment, after which the segment is forced to flush.

useCompoundFile

protected volatile boolean useCompoundFile

True if segment flushes should use compound file format

commitOnClose

protected boolean commitOnClose

True if calls to IndexWriter.close() should first do a commit.

indexSort

protected Sort indexSort

The sort order to use to write merged segments.

leafSorter

protected Comparator<LeafReader> leafSorter

The comparator for sorting leaf readers.

indexSortFields

protected Set<String> indexSortFields

The field names involved in the index sort

checkPendingFlushOnUpdate

protected volatile boolean checkPendingFlushOnUpdate

if an indexing thread should check for pending flushes on update in order to help out on a full flush

softDeletesField

protected String softDeletesField

soft deletes field

maxFullFlushMergeWaitMillis

protected volatile long maxFullFlushMergeWaitMillis

Amount of time to wait for merges returned by MergePolicy.findFullFlushMerges(...)

Method Detail

getAnalyzer

public Analyzer getAnalyzer()

Returns the default analyzer to use for indexing documents.

setRAMBufferSizeMB

public LiveIndexWriterConfig setRAMBufferSizeMB(double ramBufferSizeMB)

Determines the amount of RAM that may be used for buffering added documents and deletions before they are flushed to the Directory. Generally for faster indexing performance it's best to flush by RAM usage instead of document count and use as large a RAM buffer as you can.

When this is set, the writer will flush whenever buffered documents and deletions use this much RAM. Pass in IndexWriterConfig.DISABLE_AUTO_FLUSH to prevent triggering a flush due to RAM usage. Note that if flushing by document count is also enabled, then the flush will be triggered by whichever comes first.

The maximum RAM limit is inherently determined by the JVMs available memory. Yet, an IndexWriter session can consume a significantly larger amount of memory than the given RAM limit since this limit is just an indicator when to flush memory resident documents to the Directory. Flushes are likely happen concurrently while other threads adding documents to the writer. For application stability the available memory in the JVM should be significantly larger than the RAM buffer used for indexing.

NOTE: the account of RAM usage for pending deletions is only approximate. Specifically, if you delete by Query, Lucene currently has no way to measure the RAM usage of individual Queries so the accounting will under-estimate and you should compensate by either calling commit() or refresh() periodically yourself.

NOTE: It's not guaranteed that all memory resident documents are flushed once this limit is exceeded. Depending on the configured FlushPolicy only a subset of the buffered documents are flushed and therefore only parts of the RAM buffer is released.

The default value is IndexWriterConfig.DEFAULT_RAM_BUFFER_SIZE_MB.

Takes effect immediately, but only the next time a document is added, updated or deleted.

Throws:

IllegalArgumentException - if ramBufferSize is enabled but non-positive, or it disables ramBufferSize when maxBufferedDocs is already disabled

See Also:

IndexWriterConfig.setRAMPerThreadHardLimitMB(int)

getRAMBufferSizeMB

public double getRAMBufferSizeMB()

Returns the value set by setRAMBufferSizeMB(double) if enabled.

setMaxBufferedDocs

public LiveIndexWriterConfig setMaxBufferedDocs(int maxBufferedDocs)

Determines the minimal number of documents required before the buffered in-memory documents are flushed as a new Segment. Large values generally give faster indexing.

When this is set, the writer will flush every maxBufferedDocs added documents. Pass in IndexWriterConfig.DISABLE_AUTO_FLUSH to prevent triggering a flush due to number of buffered documents. Note that if flushing by RAM usage is also enabled, then the flush will be triggered by whichever comes first.

Disabled by default (writer flushes by RAM usage).

Takes effect immediately, but only the next time a document is added, updated or deleted.

Throws:

IllegalArgumentException - if maxBufferedDocs is enabled but smaller than 2, or it disables maxBufferedDocs when ramBufferSize is already disabled

See Also:

setRAMBufferSizeMB(double)

getMaxBufferedDocs

public int getMaxBufferedDocs()

Returns the number of buffered added documents that will trigger a flush if enabled.

See Also:

setMaxBufferedDocs(int)

setMergePolicy

public LiveIndexWriterConfig setMergePolicy(MergePolicy mergePolicy)

Expert: MergePolicy is invoked whenever there are changes to the segments in the index. Its role is to select which merges to do, if any, and return a MergePolicy.MergeSpecification describing the merges. It also selects merges to do for forceMerge.

Takes effect on subsequent merge selections. Any merges in flight or any merges already registered by the previous MergePolicy are not affected.

setMergedSegmentWarmer

public LiveIndexWriterConfig setMergedSegmentWarmer(IndexWriter.IndexReaderWarmer mergeSegmentWarmer)

Set the merged segment warmer. See IndexWriter.IndexReaderWarmer.

Takes effect on the next merge.

getMergedSegmentWarmer

public IndexWriter.IndexReaderWarmer getMergedSegmentWarmer()

Returns the current merged segment warmer. See IndexWriter.IndexReaderWarmer.

getOpenMode

public IndexWriterConfig.OpenMode getOpenMode()

Returns the IndexWriterConfig.OpenMode set by IndexWriterConfig.setOpenMode(OpenMode).

getIndexCreatedVersionMajor

public int getIndexCreatedVersionMajor()

Return the compatibility version to use for this index.

See Also:

IndexWriterConfig.setIndexCreatedVersionMajor(int)

getIndexDeletionPolicy

public IndexDeletionPolicy getIndexDeletionPolicy()

Returns the IndexDeletionPolicy specified in IndexWriterConfig.setIndexDeletionPolicy(IndexDeletionPolicy) or the default KeepOnlyLastCommitDeletionPolicy/

getIndexCommit

public IndexCommit getIndexCommit()

Returns the IndexCommit as specified in IndexWriterConfig.setIndexCommit(IndexCommit) or the default, null which specifies to open the latest index commit point.

getSimilarity

public Similarity getSimilarity()

Expert: returns the Similarity implementation used by this IndexWriter.

getMergeScheduler

public MergeScheduler getMergeScheduler()

Returns the MergeScheduler that was set by IndexWriterConfig.setMergeScheduler(MergeScheduler).

getCodec

public Codec getCodec()

Returns the current Codec.

getMergePolicy

public MergePolicy getMergePolicy()

Returns the current MergePolicy in use by this writer.

See Also:

IndexWriterConfig.setMergePolicy(MergePolicy)

getReaderPooling

public boolean getReaderPooling()

Returns true if IndexWriter should pool readers even if DirectoryReader.open(IndexWriter) has not been called.

getRAMPerThreadHardLimitMB

public int getRAMPerThreadHardLimitMB()

Returns the max amount of memory each DocumentsWriterPerThread can consume until forcefully flushed.

See Also:

IndexWriterConfig.setRAMPerThreadHardLimitMB(int)

getInfoStream

public InfoStream getInfoStream()

Returns InfoStream used for debugging.

See Also:

IndexWriterConfig.setInfoStream(InfoStream)

setUseCompoundFile

public LiveIndexWriterConfig setUseCompoundFile(boolean useCompoundFile)

Sets if the IndexWriter should pack newly written segments in a compound file. Default is true.

Use false for batch indexing with very large ram buffer settings.

Note: To control compound file usage during segment merges see MergePolicy.setNoCFSRatio(double) and MergePolicy.setMaxCFSSegmentSizeMB(double). This setting only applies to newly created segments.

getUseCompoundFile

public boolean getUseCompoundFile()

Returns true iff the IndexWriter packs newly written segments in a compound file. Default is true.

getCommitOnClose

public boolean getCommitOnClose()

Returns true if IndexWriter.close() should first commit before closing.

getIndexSort

public Sort getIndexSort()

Get the index-time Sort order, applied to all (flushed and merged) segments.

getIndexSortFields

public Set<String> getIndexSortFields()

Returns the field names involved in the index sort

getLeafSorter

public Comparator<LeafReader> getLeafSorter()

Returns a comparator for sorting leaf readers. If not null, this comparator is used to sort leaf readers within DirectoryReader opened from the IndexWriter of this configuration.

Returns:

a comparator for sorting leaf readers

isCheckPendingFlushOnUpdate

public boolean isCheckPendingFlushOnUpdate()

Expert: Returns if indexing threads check for pending flushes on update in order to help our flushing indexing buffers to disk

WARNING: This API is experimental and might change in incompatible ways in the next release.

setCheckPendingFlushUpdate

public LiveIndexWriterConfig setCheckPendingFlushUpdate(boolean checkPendingFlushOnUpdate)

Expert: sets if indexing threads check for pending flushes on update in order to help our flushing indexing buffers to disk. As a consequence, threads calling DirectoryReader.openIfChanged(DirectoryReader, IndexWriter) or IndexWriter.flush() will be the only thread writing segments to disk unless flushes are falling behind. If indexing is stalled due to too many pending flushes indexing threads will help our writing pending segment flushes to disk.

WARNING: This API is experimental and might change in incompatible ways in the next release.

getSoftDeletesField

public String getSoftDeletesField()

Returns the soft deletes field or null if soft-deletes are disabled. See IndexWriterConfig.setSoftDeletesField(String) for details.

getMaxFullFlushMergeWaitMillis

public long getMaxFullFlushMergeWaitMillis()

Expert: return the amount of time to wait for merges returned by by MergePolicy.findFullFlushMerges(...). If this time is reached, we proceed with the commit based on segments merged up to that point. The merges are not cancelled, and may still run to completion independent of the commit.

toString

public String toString()

Overrides:

toString in class Object