public class NRTCachingDirectory extends FilterDirectory implements Accountable
RAMDirectory
around any provided delegate directory, to
be used during NRT search.
This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index").toPath());
NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0);
IndexWriterConfig conf = new IndexWriterConfig(analyzer);
IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges
whose expected segment size is <= 5 MB, unless the net
cached bytes exceeds 60 MB at which point all writes will
not be cached (until the net bytes falls below 60 MB).
inNULL_ACCOUNTABLE| Constructor and Description |
|---|
NRTCachingDirectory(Directory delegate,
double maxMergeSizeMB,
double maxCachedMB)
We will cache a newly created output if 1) it's a
flush or a merge and the estimated size of the merged segment is
<= maxMergeSizeMB, and 2) the total cached bytes is
<= maxCachedMB |
| Modifier and Type | Method and Description |
|---|---|
void |
close()
Close this directory, which flushes any cached files
to the delegate and then closes the delegate.
|
IndexOutput |
createOutput(String name,
IOContext context)
Creates a new, empty file in the directory and returns an
IndexOutput
instance for appending data to this file. |
IndexOutput |
createTempOutput(String prefix,
String suffix,
IOContext context)
Creates a new, empty, temporary file in the directory and returns an
IndexOutput
instance for appending data to this file. |
void |
deleteFile(String name)
Removes an existing file in the directory.
|
protected boolean |
doCacheWrite(String name,
IOContext context)
Subclass can override this to customize logic; return
true if this file should be written to the RAMDirectory.
|
long |
fileLength(String name)
Returns the byte length of a file in the directory.
|
Collection<Accountable> |
getChildResources()
Returns nested resources of this class.
|
String[] |
listAll()
Returns names of all files stored in this directory.
|
String[] |
listCachedFiles() |
IndexInput |
openInput(String name,
IOContext context)
Opens a stream for reading an existing file.
|
long |
ramBytesUsed()
Return the memory usage of this object in bytes.
|
void |
rename(String source,
String dest)
Renames
source file to dest file where
dest must not already exist in the directory. |
void |
sync(Collection<String> fileNames)
Ensures that any writes to these files are moved to
stable storage (made durable).
|
String |
toString() |
getDelegate, getPendingDeletions, obtainLock, syncMetaData, unwrapcopyFrom, ensureOpen, getTempFileName, openChecksumInputpublic NRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
<= maxMergeSizeMB, and 2) the total cached bytes is
<= maxCachedMBpublic String toString()
toString in class FilterDirectorypublic String[] listAll() throws IOException
DirectoryString.compareTo(java.lang.String)) order.listAll in class FilterDirectoryIOException - in case of I/O errorpublic void deleteFile(String name) throws IOException
DirectoryNoSuchFileException or FileNotFoundException
if name points to a non-existing file.deleteFile in class FilterDirectoryname - the name of an existing file.IOException - in case of I/O errorpublic long fileLength(String name) throws IOException
DirectoryNoSuchFileException or FileNotFoundException
if name points to a non-existing file.fileLength in class FilterDirectoryname - the name of an existing file.IOException - in case of I/O errorpublic String[] listCachedFiles()
public IndexOutput createOutput(String name, IOContext context) throws IOException
DirectoryIndexOutput
instance for appending data to this file.
This method must throw FileAlreadyExistsException if the file
already exists.createOutput in class FilterDirectoryname - the name of the file to create.IOException - in case of I/O errorpublic void sync(Collection<String> fileNames) throws IOException
Directorysync in class FilterDirectoryIOExceptionDirectory.syncMetaData()public void rename(String source, String dest) throws IOException
Directorysource file to dest file where
dest must not already exist in the directory.
It is permitted for this operation to not be truly atomic, for example
both source and dest can be visible temporarily in Directory.listAll().
However, the implementation of this method must ensure the content of
dest appears as the entire source atomically. So once
dest is visible for readers, the entire content of previous source
is visible.
This method is used by IndexWriter to publish commits.rename in class FilterDirectoryIOExceptionpublic IndexInput openInput(String name, IOContext context) throws IOException
DirectoryNoSuchFileException or FileNotFoundException
if name points to a non-existing file.openInput in class FilterDirectoryname - the name of an existing file.IOException - in case of I/O errorpublic void close()
throws IOException
close in interface Closeableclose in interface AutoCloseableclose in class FilterDirectoryIOExceptionprotected boolean doCacheWrite(String name, IOContext context)
public IndexOutput createTempOutput(String prefix, String suffix, IOContext context) throws IOException
DirectoryIndexOutput
instance for appending data to this file.
The temporary file name (accessible via IndexOutput.getName()) will start with
prefix, end with suffix and have a reserved file extension .tmp.createTempOutput in class FilterDirectoryIOExceptionpublic long ramBytesUsed()
AccountableramBytesUsed in interface Accountablepublic Collection<Accountable> getChildResources()
AccountablegetChildResources in interface AccountableAccountablesCopyright © 2000-2021 Apache Software Foundation. All Rights Reserved.