org.apache.lucene.analysis.cjk

Class CJKBigramFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.lucene.analysis.cjk.CJKBigramFilter

All Implemented Interfaces:

Closeable, AutoCloseable

public final class CJKBigramFilter
extends TokenFilter

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

CJK types are set by these tokenizers, but you can also use CJKBigramFilter(TokenStream, int) to explicitly control which of the CJK scripts are turned into bigrams.

By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the outputUnigrams flag in CJKBigramFilter(TokenStream, int, boolean). This can be used for a combined unigram+bigram approach.

Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries. Korean Hangul characters are treated the same as many other scripts' letters, and as a result, StandardTokenizer can produce tokens that mix Hangul and non-Hangul characters, e.g. "abc". Such mixed-script tokens are typed as <ALPHANUM> rather than <HANGUL>, and as a result, will not be converted to bigrams by CJKBigramFilter. In all cases, all non-CJK input is passed thru unmodified.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
static String	DOUBLE_TYPE when we emit a bigram, it's then marked as this type
static int	HAN bigram flag for Han Ideographs
static int	HANGUL bigram flag for Hangul
static int	HIRAGANA bigram flag for Hiragana
static int	KATAKANA bigram flag for Katakana
static String	SINGLE_TYPE when we emit a unigram, it's then marked as this type

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
CJKBigramFilter(TokenStream in) Calls CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)
CJKBigramFilter(TokenStream in, int flags) Calls CJKBigramFilter(in, flags, false)
CJKBigramFilter(TokenStream in, int flags, boolean outputUnigrams) Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

Method Summary

Modifier and Type	Method and Description
boolean	incrementToken()
void	reset()

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

HAN

public static final int HAN

bigram flag for Han Ideographs

See Also:

Constant Field Values

HIRAGANA

public static final int HIRAGANA

bigram flag for Hiragana

See Also:

Constant Field Values

KATAKANA

public static final int KATAKANA

bigram flag for Katakana

See Also:

Constant Field Values

HANGUL

public static final int HANGUL

bigram flag for Hangul

See Also:

Constant Field Values

DOUBLE_TYPE

public static final String DOUBLE_TYPE

when we emit a bigram, it's then marked as this type

See Also:

Constant Field Values

SINGLE_TYPE

public static final String SINGLE_TYPE

when we emit a unigram, it's then marked as this type

See Also:

Constant Field Values

Constructor Detail

CJKBigramFilter

public CJKBigramFilter(TokenStream in)

Calls CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)

CJKBigramFilter

public CJKBigramFilter(TokenStream in,
                       int flags)

Calls CJKBigramFilter(in, flags, false)

CJKBigramFilter

public CJKBigramFilter(TokenStream in,
                       int flags,
                       boolean outputUnigrams)

Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

Parameters:

flags - OR'ed set from HAN, HIRAGANA, KATAKANA, HANGUL

outputUnigrams - true if unigrams for the selected writing systems should also be output. when this is false, this is only done when there are no adjacent characters to form a bigram.

Method Detail

incrementToken

public boolean incrementToken()
                       throws IOException

Specified by:

incrementToken in class TokenStream

Throws:

IOException

reset

public void reset()
           throws IOException

Overrides:

reset in class TokenFilter

Throws:

IOException