org.apache.lucene.search.similarities

Class BM25Similarity

java.lang.Object

org.apache.lucene.search.similarities.Similarity

org.apache.lucene.search.similarities.BM25Similarity

public class BM25Similarity
extends Similarity

BM25 Similarity. Introduced in Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 1994). Gaithersburg, USA, November 1994.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity

Similarity.SimScorer

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	discountOverlaps True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.

Constructor Summary

Constructors

Constructor and Description
BM25Similarity() BM25 with these default values: k1 = 1.2 b = 0.75
BM25Similarity(float k1, float b) BM25 with the supplied parameter values.

Method Summary

Modifier and Type	Method and Description
protected float	avgFieldLength(CollectionStatistics collectionStats) The default implementation computes the average as sumTotalTermFreq / docCount
long	computeNorm(FieldInvertState state) Computes the normalization value for a field, given the accumulated state of term processing for this field (see FieldInvertState).
float	getB() Returns the b parameter
boolean	getDiscountOverlaps() Returns true if overlap tokens are discounted from the document's length.
float	getK1() Returns the k1 parameter
protected float	idf(long docFreq, long docCount) Implemented as log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5)).
Explanation	idfExplain(CollectionStatistics collectionStats, TermStatistics termStats) Computes a score factor for a simple term and returns an explanation for that score factor.
Explanation	idfExplain(CollectionStatistics collectionStats, TermStatistics[] termStats) Computes a score factor for a phrase.
protected float	scorePayload(int doc, int start, int end, BytesRef payload) The default implementation returns 1
Similarity.SimScorer	scorer(float boost, CollectionStatistics collectionStats, TermStatistics... termStats) Compute any collection-level weight (e.g.
void	setDiscountOverlaps(boolean v) Sets whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

discountOverlaps

protected boolean discountOverlaps

True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.

Constructor Detail

BM25Similarity

public BM25Similarity(float k1,
                      float b)

BM25 with the supplied parameter values.

Parameters:

k1 - Controls non-linear term frequency normalization (saturation).

b - Controls to what degree document length normalizes tf values.

Throws:

IllegalArgumentException - if k1 is infinite or negative, or if b is not within the range [0..1]

BM25Similarity

public BM25Similarity()

BM25 with these default values:

• k1 = 1.2
• b = 0.75

Method Detail

idf

protected float idf(long docFreq,
                    long docCount)

Implemented as log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5)).

scorePayload

protected float scorePayload(int doc,
                             int start,
                             int end,
                             BytesRef payload)

The default implementation returns 1

avgFieldLength

protected float avgFieldLength(CollectionStatistics collectionStats)

The default implementation computes the average as sumTotalTermFreq / docCount

setDiscountOverlaps

public void setDiscountOverlaps(boolean v)

Sets whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm. By default this is true, meaning overlap tokens do not count when computing norms.

getDiscountOverlaps

public boolean getDiscountOverlaps()

Returns true if overlap tokens are discounted from the document's length.

See Also:

setDiscountOverlaps(boolean)

computeNorm

public final long computeNorm(FieldInvertState state)

Description copied from class: Similarity

Computes the normalization value for a field, given the accumulated state of term processing for this field (see FieldInvertState).

Matches in longer fields are less precise, so implementations of this method usually set smaller values when state.getLength() is large, and larger values when state.getLength() is small.

Note that for a given term-document frequency, greater unsigned norms must produce scores that are lower or equal, ie. for two encoded norms n1 and n2 so that Long.compareUnsigned(n1, n2) > 0 then SimScorer.score(freq, n1) <= SimScorer.score(freq, n2) for any legal freq.

0 is not a legal norm, so 1 is the norm that produces the highest scores.

Specified by:

computeNorm in class Similarity

Parameters:

state - current processing state for this field

Returns:

computed norm value

idfExplain

public Explanation idfExplain(CollectionStatistics collectionStats,
                              TermStatistics termStats)

Computes a score factor for a simple term and returns an explanation for that score factor.

The default implementation uses:

 idf(docFreq, docCount);
 

Note that CollectionStatistics.docCount() is used instead of IndexReader#numDocs() because also TermStatistics.docFreq() is used, and when the latter is inaccurate, so is CollectionStatistics.docCount(), and in the same direction. In addition, CollectionStatistics.docCount() does not skew when fields are sparse.

Parameters:

collectionStats - collection-level statistics

termStats - term-level statistics for the term

Returns:

an Explain object that includes both an idf score factor and an explanation for the term.

idfExplain

public Explanation idfExplain(CollectionStatistics collectionStats,
                              TermStatistics[] termStats)

Computes a score factor for a phrase.

The default implementation sums the idf factor for each term in the phrase.

Parameters:

collectionStats - collection-level statistics

termStats - term-level statistics for the terms in the phrase

Returns:

an Explain object that includes both an idf score factor for the phrase and an explanation for each term.

scorer

public final Similarity.SimScorer scorer(float boost,
                                         CollectionStatistics collectionStats,
                                         TermStatistics... termStats)

Description copied from class: Similarity

Compute any collection-level weight (e.g. IDF, average document length, etc) needed for scoring a query.

Specified by:

scorer in class Similarity

Parameters:

boost - a multiplicative factor to apply to the produces scores

collectionStats - collection-level statistics, such as the number of tokens in the collection.

termStats - term-level statistics, such as the document frequency of a term across the collection.

Returns:

SimWeight object with the information this Similarity needs to score a query.

toString

public String toString()

Overrides:

toString in class Object

getK1

public final float getK1()

Returns the k1 parameter

See Also:

BM25Similarity(float, float)

getB

public final float getB()

Returns the b parameter

See Also:

BM25Similarity(float, float)