Parsing Engine

danbikel.parser
Class InterpolatedKnesserNeyModel

java.lang.Object
  extended bydanbikel.parser.Model
      extended bydanbikel.parser.InterpolatedKnesserNeyModel
All Implemented Interfaces:
Serializable

public class InterpolatedKnesserNeyModel
extends Model

Implements a model that uses interpolated Knesser-Ney smoothing.

See Also:
Serialized Form

Field Summary
protected  double optimalDiscountEstimate
           
 
Fields inherited from class danbikel.parser.Model
backOffMap, cache, cacheAccesses, cacheHits, canonicalEvents, counts, createHistBackOffMap, deleteCountsWhenPrecomputingProbs, dontAddNewParams, doPruning, globalDoPruning, histBackOffMap, historiesToPrune, lambdaFudge, lambdaFudgeTerm, lambdaPenalty, logOneMinusLambdaPenalty, numLevels, precomputedLambdas, precomputedNPBProbCalls, precomputedNPBProbHits, precomputedProbCalls, precomputedProbHits, precomputedProbs, precomputeProbs, printPrunedEvents, printUnprunedEvents, pruningThreshold, saveBackOffMap, saveHistBackOffMap, saveSmoothingParams, shortStructureClassName, smoothingParams, smoothingParamsFile, structure, structureClassName, topLevelCache, transitionsToPrune, useCache, useSmoothingParams, verbose, warnSmoothingHasHistoryNotInTraining
 
Constructor Summary
InterpolatedKnesserNeyModel(ProbabilityStructure structure)
          Constructs a Model instance that uses interpolated Knesser-Ney smoothing instead of the default smoothing method when estimating probabilities.
 
Method Summary
 void deriveCounts(CountsTable trainerCounts, Filter filter, double threshold, FlexibleMap canonical, boolean deriveOtherModelCounts)
          Derives all counts from the specified counts table, using the probability structure specified in the constructor.
protected  double estimateProb(ProbabilityStructure probStructure, TrainerEvent event)
          Returns the smoothed probability estimate of a transition contained in the specified TrainerEvent object.
protected  void precomputeProbs(MapToPrimitive.Entry transEntry, double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel)
          Precomputes the probabilities and smoothing values for the Transition object contained as a key within the specified map entry, where the value is the count of the transition.
protected  void precomputeProbs(TrainerEvent event, Transition[] transitions, Event[] histories)
          Deprecated. This method is called by Model.precomputeProbs(CountsTable,Filter), which is also deprecated.
protected  void storePrecomputedProbs(double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel)
          Stores the specified smoothing values (lambdas) and smoothed probability estimates in the Model.precomputedProbs and Model.smoothingParams map arrays.
 
Methods inherited from class danbikel.parser.Model
beQuiet, beVerbose, canonicalize, canonicalize, canonicalizeEvent, cleanup, computeHistoriesAndTransitionsToPrune, deriveCounts, deriveDiversityCounts, deriveHistories, estimateLogProb, estimateLogProbUsingPrecomputed, estimateLogProbUsingPrecomputed, estimateProb, estimateProbOld, getCacheStats, getCanonical, getModel, getProbStructure, getTransitions, initializeSmoothingParams, numModels, precomputeProbs, precomputeProbs, pruneHistoriesAndTransitions, pruneHistoriesAndTransitionsOld, readSmoothingParams, readSmoothingParams, savePrecomputeData, setCanonicalEvents, share, writeSmoothingParams, writeSmoothingParams
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

optimalDiscountEstimate

protected double optimalDiscountEstimate
Constructor Detail

InterpolatedKnesserNeyModel

public InterpolatedKnesserNeyModel(ProbabilityStructure structure)
Constructs a Model instance that uses interpolated Knesser-Ney smoothing instead of the default smoothing method when estimating probabilities.

Parameters:
structure - the probability structure for which this model will estimate probabilities
Method Detail

deriveCounts

public void deriveCounts(CountsTable trainerCounts,
                         Filter filter,
                         double threshold,
                         FlexibleMap canonical,
                         boolean deriveOtherModelCounts)
Description copied from class: Model
Derives all counts from the specified counts table, using the probability structure specified in the constructor.

Overrides:
deriveCounts in class Model
Parameters:
trainerCounts - a map from TrainerEvent objects to their counts (as doubles) from which to derive counts
filter - used to filter out TrainerEvent objects whose derived counts should not be derived for this model
threshold - a (currently unused) count cut-off threshold
canonical - a reflexive map used to canonicalize objects created when deriving counts
deriveOtherModelCounts - an unused parameter, as this class does not contain other, internal Model instances

estimateProb

protected double estimateProb(ProbabilityStructure probStructure,
                              TrainerEvent event)
Returns the smoothed probability estimate of a transition contained in the specified TrainerEvent object. The smoothing method employed will be the interpolated version of Knesser-Ney.

Overrides:
estimateProb in class Model
Parameters:
probStructure - a ProbabilityStructure object that is either Model.structure or a copy of it, used for temporary storage during the computation performed by this method
event - the TrainerEvent containing a transition from a history to a future whose smoothed probability is to be computed
Returns:
the smoothed probability estimate of a transition contained in the specified TrainerEvent object

precomputeProbs

protected void precomputeProbs(MapToPrimitive.Entry transEntry,
                               double[] lambdas,
                               double[] estimates,
                               Transition[] transitions,
                               Event[] histories,
                               int lastLevel)
Description copied from class: Model
Precomputes the probabilities and smoothing values for the Transition object contained as a key within the specified map entry, where the value is the count of the transition.

Overrides:
precomputeProbs in class Model
Parameters:
transEntry - a map entry mapping a Transition object to its count (a double)
lambdas - an array in which to store the smoothing value for each of the back-off levels
estimates - an array in which to store the maximum-likelihood estimate at each of the back-off levels
transitions - an array in which to store the Transition instance for each of the back-off levels
histories - an array in which to store the history, an Event instance, for each of the back-off levels
lastLevel - the last back-off level (the value equal to Model.numLevels - 1)
See Also:
Model.precomputeProbs()

storePrecomputedProbs

protected void storePrecomputedProbs(double[] lambdas,
                                     double[] estimates,
                                     Transition[] transitions,
                                     Event[] histories,
                                     int lastLevel)
Description copied from class: Model
Stores the specified smoothing values (lambdas) and smoothed probability estimates in the Model.precomputedProbs and Model.smoothingParams map arrays.

Overrides:
storePrecomputedProbs in class Model
Parameters:
lambdas - an array containing the smoothing value for each of the back-off levels
estimates - an array containing the maximum-likelihood estimate at each of the back-off levels
transitions - an array containing the Transition instance for each of the back-off levels
histories - an array in which to store the history, an Event instance, for each of the back-off levels
lastLevel - the last back-off level (the value equal to Model.numLevels - 1)
See Also:
Model.precomputeProbs()

precomputeProbs

protected void precomputeProbs(TrainerEvent event,
                               Transition[] transitions,
                               Event[] histories)
Deprecated. This method is called by Model.precomputeProbs(CountsTable,Filter), which is also deprecated.

Precomputes probabilities for the specified event, using the specified arrays as temporary storage during this method invocation.

Overrides:
precomputeProbs in class Model
Parameters:
event - the TrainerEvent object from which probabilities are to be precomputed
transitions - temporary storage to be used during an invocation of this method
histories - temporary storage to be used during an invocation of this method

Parsing Engine

Author: Dan Bikel.