# opentl::modalities::HarrisKeypoints Class Reference

Inherits opentl::modalities::Modality.

List of all members.

## Classes
class | HarrisKeyframe
| A key-frame contains: an image, and its filtered version (for NCC matching) the annotated pose of the visible target (NOTE: only one target per keyframe!) the detected (and back-projected) Harris keypoints. More...
## Public Types
## Public Member Functions
void | addKeyFrame (const boost::shared_ptr< opentl::models::Target > &target, opentl::core::cvdata::Image *img=NULL)
| Add key-frame.
virtual HarrisKeypoints * | clone () const
| Clone: call the copy constructor.
int | countOfflineMatchings (int targetId)
void | drawCurrKeypoints (opentl::core::cvdata::Image *img)
| Draw current detected features (red crosses).
void | drawKeyframe (opentl::core::cvdata::Image *img, int keyInd)
| Superimpose keyframe on the image.
void | drawNearestKeyframe (opentl::core::cvdata::Image *img, int targetId)
| Superimpose nearest keyframe to a given target on the image.
void | drawOfflineKeypoints (opentl::core::cvdata::Image *img, int targetId=0)
| Draw off-line (previously matched) features (purple crosses).
void | drawOfflineMatchings (opentl::core::cvdata::Image *img, int targetId=0)
| Draw keyframe (= off-line) matching features (blue dots and lines).
void | drawOnlineKeypoints (opentl::core::cvdata::Image *img, int targetId=0)
| Draw on-line (previously matched) features (purple crosses).
void | drawOnlineMatchings (opentl::core::cvdata::Image *img, int targetId=0)
| Draw on-line matching features (purple dots and lines).
int | getNKeyFrames ()
| Number of available keyframes.
void | getOfflineMatchings (std::vector< math::Vector3 > &bodyPoints, std::vector< math::Vector2 > &screenPoints, int targetId=0)
| Obtain keyframe (= off-line) matchings.
void | getOnlineMatchings (std::vector< math::Vector3 > &bodyPoints, std::vector< math::Vector2 > &screenPoints, int targetId=0)
| Obtain on-line matchings.
| HarrisKeypoints (const HarrisKeypoints &c)
| Copy constructor.
| HarrisKeypoints (opentl::modelprojection::WarpBack *warpBack, int camIdx, modelprojection::GLRenderer *renderer=NULL)
| Constructor.
virtual void | init ()
| Allocate memory and set parameters.
virtual int | matchFeatLevel (const TargetPtrVector &targets, T_MEAS_FEATPtrVector &outputMeas, std::size_t partitionIdx)
| matchFeatLevel. Match detected keypoints on short-baseline (previous frame) and wide-baseline (nearest key-frame)
int | matchKeypointsNCC (opentl::core::cvdata::HarrisKeypoints *feat1, opentl::core::cvdata::HarrisKeypoints *feat2)
| Match two feature sets with normalized cross-correlation (NCC).
virtual int | matchObjLevel (const TargetPtrVector &targets, T_MEAS_OBJPtrVector &outputMeas, std::size_t partitionIdx)
| matchObjLevel. Estimate pose from point correspondences (it calls matchFeatLevel first)
virtual int | preProcess (const opentl::core::cvdata::Image &image, const std::vector< std::vector< int > > &preProcessROIs)
| Detect Harris keypoints in the current image.
void | removeAllKeyframes ()
| Remove all key-frames.
virtual int | sampleModelFeatures (const TargetPtrVector &targets)
| Sample visible model features M_off, from the off-line given ShapeAppearance model (i.e. off-line).
void | shortBaselineWarp (opentl::core::cvdata::HarrisKeypoints *onlineFeatures, const opentl::core::State &predState)
| Re-project keypoint positions from the previous frame to the current, predicted pose.
virtual int | updateModelFeatures (const TargetPtrVector &targets)
| Update model features M_on, from the on-line image stream.
void | wideBaselineWarp (HarrisKeyframe *keyFrame, opentl::core::cvdata::HarrisKeypoints *offlineFeatures, const opentl::core::State &predState)
| Warp keypoint descriptors from the nearest keyframe (pre-computed) to the current position in the screen. For each patch (point and normal) in object-space, compute the invert homography from the current (predicted) view, back to the keyframe and warp the keyframe descriptor.
virtual | ~HarrisKeypoints ()
| Destructor.

## Member Enumeration Documentation

enum opentl::modalities::HarrisKeypoints::OfflineParameters

Enumerator:

  maxFeatures | (int) Maximum number of detectable features on any image (keyframes or current frames)
  corrWin | (int) Size of the correlation window
  OFFLINE_COUNT |

Reimplemented from opentl::modalities::Modality.

enum opentl::modalities::HarrisKeypoints::OnlineParameters

Enumerator:

  computeJacobians | <bool> Flag for computing Jacobians of re-projected keypoints (for example, needed by DataFusion::upgradeToObjLevel) Default: false
  minDetectDist | (double) Minimum distance between detected features
  qualityLevel | (double) Minimum acceptable "quality" for each feature
  maxMatchDist | (double) Maximum search distance for matching (~ std of feature-level measurement noise, R)
  nccThresh | (double) Minimum NCC value to accept a match
  measStdFeatOffline | (double) Standard deviation of measurement noise (feature-level), for the off-line sampled keypoints
  measStdFeatOnline | (double) Standard deviation of measurement noise (feature-level), for the on-line updated keypoints
  measStdObj | (double) Standard deviation of measurement noise (object-level)
  poseElementWeights | (math::Transform) Weights (Mahalanobis distance) for the finite pose parameters, in order to evaluate "distances" in pose-space
  usePoseElementWeights | (bool) Whether to use pose element weights
  maxKeyframes | (int) Maximum number of possible key-frames
  renderKeyframes | (bool) If true the key-frames are rendered using the texture model. Otherwise, the current image is used. NOTE the latter may introduce errors, if the pose is not well aligned!
  createKeyframeDist | (double) If minimum distance between current frame and the nearest keyframe is higher than this, then a new keyframe is created default: -1 (no keyframe is created, apart from the first)
  removeKeyframeDist | (double) If distance between current frame and a keyframe is higher than this, then the keyframe is removed default: -1 (no keyframe is removed)
  ONLINE_COUNT |

Reimplemented from opentl::modalities::Modality.

## Constructor & Destructor Documentation

opentl::modalities::HarrisKeypoints::HarrisKeypoints | ( | opentl::modelprojection::WarpBack * | warpBack,
|  | int | camIdx,
|  | modelprojection::GLRenderer * | renderer = `NULL` |
| ) |  |  |

Constructor.

opentl::modalities::HarrisKeypoints::HarrisKeypoints | ( | const HarrisKeypoints & | c | ) |

Copy constructor.

virtual opentl::modalities::HarrisKeypoints::~HarrisKeypoints | ( |  | ) | `[virtual]`

Destructor.

## Member Function Documentation

void opentl::modalities::HarrisKeypoints::addKeyFrame | ( | const boost::shared_ptr< opentl::models::Target > & | target,
|  | opentl::core::cvdata::Image * | img = `NULL` |
| ) |  |  |

Add key-frame.

Parameters:

  | target | Input target
  | img | Input (optional) input image; if absent, the key-frame will be rendered from the target model

Returns:
  Number of matches found

virtual HarrisKeypoints* opentl::modalities::HarrisKeypoints::clone | ( |  | ) | const`[inline, virtual]`

Clone: call the copy constructor.

Implements opentl::modalities::Modality.

int opentl::modalities::HarrisKeypoints::countOfflineMatchings | ( | int | targetId | ) |

void opentl::modalities::HarrisKeypoints::drawCurrKeypoints | ( | opentl::core::cvdata::Image * | img | ) |

Draw current detected features (red crosses).

void opentl::modalities::HarrisKeypoints::drawKeyframe | ( | opentl::core::cvdata::Image * | img,
|  | int | keyInd |
| ) |  |  |

Superimpose keyframe on the image.

void opentl::modalities::HarrisKeypoints::drawNearestKeyframe | ( | opentl::core::cvdata::Image * | img,
|  | int | targetId |
| ) |  |  |

Superimpose nearest keyframe to a given target on the image.

void opentl::modalities::HarrisKeypoints::drawOfflineKeypoints | ( | opentl::core::cvdata::Image * | img,
|  | int | targetId = `0` |
| ) |  |  |

Draw off-line (previously matched) features (purple crosses).

void opentl::modalities::HarrisKeypoints::drawOfflineMatchings | ( | opentl::core::cvdata::Image * | img,
|  | int | targetId = `0` |
| ) |  |  |

Draw keyframe (= off-line) matching features (blue dots and lines).

void opentl::modalities::HarrisKeypoints::drawOnlineKeypoints | ( | opentl::core::cvdata::Image * | img,
|  | int | targetId = `0` |
| ) |  |  |

Draw on-line (previously matched) features (purple crosses).

void opentl::modalities::HarrisKeypoints::drawOnlineMatchings | ( | opentl::core::cvdata::Image * | img,
|  | int | targetId = `0` |
| ) |  |  |

Draw on-line matching features (purple dots and lines).

int opentl::modalities::HarrisKeypoints::getNKeyFrames | ( |  | ) | `[inline]`

Number of available keyframes.

void opentl::modalities::HarrisKeypoints::getOfflineMatchings | ( | std::vector< math::Vector3 > & | bodyPoints,
|  | std::vector< math::Vector2 > & | screenPoints,
|  | int | targetId = `0` |
| ) |  |  |

Obtain keyframe (= off-line) matchings.

void opentl::modalities::HarrisKeypoints::getOnlineMatchings | ( | std::vector< math::Vector3 > & | bodyPoints,
|  | std::vector< math::Vector2 > & | screenPoints,
|  | int | targetId = `0` |
| ) |  |  |

Obtain on-line matchings.

virtual void opentl::modalities::HarrisKeypoints::init | ( |  | ) | `[virtual]`

Allocate memory and set parameters.

Reimplemented from opentl::modalities::Modality.

virtual int opentl::modalities::HarrisKeypoints::matchFeatLevel | ( | const TargetPtrVector & | targets,
|  | T_MEAS_FEATPtrVector & | outputMeas,
|  | std::size_t | partitionIdx |
| ) |  |  | `[virtual]`

matchFeatLevel. Match detected keypoints on short-baseline (previous frame) and wide-baseline (nearest key-frame)

Parameters:

  | targets | Input vector of targets
  | outputMeas | Output feature-level measurements (matched keypoints, residuals and evtl. Jacobians) NOTE: The second index of outputMeas (state inside the target) is not used, and always set to 0.

Reimplemented from opentl::modalities::Modality.

int opentl::modalities::HarrisKeypoints::matchKeypointsNCC | ( | opentl::core::cvdata::HarrisKeypoints * | feat1,
|  | opentl::core::cvdata::HarrisKeypoints * | feat2 |
| ) |  |  |

Match two feature sets with normalized cross-correlation (NCC).

Parameters:

  | feat1 | Input feature set 1; found correspondences with feat2 are stored inside
  | feat2 | Input feature set 2; found correspondences with feat1 are stored inside

Returns:
  Number of matches found

virtual int opentl::modalities::HarrisKeypoints::matchObjLevel | ( | const TargetPtrVector & | targets,
|  | T_MEAS_OBJPtrVector & | outputMeas,
|  | std::size_t | partitionIdx |
| ) |  |  | `[virtual]`

matchObjLevel. Estimate pose from point correspondences (it calls matchFeatLevel first)

Parameters:

  | targets | Input vector of targets
  | outputMeas | Output object-level measurements (state-space residuals) NOTE: The second index of outputMeas (state inside the target) is not used, and always set to 0.

Reimplemented from opentl::modalities::Modality.

virtual int opentl::modalities::HarrisKeypoints::preProcess | ( | const opentl::core::cvdata::Image & | image,
|  | const std::vector< std::vector< int > > & | preProcessROIs |
| ) |  |  | `[virtual]`

Detect Harris keypoints in the current image.

Parameters:

  | image | Input sensor data (e.g. camera image in RGB)
  | preProcessROIs | Regions of interest (x0,y0,width,height), per target

Reimplemented from opentl::modalities::Modality.

void opentl::modalities::HarrisKeypoints::removeAllKeyframes | ( |  | ) |

Remove all key-frames.

virtual int opentl::modalities::HarrisKeypoints::sampleModelFeatures | ( | const TargetPtrVector & | targets | ) | `[virtual]`

Sample visible model features M_off, from the off-line given ShapeAppearance model (i.e. off-line).

Get the nearest key-frame (considering all targets.

Reimplemented from opentl::modalities::Modality.

void opentl::modalities::HarrisKeypoints::shortBaselineWarp | ( | opentl::core::cvdata::HarrisKeypoints * | onlineFeatures,
|  | const opentl::core::State & | predState |
| ) |  |  |

Re-project keypoint positions from the previous frame to the current, predicted pose.

Parameters:

  | onlineFeatures | Previous frame (back-projected in space) keypoints
  | predState | Predicted state for this target

virtual int opentl::modalities::HarrisKeypoints::updateModelFeatures | ( | const TargetPtrVector & | targets | ) | `[virtual]`

Update model features M_on, from the on-line image stream.

1 - Update on-line features with the detected ones (from the pre-processing step) 2 - If no nearby key-frame is present, then create a new one under the estimated pose, and back-project the detected features

Parameters:

  | targets | Vector of targets

Reimplemented from opentl::modalities::Modality.

void opentl::modalities::HarrisKeypoints::wideBaselineWarp | ( | HarrisKeyframe * | keyFrame,
|  | opentl::core::cvdata::HarrisKeypoints * | offlineFeatures,
|  | const opentl::core::State & | predState |
| ) |  |  |

Warp keypoint descriptors from the nearest keyframe (pre-computed) to the current position in the screen. For each patch (point and normal) in object-space, compute the invert homography from the current (predicted) view, back to the keyframe and warp the keyframe descriptor.

Parameters:

  | keyFrame | Key-frame information (image and keypoints)
  | predState | Predicted state for this target