MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection

MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection

human-object interaction detection

  • understand interactions between humans and other objects detected in the image

motivation

  • quadratic complexity and sequential nature of previous detector approaches

UnionDet

  • problems
    • IoU is insufficient for interactions
    • action is biased towards humans
    • union regions overlap with one another
  • compared to previous works, speed was greatly improved

HOTR

  • inference still dependent on pairs
    • previous works still did detection into NMS
  • take inspiration from DETR
  • image → CNN: get image features
  • feed features into Transformer Encoder
  • Run Encoder output through Instance Decoder & Interaction Decoder
    • Instance Decoder: object detection
    • Interaction Decoder: interaction detection
  • use HO pointers to merge the output from both decoders to calculate similarity scores

MSTR

  • moving into multiscale features causes significant overhead!
  • self attention within encoder architecture
  • instead of pairing human & object to reference point, propose “dual-entity attention”
  • entity-conditioned context → removing the surrounding context prevents us from utilizing the transformer’s ability to attend to global features as well