receptive fields are how CNNs see objects

DynOPool applicable to most operations that control size of receptive field

related works

• NAS: generic, not designed for feature map size optimization
• dynamic kernel shape: changing filter size
• Shape Adaptor: directly learn feature map size
  • only handles symmetric feature maps and selects a predefined resizing ration range
• DiffStride: spectral pooling to determine optimal scale factor of pooling layer
  • implement downsampling with cropping in the frequency domain

DynOPool

• differentiable pooling layer
  • scaling factor for horizontal and vertical directions to be optimized
  • find four query points in each grid
  • derive output feature via max pooling of these query points
• optimization requires additional work because of rounding function
  • output size width and height must be integers
  • optimization by relaxation
  • regularization for computational complexity in addition to task loss
• shown to also be compatible with NAS, semantic segmentation, etc.

few-shot font generation

• generate a new font library (> 20,000 characters) with few references
• without additional training procedures

goal

• capture “font style” from very few glyph images
• font styles are both global, but very local
• character content is also very complex and sensitive to local change!

previous works

• reference images → style, source image → features, merge to generate new image’
  • for font, single representation is insufficient to represent “style”

proposed method

• compositionality
  • decompose characters into a number of components
  • 24 of top 30 used languages has compositionality
• define “multiple localized” style representation using compositionality
  • capture multiple local features from each image
  • requires guidance
• assign most likely components to each expert
  • predict probability by passing multiple local features to component classifier
  • match components to express to maximize summation over prediction probability
• style features are trained to
  • correctly classify style labels predicted by style classifier
  • fooling component labels predicted by component classifier
• maximize independence using HSIC as loss

previous work

• combat lack of “foggy” training data with synthetic “fog”
• curriculum adaptation
  • increase density of “fog” as training progresses
  • finally adapt to real light fog + dense synthetic fog data
• previous work showed performance degradation on clear images! (-11.3%)
• were also very dependent on hyperparameter tuning and external fog density estimator

proposed method

• learn features invariant to fog conditions
  • no hyper-parameters that control fog density in training
  • no extra modules needed to estimate fog density in real setting
• clear weather, synthetic fog, real fog all given as input to segmentation network
• create embedding “fog factor”
  • constrain so that “fog factor”s are unaffected by fog conditions
• fog affects image style → use Gram matrix
  • account for other background and object effects with a “fog-pass filtering module”
• update alternately between segmentation network & fog-pass filter
  • at test time, only run forward pass on segmentation network
• loss
  • fog-pass filter
    • attracting L2 distance in embedding space for similar fog conditions, repelling L2 distance for different fog conditions
  • segmentation
    • segmentation loss + fog style matching loss + prediction loss
• effect of fog-pass filter
  • using only Gram matrix fails to differentiate between clear and synthetic fog conditions
• image reconstructing using FIFO also works!

ablations and analysis

• outperforms previous methods in both foggy and clear conditions
• all components shown to have meaningful impact on results
• results on unseen weather conditions like rain prove “fog factor”s can help with those conditions as well (to an extent)

MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection

human-object interaction detection

• understand interactions between humans and other objects detected in the image

motivation

• quadratic complexity and sequential nature of previous detector approaches

UnionDet

• problems
  • IoU is insufficient for interactions
  • action is biased towards humans
  • union regions overlap with one another
• compared to previous works, speed was greatly improved

HOTR

• inference still dependent on pairs
  • previous works still did detection into NMS
• take inspiration from DETR
• image → CNN: get image features
• feed features into Transformer Encoder
• Run Encoder output through Instance Decoder & Interaction Decoder
  • Instance Decoder: object detection
  • Interaction Decoder: interaction detection
• use HO pointers to merge the output from both decoders to calculate similarity scores

MSTR

• moving into multiscale features causes significant overhead!
• self attention within encoder architecture
• instead of pairing human & object to reference point, propose “dual-entity attention”
• entity-conditioned context → removing the surrounding context prevents us from utilizing the transformer’s ability to attend to global features as well

challenges in CV

• training
  • fairness
  • privacy
  • cost
• deployment
  • explainability
  • robustness
  • social impact

exponential growth in model size & dataset size!

• growing need for large, curated (labelled) datasets → bottleneck

do we need labels?

• something people have been trying to address through self-supervised learning

current trend is heavily dependent on resources!

• more GPUs = better computation = better everything!
• should be the opposite (sensing » computation)

with the correct sensing, we can reduce the required amount of computations to feasible levels

“Greatest Hits” dataset

• 45,000 short videos of drumstick hitting objects
• prediction of audio from silent videos!
• showed promising results!

tactile sensors are snubbed, even though they are the most important for humans

learning to see through tactile information

• using tactile sensor input maps, prediction of walking gait has been attempted!

learning to see by drawing images

• basically what GANs and other generative models do
• but what do these well-performing generative models learn?

network dissection (GAN)

• pass random vectors through network, generate lots of images
• check activation maps of the layers + run a pre-trained segmentation model on the generated images
  • check agreement between activation and segmentation maps
• turns out units are assigning themselves labels (affinity for certain classes when drawing new images)

disabling certain neurons

• when researchers deactivated all “window” neurons, or activated new “door” neurons, the networks understood what to do
  • didn’t draw doors in the sky
  • replaced the window with a wall and painting (something that makes sense semantically)

DatasetGAN

• take images generated with GAN
• manually annotate a few images
• sample from DatasetGAN to create infinite, annotated data!

BigDatasetGAN

• same thing, but with ImageNet

unsupervised contrastive learning

• perturb images and claim those images should belong to same class
• motivation: using simulators to create synthetic data is hard!
  • lots of detail
  • lots of background knowledge required
• test ImageNet100 performance using AlexNet-based encoder
  • Minecraft, CLeVr data are examples of synthetic data used in training
  • Minecraft works ok, but it is a pretty hardcore simulator!

Learning to See by Looking at Noise

• capture features of natural images via a stochastic process that generates images
• “Dead leaves model”: introduced in the 60’s
  • used as a representation of natural images
• use images generated via the “Dead leaves model” and use those to train an encoder and inference on ImageNet100!
  • similar results to Minecraft-trained encoder
  • much less overhead for simulator preparation
• sample from an untrained GAN → provides a representation of natural images!
• Shaders-21K dataset: short generative programs in OpenGL!
  • results that are almost as good as models trained on real data!

problem

• low contrast image!

previous works

• normally perform channel-wise intensity transform or some encoder-decoder architecture

proposed method

• representative color transform
  • run image through encoder
  • embedding vector $R$ is in $\mathbb{R}^{N \times C}$ where $C$ is the number of “representative color”s
  • compare with $\mathbb{R}^{N\times3}$ vector $T$ showing R, G, B values post-transform
  • get features $F$ from image via CNN
  • $A = softmax(FR^T/\tau)$, enhanced image $Y = AT^T$
• local RCT (the above would be the “global” version)
  • use fine-scale features covering a subset of the original image

Faster-RCNN object detection

• Region Proposal Network (RPN)
• region-based detection network

SiamMOT

• Faster-RCNN + region-based Siamese tracker
• Siamese tracker can model “instance-level motion”
  • comparing frames at $t’=t$ and $t’=t+\delta$

Overview of SiamMOT architecture

Motion modelling with Siamese tracker

• $i$ is detected at $t’=t$ in frame $\mathbf{I}^t$
• $R_i^t$ is “contextual window” around position of $i$ in frame $\mathbf{I}^t$
• $\tau$ is Siamese tracker

• adjacent frame is $I^{t+\delta}$

• define search region $S_i^{t+\delta}$ by expanding $R_i^t$ by fixed factor $r (> 1)$
• define $v_i^{t+\delta}$ as confidence of visibility for $i$ in frame $I^{t+\delta}$

• $f_{R_i}^t$ and $f_{S_i}^{t+\delta}$ are outputs from ROIAlign layer of Mask-RCNN (feature extraction)

• $(v_i^{t+\delta}, \tilde{R_i}^{t+\delta}) = \tau(f_{R_i}^t, f_{S_i}^{t+\delta}; \Theta)$

Two possible ways to model motion!

Implicit Motion Model (IMM)

• concatenate $f_{R_i}^t$ and $f_{S_i}^{t+\delta}$ and feed into MLP
• predict
  • confidence of visibility $v_i$

  • relative location & scale change

    $\Delta x / w^t$, $\Delta y/h^t$, $log(\frac{w^{t+\delta}}{w^t})$, $log(\frac{h^{t+\delta}}{h^t})$

    

    Loss used for training IMM (* indicates GT values)

Explicit Motion Model (EMM)

• channel-wise cross-correlation operator
• CNN for generating visibility confidence map $v_i$
  • shows likelihood pixel contains the detected person
• CNN for generating location map $p_i$
  • outputs offset from each location to bbox corners

Explicit Motion Model overview

Formula for penalty map

Formula for decoding output from EMM

Loss used for training EMM

existing work

• semantic segmentation
  • performance issues
• anchor-based detection considering straight lines as anchors
  • hard to deal with complicated lanes!

proposed method

• eigenlane space: lane matrix $A$ from image
• apply SVD to $A$ ($A = U\Sigma V^T$)
• ignore most of the singular values near or equal to 0!
  • we can compress the information into smaller matrices (less computation)
  • the remaining columns of $U$ are “eigenlane”s, representing the lane itself
• generating lane candidates
  • clustering in eigenlane space (relatively low-dimensional)
• involves work that checks the harmony between proposed lines
  • based on prior works from the same authors regarding semantic line detection

person search

• person detection + re-ID
• detection + re-ID sequential two-step approach VS end-to-end approach

problems in end-to-end

• task conflict
  • person detection needs to embed some common features that apply to all humans
  • person re-ID requires the embedding of uniqueness for each person to discern identity
• task dependency
  • poor quality of detection will directly affect re-ID performance

proposed method

• adaptive gradient backprop
  • cross-entropy loss w/ softmax prob
  • adaptive gradient weighting function
    • similarity score and detection confidence score are used as weighting factors!
• additional task (part classification)
  • predicting classes of parts of the human body
  • prevent overfitting to either re-ID or detection task!

motivation

• face age transformation should
  • maintain the identity of the input image
  • have wide-range target age characteristics
  • generate natural-looking faces

proposed method

• output from encoder is given as input along with the target age to a simple CNN
• the output from this “modulator” is given to the decoder along with the encoder output to generate a new image that has a more consistent identity compared to prior works
• each layer of the decoder if fed output from the age modulator! (age-aware features)
• use a cycle-consistency loss where we age from $t$ to $t’$ and from $t’$ to $t$ again
  • the end result should be similar to the original image
• also use a reconstruction loss where aging from $t$ to $t$ should make almost no difference

extension (video re-aging)

• requires some form of temporal consistency
  • previous, frame-based approaches lack consistency
• encoder and modulator networks involve some form of smoothing
  • modulator also includes some new multi-scale aspects
  • this provides some (unexplained) temporal consistency