FIFO: Learning Fog-invariant Features for Foggy Scene Segmentation

previous work

combat lack of “foggy” training data with synthetic “fog”
curriculum adaptation
- increase density of “fog” as training progresses
- finally adapt to real light fog + dense synthetic fog data
previous work showed performance degradation on clear images! (-11.3%)
were also very dependent on hyperparameter tuning and external fog density estimator

proposed method

learn features invariant to fog conditions
- no hyper-parameters that control fog density in training
- no extra modules needed to estimate fog density in real setting
clear weather, synthetic fog, real fog all given as input to segmentation network
create embedding “fog factor”
- constrain so that “fog factor”s are unaffected by fog conditions
fog affects image style → use Gram matrix
- account for other background and object effects with a “fog-pass filtering module”
update alternately between segmentation network & fog-pass filter
- at test time, only run forward pass on segmentation network
loss
- fog-pass filter
  - attracting L2 distance in embedding space for similar fog conditions, repelling L2 distance for different fog conditions
- segmentation
  - segmentation loss + fog style matching loss + prediction loss
effect of fog-pass filter
- using only Gram matrix fails to differentiate between clear and synthetic fog conditions
image reconstructing using FIFO also works!

ablations and analysis

outperforms previous methods in both foggy and clear conditions
all components shown to have meaningful impact on results
results on unseen weather conditions like rain prove “fog factor”s can help with those conditions as well (to an extent)