Beyond Fixed Grid:
Learning Geometric Image Representation with a Deformable Grid

Jun Gao
Zian Wang
Jinchen Xuan
Sanja Fidler

ECCV, 2020

We deform grid's vertices, such that grid's edges align with image boundaries, while keeping topology fixed.

In modern computer vision, images are typically represented as a fixed uniform grid with some stride and processed via a deep convolutional neural network. We argue that deforming the grid to better align with the high-frequency image content is a more effective strategy. We introduce Deformable Grid (DefGrid), a learnable neural network module that predicts location offsets of vertices of a 2-dimensional triangular grid, such that the edges of the deformed grid align with image boundaries. We showcase our DefGrid in a variety of use cases, i.e., by inserting it as a module at various levels of processing. We utilize DefGrid as an end-to-end learnable geometric downsampling layer that replaces standard pooling methods for reducing feature resolution when feeding images into a deep CNN. We show significantly improved results at the same grid resolution compared to using CNNs on uniform grids for the task of semantic segmentation. We also utilize DefGrid at the output layers for the task of object mask annotation, and show that reasoning about object boundaries on our predicted polygonal grid leads to more accurate results over existing pixel-wise and curve-based approaches. We finally showcase DefGrid as a standalone module for unsupervised image partitioning, showing superior performance over existing approaches


Jun Gao, Zian Wang, Jinchen Xuan, Sanja Fidler

Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid

[Code Released!]

Please address correspondence regarding this paper to Jun Gao and Sanja Fidler.



Semantic Segmentation on DefGrid v.s. Fixed Grid

Boundary Labeling using DefGrid

Superpixels using DefGrid

We thank David Acuna for his website template.