Colloquium / Seminars
Topic：Un-rectifying Non-linear networks
Speaker：Prof. Hwang, Wen-Liang
(The Institute of Information Science, Academia Sinica)
Date time：June 16, 2020 14:00 - 15:00
Tea Party：June 16 13:30 SA205
Abstract：Abstract. I invent the novel "un-rectifying" technique for analysis of deep neural network (DNN).
Deep feedforward neural networks with piecewise linear activations can be viewed as piecewise affine functions, affine linear on polytopes partitioning the input space. We thus consider networks with rectified linear units and max-pooling operations from a signal representation perspective. In this view, such representations mark the significant transition from using a single linear representation to utilizing a large collection of affine linear representations tailored to particular regions of the signal space. However, the expression power of a network cannot be fully leveraged in signal processing without explicit expressions of the affine linear operators, their domains, ranges, and composition from the weight and bias parameters of the network. This article addresses the problem and provides a precise description of the individual affine representations and corresponding domain regions that the neural network associates to each signal of the input space. In particular, we describe weighted atomic decompositions of the representations and, based on estimating their Lipschitz regularity, draw a connection between sparse or compressible weight distributions and the stability in representation and learning, independent of the network depth. Such analysis may facilitate understanding networks and promote further theoretical insight from both the signal processing and machine learning communities.