This is the accompanying repository for our NeurIPS 2020 paper titled "On 1/n neural representation and robustness".
A recent finding by Stringer et al. 2019 found that the eigenspectrum of the empirical covariance matrix of the neural activity in mouse V1 followed a power-law, i.e.
regardless of the input statistics, where for natural images a universal exponent of
was observed. A corresponding theory was put forward as a potential rationale for the existence of a
eigenspectrum. While the theory was illuminating, the advantages of a representation with eigenspectrum decaying slightly faster than
is not apparent. To investigate further, we turn to deep neural networks as a testbed.
In general, the distribution of eigenvalues of a deep neural network is intractable and a priori there is no reason to believe it should follow a power law. To enforce a eigenspectrum, we directly regularize the eigenvalues of the activations at layer l, denoted by
, towards a desired eigenspectrum, denoted by
, which follows a
power-law:
where \tau is a cut-off that dictates which eigenvalues should be regularized and \beta is a hyperparameter that controls the strength of the regularizer.
In neurips_experiments/, experiment_1.py, experiment_2.py and experiment_3.py correspond to the experiments ran in sections 4.1, 4.2 and 4.3 respectively.
The vanilla networks can be trained by calling neurips_experiments/experiment_1.py --vanilla=True.
The spectrally regularized networks can be trained by calling neurips_experiments/experiment_1.py --vanilla=False.
Note that the networks can be trained on the GPU by passing in --cuda=True.
The vanilla MLPs can be trained by calling neurips_experiments/experiment_2.py --vanilla=True --arch="mlp" .
The MLPs with whitened intermediate representations (denoted as Vanilla-Wh in section 4.2) can be trained by calling neurips_experiments/experiment_2.py --vanilla=True --arch="mlp" --flat=True.
The MLPs where only the last hidden layer is spectrally regularized (denoted as SpecReg in section 4.2) can be trained by calling neurips_experiments/experiment_2.py --vanilla=False --arch="mlp" .
The MLPs where only the last hidden layer is spectrally regularized and the intermediate representation is whitend (denoted as SpecReg-Wh in section 4.2) can be trained by calling neurips_experiments/experiment_2.py --vanilla=False --arch="mlp" --flat=True .
(Note that to repeat these same experiments on CNNs pass in --arch="cnn" instead. Also, networks can be trained on the GPU by passing in --cuda=True.)
The vanilla MLPs can be trained by calling neurips_experiments/experiment_3.py --vanilla=True --arch="mlp" .
The MLPs where every hidden layer is spectrally regularized (denoted as SpecReg in section 4.3) can be trained by calling neurips_experiments/experiment_3.py --vanilla=False --arch="mlp" .
The MLPs whose input-output Jacobian is regularized (denoted as Jac in section 4.3) can be trained by calling neurips_experiments/experiment_3.py --vanilla=True --arch="mlp" --jac=True.
(Note that to repeat these same experiments on CNNs pass in --arch="cnn" instead. Also, networks can be trained on the GPU by passing in --cuda=True.)