Today, machine learning is developing ever more complex artificial neural networks that are becoming increasingly proficient in mimicking the perceptual inference abilities of humans and animals. This progress sparks many exciting opportunities for Computational Neuroscience. The most basic application is to use deep learning as a tool for fitting data. More generally, however, functionally impressive deep neural networks can be understood as novel model systems that join and extend the range of biological model systems (eg fly, rodent, or monkey) studied today. These artificial model systems are particularly useful to study the relation between structure and function, because the full connectome and responses of all neurons are readily available, and the absence of experimental limitations triggers new questions on what it takes to understand neural networks. Deep neural networks can also be used as ground truth models to better assess what conclusions can be drawn from neurophysiological experiments by simulating the experiments under the same limitations we face for biological model systems.