Nonlinear computation in deep linear networks

OpenAI published a research finding that deep linear networks—neural networks with only linear activation functions—can perform nonlinear computation under specific conditions. Traditionally, linear networks were considered unable to model nonlinear functions without explicit nonlinearities. The study demonstrates that when the network is sufficiently deep and its parameters are appropriately constrained, the composite function can exhibit nonlinear behavior. This challenges long-held assumptions in deep learning theory, suggesting that depth alone may induce nonlinearity in certain regimes. For builders of AI workflows, this could open avenues for designing simpler or more interpretable models that still capture complex patterns, potentially reducing computational overhead. However, practical applications remain theoretical, as the conditions for nonlinearity may be difficult to achieve in standard training.