Rumelhart et al. present a trivial example with no hidden layers that acts as a shift register;
The net is trained to shift binary patterns one bit to the left, but the training set consists of patterns that have been shifted two bits (that is, the `expanded' net has 4 layers, not 2), thereby demonstrating that such nets do not need to know all the intermediate representations to learn the task.