Embedding learning ability in robotic systems is one of the long sought-after objectives of artificial intelligence research. Despite the recent advancements in hardware, large-scale machine learning algorithms and theoretical understanding of deep learning, it is still quite unrealistic to deploy an end-to-end learning agent in the wild, attempting to learn everything from scratch. Instead, we identify the importance of imposing strong prior knowledge on capable robotic systems. We verify our theories through analyses of data efficiency and robotic systems that combine learning and planning. The new approaches integrate structured prior knowledge with statistical machine learning methods to achieve state-of-the-art performance on complex long-horizon robot manipulation tasks.