Real-time prediction of clinical interventions remains a challenge within intensive care units (ICUs). This task is complicated by data sources that are sparse, noisy, heterogeneous and outcomes that are imbalanced. In this work, we integrate data across many ICU sources — vitals, labs, notes, demographics — and focus on learning rich representations of this data to predict onset and weaning of multiple invasive interventions. In particular, we compare both long short-term memory networks (LSTM) and convolutional neural networks (CNN) for prediction of five intervention tasks: invasive ventilation, non-invasive ventilation, vasopressors, colloid boluses, and crystalloid boluses. Our predictions are done in a forward-facing manner after a six hour gap time to support clinically actionable planning. We achieve state-of-the-art results on these predictive tasks using deep architectures. Further, we explore the use of feature occlusion to interpret LSTM models, and compare this to the interpretability gained from examining inputs that maximally activate CNN outputs. We show that our models are able to significantly outperform baselines for intervention prediction, and provide insight into model learning.