Causal Discovery via Common Entropy

Abstract

Distinguishing causation from correlation from observational data requires assumptions. We consider the setting where the unobserved confounder between two observed variables is simple in an information-theoretic sense, captured by its entropy. When the observed dependence is not due to causation, there exists a small-entropy variable that can make the observed variables conditionally independent. The smallest such entropy is known as common entropy in information theory. We extend this notion to Renyi common entropy by minimizing the Renyi entropy of the latent variable. We establish identifiability results with Renyi-0 common entropy, and a special case of (binary) Renyi-1 common entropy. To efficiently compute common entropy, we propose an iterative algorithm that can be used to discover the trade-off between the entropy of the latent variable and the conditional mutual information of the observed variables. We show that our algorithm can be used to distinguish causation from correlation in such simple two-variable systems. Additionally, we show that common entropy can be used to improve constraint-based methods such as the PC algorithm in the small-sample regime, where such methods are known to struggle. We propose modifying these constraint-based methods to assess if a separating set found by these algorithms is valid using common entropy. We finally evaluate our algorithms on synthetic and real data to establish their performance.