In this presentation we will report on a new information theoretic perspective for understanding the Exact Renormalization Group (ERG). In particular, by utilizing the picture of an ERG flow as a functional diffusion process, we shall outline how renormalization can be understood as an inverse process dual to a dynamical Bayesian inference scheme. A salient feature of this correspondence is that it identifies the Fisher information metric as an emergent renormalization scale related to the precision with which nearby points in model/theory space can be differentiated. This introduces new possibilities for the implementation of renormalization to systems with spatially non-local interactions, or even systems without any notion of spatial locality at all.