A long-standing problem in QFT and quantum gravity is the construction of an “IR-finite” S-matrix. Infrared divergences in scattering theory are intimately tied to the “memory effect” and the existence of an infinite number of “large gauge charges”. A suitable “IR finite” S-matrix requires the inclusion of states with memory (which do not lie in the standard Fock space). For QED such a construction was achieved by Faddeev and Kulish by appropriately “dressing” charged particles with memory. However, we show that this construction fails in the case of massless QED, Yang-Mills theories, linearized quantum gravity with massless/massive sources, and in full quantum gravity. In the case of quantum gravity, we prove that the only "Faddeev-Kulish" state is the vacuum state. We also show that non-Faddeev Kulish representations are also unsatisfactory. Thus, in general, it appears there is no preferred Hilbert space for scattering in QFT and quantum gravity. Nevertheless we show how scattering can be formulated in a manner that manifestly IR-finite without any “ad-hoc” restrictions or dressing on the states. Finally, we investigate the consequences of the superselection due to the “large gauge charges” and illustrate that, in QED, nearly all scattering states are completely decohered in the bulk.