We study the dark-time decay of the retrieval efficiency for light stored in a Rydberg state in an ultracold gas of 87Rb atoms based on electromagnetically induced transparency (EIT). Using low atomic density to avoid dephasing caused by atom-atom interactions, we measure a 1/e time of 30µs for the 80S state in free expansion. One of the dominant limitations is the combination of photon recoil and thermal atomic motion at 0.2µK. If the 1064-nm dipole trap is left on, then the 1/e time is reduced to 13 µs, in agreement with a model taking differential light shifts and gravitational sag into account. To characterize how coherent the retrieved light is, we overlap it with reference light and measure the visibility V of the resulting interference pattern, obtaining V>90% for short dark time. Our experimental work is accompanied by a detailed model for the dark-time decay of the retrieval efficiency of light stored in atomic ensembles. The model is generally applicable for photon storage in Dicke states, such as in EIT with $\lamda$-type or ladder-type level schemes and in Duan-Lukin-Cirac-Zoller single-photon sources. The model includes a treatment of the dephasing caused by thermal atomic motion combined with net photon recoil, as well as the influence of trapping potentials. It takes into account that the signal light field is typically not a plane wave. The model maps the retrieval efficiency to single-atom properties and shows that the retrieval efficiency is related to the decay of fringe visibility in Ramsey spectroscopy and to the spatial first-order coherence function of the gas.