We're willing for players to deal with that barrier as they see fit! You're free to have the memories return of their own accord after a few days, piecemeal or all at once, or you can require that they need to be jogged back into place in some fashion. Alternatively, you can have them simply be...gone forever, or require a request to the Storyteller to be restored.