Replay server post-upgrade cleanup tasks

There are several post-upgrade cleanup tasks to do on the Replay server.