Eventually only a few lonely souls remain, remembering times past. But in time they too wander on or die and the town is a shell.
Its been happening for thousands of years, and continues to happen today. A town is much like a living organism. It lives. It thrives. It declines. It dies.