Source of the sorrow

There's a red brick building, 
Roof layered in slate,
With moss on on the flashing
Like mold on a cake,
It has an old white barn door,
That blocks out the rain,
Inside there's an old man,
I can't tell you his name,
He's the source of much sorrow
And he wont go away.