Both sides, and the conversation that comes with it
A privacy fence only does its job if both faces are protected, and the side you never look at is often the side taking afternoon sun. Where a fence sits on a boundary this involves the neighboring property, which is worth raising before work is scheduled rather than on the morning. Most neighbors are glad to have it done. It is still better asked than assumed.
