If you wanted privacy then why did you remove the conifer in the first place? It will take 10 years+ for you to get an real privacy back whatever you plant.
Looks like the neighbour on the other side of fence chopped their evergreens down too.
Bamboo will give you privacy and can provide good height without too much width. Make sure you look for clumping varieties though.
I'm looking at adding an Acer to my garden - sango kaku. In a small garden you need plants which will work hard all year round to maintain interest.