when you say starting to grow do you mean they look like a fruit or are they quite tiny? I would suggest removing anything bigger than pea size. This will leave the smaller embryonic fruit to ripen during next summer. Figs usually have two crops coming at once when they are established. The current crop which you will pick when ripe and the next crop to come. Our weather doesnt allow them to ripen f(the second crop) fully outside in the way that they do in their native environment so you will always end up with fruit that wont fully ripen later in the season (hence removing it).
Alan, did I not answer this question for you recently?
MRweeds has just said the same thing that I said.
Wait!! Golf ball size. We are in August, these may ripen. Give them time.
I apologise, Alan. I often give the same advice about removing small figs late in the Autumn I get in a muddle about how many people I have said it to!!