You dont need poison, just make sure the compost is damp.
They only infest dry compost.
Put a piece of landscape fabric in the bottom. Water can get through but nasties can't crawl in the drainage holes
It's pretty hard to get rid once they're in and they can kill the plants with their tunnelling.
Water them wel, if possible by sinking the pots in a bucket of water and waiting till the air bubbles stop then allow to drain. Water again with a solution of one small bottle of essential oil of cloves in 5 litres of water. They hate the smell and will move on. Keep your pots moist in future and use the oil of cloves form time to time.
It works on borders too but is best after wetting the soil as it soaks in better rather than running off the top.