Welcome to PB.

Put down a high-quality geotextile underlayment before you put the stone down.

The fabric will prevent deep-rooted vegetation from coming up through the layer of stone, and it will keep the stone from disappearing into the mud.
Some plant growth will certainly occur in the layer of stone, but it will be shallow-rooted and easy to control by chemical or mechanical means.