I doubt color would have any adverse effect as structure. In fact, I would think the light reflection of lighter colors would actually speed algae growth and quicken it's attraction to bait fish over darker colors.

As ewest said, roughening any smooth structure to provide a better substrate for plant growth is what is most beneficial, on any color or composition.