From what I know about lilies, they are voracious consumers of nutrients, and need fertilizer tablets if placed in pots. You may want to try a quarter or third of a 55 gallon drum filled with topsoil and covered with some sand or gravel to get them growing and to keep them there.
robert B
