Hi Rachelle,
I don't use fabric, I prefer paper. I use white & black. I know some photographers use black velvet, or synthetic versions, because it absorbs light without reflections.
Just remember that most camera light meters will try and render all objects to a grey colour, (18% grey). So if you shoot black or white backgrounds without exposure compensation you will end up in both instances with a grey background, perhaps this is what happened to the darker background you used.
Norman