...When I use the 3D effects, it pixelates the design and then my illustrations are rejected for "rough edges"...

I have used the Illustarator 3D effects for many images which have been accepted at all sites (except IS, where I no longer submit). But it is not as simple as just applying the effect. You have to play around with the 3D effects quite a bit to get usable results.

As for doing 'real' 3D, the most widely used software is Autodesk 3DSMax + VRay rendering engine. But be prepared to pay about $4,000 US or subscribe.

Many other options are available, some free, such as Blender. But be prepared for a long and steep learning curve to get the concepts of 3D modeling and rendering. Learning to use Max or Blender well will probably take at least five or ten times longer than learning Illustrator.

Learning Blender isn't too bad.  Some really good sites to learn from and you can't argue with the price of it.  Learning the interface and finding where everything is will be a bit of a pain.  I've been using it for about 2 years now and didn't start getting serious with it until about last July.  There's a site called cgcookie that has a section for blender that you can sort out the tutorials for beginners.

Here's a series to get you started http://cgcookie.com/blender/category/getting-started/  That series is for Blender 2.5 but the interface is the same in version 2.6.

If you get stuck don't be afraid to ask questions.  I've only gotten the rough edges rejection once on one of my 3d images when I first started to use it for stock.  Learning curve is high but take it in bits and pieces and you'll get the hang of it in no time.

*edit to add*

Just to let you know I don't know everything but still learning.  It's a constant learning process especially when they add new features and stuff to it.

Thanks, Anita! I will take into consideration your message! Thanks!

What about microstock. Is it possible to create vectors in Blender or is there a way to export it in Illustrator and make it somehow a vector there?

Thanks, Microbius! I will look for Cinema 4d too!