I just notice it now as I am new to Illustrator.
When I am in RGB mode, blending transparency seems to be easier as mode such as Screen, Multiply or Overlay looks good naturally.
If I switch the document mode to CMYK then the image is getting messed up.
I'd like to know, if CMYK is not ideal for blending options?
RGB or CMYK blending modes reflect what is possible in real world use.
Many blending modes rely on the interaction of light though colors. RGB is an additive color model - adding all colors together produces white. (add to get white). Because of how RGB colors work, it's possible to "filter" one aspect of a color and allow light to pass through the remaining aspects in addition to "stacking" aspects of separate colors. These different "filtering" techniques produce the blending modes.
When working with CMYK colors, light is not a factor in the same manner. CMYK is a subtractive color model - adding all colors together produces black. (subtract to get white) When ink is on paper, there is no light passing through anything. THerefore, there is no way to deploy blending modes. You simply can not interact with inks the same way you can with light.
Illustrator, being vector-based, creates object where ink will fall.
The basis of all printing is color separations. This entails printing each color independently on a sheet of stock (paper). To sum up the process briefly, paper is run through a press and one color of ink is used. Then the paper is run through the press again and a different color ink is applied. This is repeated as many times as are needed. In most cases that entails 4 passes of the paper -- Cyan, Magenta, Yellow, and Black. (Additional passes can be used though). The only interactions the inks have with one another is lying on top of previously printed inks (overprinting). Therefore the only blend modes which reflect real-world CMYK printing are those such as Multiply. Or blending modes which are additive (build up colors). All other blend modes which tend to lighten or otherwise reduce the hue or saturation (overlay, screen, etc.) of a color won't work because you simply can't do that with ink. It is not possible to lighten or desaturate ink which has already been printed on the paper.
Now, I expect someone will come along and state something like "Photoshop allows you to use blending modes in CMYK images." This is true. But the difference is mainly that Photoshop is not a vector application. When you save a Photoshop file all colors are applied via raster methods. Vector layers or Shape layers are merely vector clipping masks on a raster fill. So, Photoshop is using a composite image to generate the final output. Illustrator doesn't use a composite for output, it uses the internal vector data.
It could be argued that there's no reason Illustrator should not allow blending modes in CMYK and it would be a fair argument. After all, it would merely take some application development to split vector paths upon output and adjust blending mode areas for proper color. Basically on-the-fly flattening within Illustrator. However, if the application actually showed this behavior you would find the artwork you draw to be constantly changing and becoming more and more difficult to edit. This is why Illustrator has a user-instigated flattening option. So that you can be aware of when artwork construction is changing and control it to some degree.
In short..... some blending modes are simply not physically possible in CMYK color mode.