So I have this image as vector and a grunge image as vector:
I can easily mask the grunge part from the image in Photoshop:
But when I try to do this on Illustrator with pathfinder always get funky results:
I can use the Pathfinder tool on simple one object at a time, subtract from front etc..
But how can I make this with these all layers and groups?
Pathfinder, while it will work in many cases, is a completely destructive method and once done, it can not be undone or adjusted easily most of the time.
Use an Opacity Mask.
The Opacity Mask method allows you to not only adjust the artwork with the same ease you already have, but also lets you adjust the texture and the amount of knock out the texture has. In addition, Opacity Masks can be used over any number of objects including live text so there's no need to expand anything for an Opacity Mask. Opacity masks can also use raster images (keeping in mind raster images have limitations regarding ppi, scaling, and output).
Clipoptions on the Transparency panel.
To adjust the contrast of the grunge areas, simply change the color of those objects. The darker the color of the texture objects, the more they will mask: