In Photoshop, is there a way to turn a vector object into a mask and keep it editable?
I often make objects in Illustrator, and then bring them into Photoshop to use them as a mask, by selecting the pixels of the layer, and then applying it as a mask to another. The downside to this is that whenever I change the smart object, I need to repeat this process.
Is there a way to apply a smart object as a mask to a layer, and keep it as a smart object?
Yes, there's quite a few ways to achieve this.
If you want to paste objects from Illustrator as Smart Objects, then you can use is a a clipping mask for layers above. To clip a layer above the Smart Object, option-click between the layers. You should end up with something like this.
That's not how I'd do it though. The simpler option is just to have a vector mask on your bitmap layer (or vector layer, gradient, pattern or text). To do this, paste your path(s) from Illustrator as a Shape Object (very important!). You'll end up with something like this.
That's a shape layer in Photoshop CS6 (CS5 and earlier versions are similar).
If you'd like to use that path as a mask on another layer, select it using the Path Selection tool (A), then press command-C to copy it.
Then go to the bitmap layer you'd like to apply the vector mask to, and press command-V to paste.
Then choose Layer > Vector Mask > Current Path to use the path as the mask (this was slightly easier in CS5, as it could be all done by option-dragging the vector mask from one layer to another).
You'll end up with a bitmap layer with a vector mask.
Why is this better? It's one layer, not two. Your vector mask is still editable. You can also now clip other layers to this layer.
There's a lot of reasons why I don't like using Smart Objects. In most cases, Shape layers are better, unless you need the Smart Object to contain more than one colour or gradient itself, which is pretty rare for me.
There's a few other ways to mask, including using group masks. It really depends on what's important though.