How do I create a worn/distressed effect in Photoshop?

I think there's more to this than noise, to me it looks like the button has had some sort of grunge texture applied. You say you don't want to apply a "pattern/texture", but I presume you mean a repeating pattern or texture, but you can get textures which consist of non-uniform high-resolution photos, which is one way to achieve this sort of effect. I tend to use one of two methods:

1. Photoshop brushes, of which there are many free "grunge" sets available online: create a layer mask and paint with your grunge brush to taste
2. Textures, typically grungy photos of crumbling concrete or similar: find an appropriate photo, put in a new layer, and adjust levels etc until it looks like it has the right amount of contrast for a layer mask (usually higher contrast than the original photo). Then select and copy your edited grungy photo layer, create a layer mask on the layer you want the effect applied to, select the mask, go to channels, and paste in your wear.