There used to be a way to do masking tricks to create rounded polygon corners in Photoshop, but I can't seem to find anything for accomplishing the same trick in Photoshop CS6. Does anyone out there have a solution?
Help me Obi-Wan Kenobi, you're my only hope.
It can also be done with a layer mask over the vector shape.
Create your polygon, then load the layers selection by Command-Clicking (Ctrl-click on pc) on the vector mask (the thumbnail of the pentagon in the layers palette below).
Then go to Select>Modify>Smooth. Enter the smooth amount.
After that it's just a matter of creating a layer mask based on the now rounded selection.
Now you have your rounded polygon fairly quickly and easily. You could also manually edit the shape with by adding points and using the convert point tool, but that would involve a great deal more work.