How to Create Rounded Corner Hexagon in photoshop using polygon tool?

With the Polygonal Tool selected, click the little gear icon on the Control Bar a and then check the Smooth Corners option.

Photoshop CC or CS6:

Photoshop CS5 or CS4:

You may also need to adjust the "Star" options....