As you can see in the above image, I am trying to rotate this photo, which has been converted to a smart object.
However, the edges are looking like a zigzag. How can I fix this?
If the results are NOT satisfactory
Rotation and scaling all in "one go" may be too much for the anti-aliasing engine in older PS versions to handle and produces the artifacts you mention.
Here is a workaround that admittedly requires you making two "nested" smartobjects (one inside another), so it's definitely a workaround, but it does produce good results, remains fully editable, and does not resort to destructive methods like rasterizing:
You may also try scaling first and then rotating, but if you scale down to under 40%, create a new smart object and then rotate, you may still get artifacts. I don't know why the order makes a difference, it just does.
By the way, this is also the way to go if you are putting a perspectival distortion on a smartobject. FIRST distort, make into new smartobject, THEN scale down.