Why is InDesign pushing short words to the next line?
As shown in the image, "the" easily fits on the line above but regardless of what I do, InDesign pushes it down to the next line.
The other answers are correct that this is down to the work of the Adobe Paragraph Composer which is balancing your whole paragraph as opposed to each line individually.
If you do want your "the" to stay on the previous line (without changing to the Single-line Composer) just replace the preceding space with a non-breaking space (Type â†’ Insert White Space â†’ Nonbreaking Space or cmd+alt+X). Although InDesign generally does a good job at this so I would leave it without careful consideration.