For example, if I had a dark blue or black background choosing white text will make it stand-out and more readable.
If I had a white or off-white background, black text will stand-out and be readable.
What do I do if the background color is neither dark enough for white to show well nor light enough for black to show well?
What you are looking for is contrast
In order to measure contrast there are a lot of ways, I personally use this website. It grades the contrast between colors, and I personally found that grade 5 and above is a pretty good contrast for type and its background