Not wearing a bra can lead to saggier breasts, depending on their size. At the same time, wearing an ill-fitted bra can also do damage. “Listen to your body.