Just keep working with them, it will get better keeping them on as you build a little more tissue.
Sounds like you're off to a good start.
Many ladies don't have any swelling at all for weeks even months, so you're doing well!
Try not to get frustrated, it will get easier.