When can I switch from 존댓말 to 반말?

I've been talking with the same conversation partner for a few weeks now and we're around the same age. At what point is it okay to use 반말? Do I just ask? I'm scared of being rude, but using 존댓말 all the time feels a bit distant too. How did you all handle this?