I'm still struggling a bit with the whole "What next?" question, though I think I've mostly come to a decision. I finally got a meeting earlier this week with one of the other profs in my research group, who has been around a bit longer than my supervisor, and has had more female grad students, but has moved up the university chain of command and is quite hard to get time with. In the end, he basically had the same things to say as my supervisor, but put it in not such a doom-and-gloomy type way. He and I talked about the risks of either path; going down the academic path now can put you in a tough spot to have children, and having children sooner rather than later can put you in tough spot when you're trying to get back in grad school.
I think that for me, it mostly comes down to, "Which would I regret not having more, if my life were to turn out that I could only have one?" And for me, it's definitely children. So I'm not going to apply to grad school right now. I'm going to try to do a bit of teaching or maybe find some industry work (what industry wants someone with a master's in physics?) until Matt and I decide the time is right to have kids. I can always try to go back to science academia later, and if it doesn't work out, that's going to be OK too. I hope I can come back to it someday, but I think that having a family is too important to me to take the risk of jeopardizing that right now.
The decision was cemented today when I got email from APS's Women In Physics mailing list. It has a link to Academe, a magazine published by the American Association of University Professors. This issue is about the great number of challenges still facing women in academia and particularly in the sciences when trying to further their careers and develop families as well. It has just reinforced all the frustrations I feel with the field, and highlights once again the pitfalls I'd be facing. Though I do somewhat regret not sticking around now to try and make it better for those coming after me, I will still try to find ways to encourage change from outside the system, and hopefully inside it later on.