They really aren't. 500 million sounds like a lot, but it is very unlikely any call by anyone in your family is caught in this net. As someone pointed out, with 250 million people owning phones in this country, as well as vast networks of systematic and/or automated calling, the portion of calls actually being tracked here is pretty tiny.
It's also a situation where having useless data costs next to nothing (economically at least, philosophically is a different debate), but missing even a small detail could be critical.
We knew five years ago that the NSA was receiving "billions of [call detail] records ... per day", just indiscriminately collecting every CDR they could -- from everyone (including my grandma and yours), not just criminals or "terrorists" or whomever the latest bogeyman is.
I have no reason to believe that that has changed, that they have stopped collecting all of this data that is so easily available to them.
With that in mind, I do not believe that "it is very unlikely any call" by anyone in my family was included in the catch. I believe it very likely that many calls by many people in my family was -- and continues to be -- included in their "net".
It's also a situation where having useless data costs next to nothing (economically at least, philosophically is a different debate), but missing even a small detail could be critical.