It is not irrelevant. If you can't build the binary from code yourself you have no way to know if it is trustworthy. Step one in finding out if it is is to look at the source and then compile it. After that you can look if it does strange things because of something you missed in the code but without step one you might as well not start at all. It will always be at most guesswork.
Virtually no one who uses software compiles it themselves, so this is not a very interesting rebuttal.
Meanwhile, it's not 1994 anymore, and people who know how to look for bugs can (I know this is hard for some people to wrap their heads around) look inside of binaries and draw conclusions about how programs work. There's a name for it; it's a kind of engineering.
