I use homebrew in /homebrew for exactly this reason. The homebrew folks make noise about this, but I don't install language specific packages via brew, so I've never had a problem.
> I don't want my package manager taking it over. That's kept me away from Homebrew
That was an early blocker for me as well; however, when I discovered that I could just install homebrew at ~/.homebrew, that made it all work out.
I created https://github.com/wilmoore/homebrew-home to automate this; however, you technically don't need this install script as you could do it yourself if you really wanted to. I like it because I no longer have to think about it.
It does bother me that Homebrew advocates using /usr/local, that is a choice that needs justification and I don't believe they offer anything convincing.
That said, you can configure homebrew to use any path (check the wiki), you just have to set other environment variables as needed (eg, PATH). For example I'm currently using /opt/homebrew.
(note: a large downside to this is that you have to compile everything and cannot take advantage of Homebrew's pre-compiled "bottles" for some packages)
