My manager doesn't think my team should spend time "socializing" with the rest of our company...

My manager recently told me that he doesn't think our team should be spending as much time with our coworkers...I asked what he meant and he told me he believes that the hard work we put in is enough to prove that we belong. I found this so puzzling, as someone who believes that it's important that work be more than just a place to get work done and a place for human connection and growth I told him I did not agree, but I now find it hard to feel like I am under the guidance of a leader who has similar values to me. Ever since he said he didn't think we should spend so much time going to work events, like happy hours or bowling or anything that is out of office or away from computer, I feel like he is watching if I go to any of these events or spend time speaking with a coworker and not at my computer every second of the day. Anyone have any advice?? I don't really know what to do to change his mind, and I also don't want to leave, because I do love the company, but I'm struggling to be motivated to work hard for him ever since this happened.