The meaning of this sentence is that even people who didn't think many things were worth gossiping about thought that he was worth gossiping about. The purpose of this was to show how effective his persona was and how much attention he received by inspiring this 'romantic speculation'.