You've gone for the headline there, not the actual point.
If you listen to the end of his answer (and the bit that quite unsurprisingly that source clipped out) you see that the point he's actually making is that to negotiate to go back in will result in years of uncertainty, and uncertainty hurts growth.
And that's really the main point. We have left we can't just "go back in". There isn't actually a way to quickly rejoin just the single market. To do so requires membership of the EEA, and Norway are fairly clear that they would block that. So to get "membership" of the single market would require years of asking and negotiating. And all the while investment would stall, as who's going to invest in the UK when you don't know what the rules are going to be in 5 years time?
The only way to get back what we have lost is to rejoin the EU in full. And that will not be a short term project. It is very unlikely that the EU would welcome us back until it was absolutely clear that there was very strong support in parliament for this. Which unfortunately is going to take at least a decade, whilst the public wake up to the fact that they were sold a pup.
He's absolutely right in one sense, that the best way forward from here is to negotiate a "better" trading relationship. And it's also absolutely clear that for that to happen there needs to be a different set of people in charge, without the absurd fixation on FoM, who can make sensible decisions on what the UK can compromise on in order to minimise the damage going forward. And who haven't just pissed the EU off with 5 years of lies and bullshit. The jury is out on if Starmer has the freedom to actually do that, or if he's still too scared of the "red wall" voters and their irrational hatred of foreigners.
But that will still result in something that isn't as good as what we had.
In a way, I applaud Starmer for playing the hand he's dealt rather than just saying that of course growth would be better if we hadn't left the EU. Unlike the Tories he does at least seem to have recognised both the limitations of the situation we find ourselves in (both domestically politically, and our terrible negotiating position with the EU) and also that we do need a new approach, as what we have very clearly is terrible.
And if you want another example of quite why we're so pissed off at you, here's a great example. The changes you have wrought on the country in the name of English exceptionalism are not like those that are made after every change of government, where they can be largely reversed at the next GE if the electorate wishes. You've baked this crap into the UK for at least a decade. That's a decade of lost opportunity, lost growth, lost international reputation.
And that will mean we'll forever fall back relative to the EU and other developed nations.
We voted for changes as versus Brussels management and policy the UK is truly truly exceptional. Brussels can't even get the basics right so to be fair everyone is exceptional versus them.
Being in the SM and CU would make no difference to GDP growth. That's what Starmer said and he is right. Even the most pro EU models which said we would have no trade deals with anyone (not EU, not Japan to any other the dozens of others we rolled over) and had lots of pro EU assumptions still only came to GDP impact of a drag of 0.2% GDP per annum.
Outside the EU the UK is going to materially outperform France and Germany. Wait and see. We are only just exiting Covid, and maybe not even yet.