I sailed with a big solo club fleet and there was everything from a 48ft Beneteau to a Fisher 25 like Dylan Winter's. So, like many things else, it's mostly down to personal preference and the first thing to do is sort out the real extent of your ambitions.
I like a well ballasted, cruiser racer of the older school, with substantial keel, and skeg, low windage hull, tiller steering, balanced ends, sprayhood (dodger), instruments you can see from the helm and a reliable inboard engine. These boats can be big enough to be very comfortable and perform nicely, yet small enough to be handy and cheap to moor. The sails will very rarely overwhelm the helm when overpressed, they are directionally stable, easy on the autohelm and go upwind quietly and well. You can steer from almost behind the sprayhood and winches generally fall to hand, they are a stable platform if you do need to perform on deck.
From a dinghy, to thinking about J24's, that probably won't appeal and you did say keelboat. However I have a lot of sympathy with slug's point; buy a boat that you think looks good.
J/24's are suitable for singlehanded if you're running the #3. However, the J/22 is a much lighter and easier to handle solo platform. You really can't go wrong with a J/22 as it's a nice balance of lightweight performance and stability in a comfortable and versatile package.
If you're tight on dough like me, I "upgraded" from dinghies to a used Mirage 27 ($5000 CAD) that I take out all the time by myself and have a blast. I just make sure my radios close by and there's not much on the shelves to fall on the floor. Plus I've learned a lot about fixing stuff.
To the OP: lots of good but random ideas here, but if you want more focused advice, you are going to have to be much more specific about your needs and wants, and budget. I think the hardest part of that is excluding the features and capabilities that you might dream of, but never actually use.
Decide what you want the boat to achieve within your budget eg; Day sailing, offshore, circumnavigation via the great capes etc. Then find one that has the fewest compromises. Oh, and some resale value as you'll probably want to change in a couple of years.
Where are you going to sail it? eg. Here in SoCal where light winds are the norm I would echo the praise for a J 22. In a breezier venue, maybe something less lively and possibly an older design. Either the old CCA or the newer IOR era Ranger 26s, for example. The old 70s Santana 25 used to be popular up in the Bay Area and its a very nicely balanced, but not terribly sexy looking, design. Hard to beat the Alerion if you have the bread. I see one single handing out on SD Bay all the time having a ball and going boat for boat with larger, crewed boats. The S2s seem to have a strong following in the Midwest and E Coast but not to common out west. All of he above, except the Alerion, can be had for a song if you shop well. Your average planing one design, not so much.