So can a 4K set up-scale an HD pic and actually make it look better? I remember when HD blu-ray players were boasting about up-scaling a regular DVD, but I never really saw good improvement.
Yes, it can look better. Remember that during the timeframe the TV makers were able to go from 2K technology to 4K technology the silicon makers and algorithm experts were also spending fortunes developing better means of scaling content in real time. So, while the early HD TVs lacked decent scaling technology, and only some of the best DVD Players with scaling chips were even "acceptable" to viewers, today's technology allows for amazing scaling abilities.
As I wrote a few years ago about 4K televisions (which resulted in a ton of rude responses saying we never need something like that in our homes), for every single pixel on a regular HD signal, there are 4 pixels on a 4K screen. That allows a scaling processor to use interpolation and other techniques to remove stair-stepping and the stuff you'd expect. But it also introduces other capabilities, such as adding to the perceived colors and grayscale levels by mixing shades.
For example, and this is real, if you have 256 shades of gray available in a source, but the background in a scene runs from an infinite black at one side and an infinite white at the other (such as a beam of light piercing through a gap in the shades on a window and washing across a white wall behind the actors), the limit of 256 shades could potentially introduce bands of gray where the compression or format is limited. With a good scaling engine that sort of scene could easily be recognized as an infinite transition from white to black and the output could use not only more pixels to smooth out the transitions, but add dots of adjacent shades into each original pixel to trick our eyes into seeing a shade the signal cannot produce. Your a photographer, you see that effect when you make an image fully black and white be removing all the gray content. There are still what appear to shades of gray in certain transitions. Or, even more memorable, that's how a black & white laser printer can claim to offer grayscale output - the toner is black and the page is white, but by spattering tiny dots of black toner with gaps of white it appears to be gray to the eye. A good scaling algorithm and processor can do that. So, instead of a brick wall of 256 shades of gray, the higher resolution screen can be used create what appears to be 512 shades (easy to do) up to 1024 shades of gray. And that's just for black and white. Add color and standard color space content can appear to be in HDR on a non-HDR screen pretty easily - and just imagine what a true HDR screen could do???
So, yes, the new 4K and future 8K televisions will offer a better experience even with old format content - assuming the video processors are up to snuff. And these days most video processors are very good and some are insanely impressive.