You are, effectively, getting "carburetor icing". It's probably not actually icing, but intake temps are dropping low enough to create the same fuel atomization/vaporization problems that exist at cold-startup. The engine "acts lean" because, even though the carburetor is delivering the same amount of fuel (the A:F ratio, by volume, may be virtually unaffected), the amount that is actually being vaporized and burned is reduced.
Consider why more fuel is needed for cold-startup and to keep an engine running, until it's up-to-temp. With the venturi, intake tube, and intake port all cold, liquid gasoline tends to remain liquid. While the combustion chamber & intake valve are also cold, only a small percentage...the most volatile fraction of the fuel...is being vaporized and it's gasoline vapor that burns. The rest passes through the engine, unburned waste. Thus, the engine can run lean on a pig-rich mixture...until it warms up. As each part of the combustion chamber and intake tract warm up, more fuel is atomized, vaporized, and burned, resulting in a functionally/chemically richer mixture which is why you have to open the choke. Now for the big variable, ambient air temp.
Due to latent heat of vaporization, the intake tract is, effectively, a refrigeration unit. Liquid gasoline, exposed to vacuum & vaporized, cools the air/fuel column and everything with which it comes into contact. The effects of this can be: seen, by intake valve coloration; measured by intake tube & carburetor surface temps. In typical riding weather, with ambient temps well above 60F, surface temps remain high enough and far enough upstream, to reach a sustainable equilibrium. Ideally, intake temp should be above ~120F+. However, it can drop well below that without causing major problems. Fuel droplets will condense out of suspension, but only to a point...and...there'll be enough surface heat, downstream, to re-atomize, then vaporize, the liquefied fuel. That is, until ambient air temp intensifies the refrigeration effect enough to overcome engine heat and cool the entire intake tract too much. An IR thermometer can quickly show you what's going on. It's not really necessary. When you're getting the lean stumbles, the intake will be icy-cold to the touch. With my own bikes, the intakes can get cool enough to sweat, like a glass of iced tea, in 85F ambient air...under just the right conditions. Stopping at a red light, after rolling along at (relatively) high speeds for a few miles, the intake will be cool to the touch. That changes quickly, once at idle...due to "heat soak". And that's what you're getting once the engine is shutdown for a minute or three. I've verified this, using a wideband O2 sensor & meter. A:F ratios can go from the high 12s into the low 15s and in as little as a mile, in temps below 50F. A brief shutdown restores the A:F ratio...briefly.
Best you can do is to fit an O-ringed intake tube (manifold) to maximize heat transfer from the head. Being a hardcore fair weather rider, I've not sought such an intake and don't know if one is available. A machine shop could easily cut a groove, in your intake, to fit an O-ring. You might also remove the phenolic insulator/spacer, at the carburetor end, for the cold season; that may require substituting an aluminum. spacer, for fitment.
