As stated above, I didn't - in fact, I was well gone 50 when I got my first 'real' railway model - I had had a Lone Star die-cast set (or parts thereof), but that barely counts, I think. Few of the other military kids had one either, for the same logistical reasons.
Living in England for most of the '60s, I don't actually remember more than one of the civilian kids having one - and his father was a solicitor. It was seen as a "posh kids" toy, I think, certainly in my circles.
And houses were that bloody cold that, for six months of the year, you only really had the one habitable room.