It is interesting, this whole different worlds thing.
My father was a country boy from Moonta Mines, whose parents came from Cornwall, and my mother a Pom from Sheffield - both left school at 14.
I never heard a racist remark from either of them, but some of the cousins on my mother's side displayed it occasionally.
(Those from the excessively drinking branch of the family.

)
A friend of my older sister had a younger sister, who was an adopted aboriginal girl - one of the "stolen generation".
She got a University education, married an Engineer, and moved to Canada with him. (Her adoptive father worked in the Physics Department as a technician.)
Our immediate neighbours, when I was a child growing up in Bowden, were an elderly childless couple.
"Joe", as he insisted I called him, had been born in Glenelg in 1872, when it was a separate town and had been a Blacksmith there.
Neither he nor his wife ever said racially critical things that I heard either, though he referred to women's breasts as "nungies".
I agree terms like "black as the Ace of Spades" were about the society, but seemed to be descriptive terms rather than meant as down-putting - rather like "white as a sheet" which is still in use, and which I still see as descriptive, not racist.
Can one describe skin colour with being racist?
I would have thought so. ( I'm a light tan, with grey-green eyes myself.)