By derivative work: Pbroks13 (talk) Hypersphere_coord.gif: Claudio Rocchini (Hypersphere_coord.gif) [CC-BY-3.0 (www.creativecommons.org/licenses/by/3.0)], via Wikimedia Commons
Richard Elwes, a visiting fellow in mathematics at the University of Leeds, has drawn some interesting parallels between descriptions of Yog-Sothoth, and current ideas of the geometry of higher dimension. Not only would a higher dimensional object look bizarre to us, as per Carl Sagan's famous take in Cosmos on Edwin A. Abbot's Flatland
but it might fold and morph in bizarre ways. One shape, called an exotic sphere, could resemble the "congeries of globes" Lovecraft used to describe the dimensional gate-thing known as Yog-Sothoth.
An even more intriguing parallel, to this curator, derives from Elwes' discussion of topology, and Henri Poincaré's conjecture that all basic shapes are reducible to spheres. This folding and stretching of shapes brings to mind the hyperspace forms of Keziah Mason and other travelers in other dimensions in "The Dreams in the Witch-House," a story completely concerning higher conjectural physics and mathematics:
"Of his own condition he could not well judge, for sight of his arms, legs, and torso seemed always cut off by some odd disarrangement of perspective; but he felt that his physical organization and faculties were somehow marvellously transmuted and obliquely projected—though not without a certain grotesque relationship to his normal proportions and properties."
"Those organic entities whose motions seemed least flagrantly irrelevant and unmotivated were probably projections of life-forms from our own planet, including human beings. What the others were in their own dimensional sphere or spheres he dared not try to think. Two of the less irrelevantly moving things—a rather large congeries of iridescent, prolately spheroidal bubbles and a very much smaller polyhedron of unknown colours and rapidly shifting surface angles"
The transformations of Keziah Mason, Walter Gilman, and others are very reminiscent of the topological twisting and exotic forms of spheres in higher dimensions that Elwes describes, like so
The amazing thing is that Lovecraft hated math. He wasn't bad at geometry, but had to retake algebra to get a mediocre grade. It was the subject that troubled him the most in high school, and there have even been suggestions (by at least S. T. Joshi) that Lovecraft's mysterious "breakdown" in his late teens and early 20s might have been due to the realization that his math scores likely prevented him from pursuing a career in science as he had planned for most of his young life. Lovecraft eventually turned his hand towards writing and fiction (an interest of his before, but largely put aside in his teens for astronomy and chemistry). Given his fixation on the Romans, I'm not sure why HPL didn't decide he could pursue history, Classical scholarship, or even archaeology, but he didn't, though as with the sciences, he kept himself somewhat educated on these topics throughout his life. So, instead of watching the heavens or diagramming the structure of materials, Lovecraft turned his aborted interest in science into his cosmic horrors, infusing a love of the exotic and weird with the jaw-dropping realities of space, time, and the nature of reality.
At Above Top Secret, Frater210 points to Thomas Hull's article "H. P. Lovecraft: A Horror in Higher Dimensions," in the February 2006 issue of Math Horizons. Hull notes that exotic math is an integral part of the Cthulhu Mythos, including staples such as hypergeometry and interdimensional travel masquerading as magic, non-Euclidean geometry, and vastly huge amounts of time measured in vigintillions. As Hull suggests, Lovecraft does seem to have followed relativity and other developments in physics in the early 20th century, at least as an auto-didactic layman. But whether Lovecraft grasped some of the weird ramifications of these new models, or if he simply got lucky in creating his strange entities and realms, he did it without any real formal training in math.