Like, there’s lots of information about Bilbo Baggins in Lotr, that doesn’t mean it was written in the third age of Middle Earth homie
The conceit of the LOTR appendices is that Lord of the Rings, as published in English, is really just the Red Book that Bilbo writes at the end. Dr. Tolkien merely found the manuscript somewhere and has graciously translated it from Third Age common language into English for the benefit of us modern people.
The deal with LLMs is that it’s very difficult to say which piece of training material went into which output. Everything gets chopped up and mixed, and it’s computationally difficult to run backwards.
My understanding of the image generators is that they operate one pixel at a time too, looking only at neighboring pixels. So in that sense, it’s not correct to say they understand the context of anything.