Everyone ate it too. The mockery was because
- they were messy to eat
- they were staples commonly eaten
- they were made and sold by black people early in their steps of economic independence following slavery.
- racism doesn’t have to make sense.
If you hate someone, anything they do can be something you use to express your hate, even if you do it to.
Robots.txt isn’t even a rule, it’s a request.
“Please do not ask for the following content if you are a robot”.
If you don’t want someone to look at your content, you ultimately have to not give it to them, not just ask them to not ask.