Software that promises to detect AI-produced text fails to deliver.

Researchers create 30 fake student accounts to submit model-generated responses to real exams. Professors grade the 200 or 1500 word responses from the AI undergrads and gave them better grades than real students 84% of the time. 6% of the bot respondents did get caught, though… for being too good. Meanwhile, AI detection tools? Total bunk.

Will AI be the new calculator… or the death of us all (obviously the only alternative).

Note: the software was NOT as good on the advanced exams, even though it handled the easier stuff.

  • kromem@lemmy.worldEnglish
    1·
    5 months ago

    This is incorrect as was shown last year with the Skill-Mix research:

    Furthermore, simple probability calculations indicate that GPT-4’s reasonable performance on k=5 is suggestive of going beyond “stochastic parrot” behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.