ChatGPT Contradicts on Repeated Science Questions

🔑 Enhanced Key Takeaways

• •The study tested ChatGPT across two versions (GPT-3.5 in 2024 and GPT-5 mini in 2025) using 719 hypotheses from business journal papers published since 2021, revealing that accuracy improvements from 76.5% to 80% remain marginal when accounting for random guessing baseline (50% chance on true/false questions).
• •ChatGPT's inconsistency rate is severe: when identical prompts were repeated 10 times, the model achieved only 73% consistency in statement evaluation, meaning roughly 1 in 4 repeated queries produced different answers on the same scientific claim.
• •The model exhibits asymmetric failure modes, correctly identifying false hypotheses only 16.4% of the time—substantially worse than true hypothesis identification—suggesting ChatGPT has a systematic bias toward confirming statements rather than detecting falsehoods.
• •Researcher Mesut Cicek characterized current AI tools as memorization systems without genuine comprehension, stating they 'don't understand the world the way we do' and 'don't have a brain,' framing the accuracy ceiling as a fundamental architectural limitation rather than a training data problem.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline