In the summer of 2023, the Halborn engineering team decided to test OpenAI’s ChatGPT on its ability to detect smart contract vulnerabilities. In this article, we’ll summarize our findings which you can read in full here.
ChatGPT’s High Success Rate in Finding Certain Vulnerabilities
In our study, we found that ChatGPT demonstrated a remarkable proficiency in identifying various types of vulnerabilities, including (but not limited to):
Bad randomness
Use of deprecated functions
Right-to-left override control character
Integer overflow
In many cases, the detection of these vulnerabilities involved recognizing specific functions or code patterns, such as the use of tx.origin or certain pragma versions. Overall, ChatGPT achieved a detection ratio of around 75% across all types of vulnerabilities and tested smart contracts.
Boosting ChatGPT’s Accuracy with Specific Prompts
Interestingly, ChatGPT's detection accuracy soared from 76.1% to 86.6% when prompted to identify a specific vulnerability rather than finding all vulnerabilities in a code sample. This highlights the tool's enhanced effectiveness with targeted queries.
Limitations in ChatGPT’s Understanding of Solidity and EVM
Despite its success, ChatGPT faced challenges in understanding the nuances of Solidity or Ethereum Virtual Machine (EVM) workings. It consistently struggled with vulnerabilities like:
HASH collisions with multiple variable length arguments
Reference to an external malicious contract
Abuse of global semantics
Insufficient gas griefing
Storage collisions
Furthermore, different ChatGPT versions had varying degrees of difficulty with certain vulnerabilities.
Can ChatGPT Solve CTFs?
CTF (Capture The Flag) challenges are crucial in cybersecurity training, simulating real-world scenarios to enhance skills in system protection. Our study found that ChatGPT could completely solve CTF challenges 43.3% of the time, with an additional 20% of cases where it offered a partial solution. Success rates varied with the complexity of the challenge.
Performance in Ethernaut and Other CTFs
In the Ethernaut challenge, ChatGPT excelled at simpler tasks but struggled with more complex scenarios. The tool showed differing capabilities in handling challenges from various platforms like Capture the Ether and Damn Vulnerable Defi.
Conclusion
Our study reveals that while ChatGPT is highly effective at identifying certain smart contract vulnerabilities, it has limitations, especially in understanding Solidity and EVM intricacies. The success rate in solving CTFs also varies based on the challenge's complexity and whether the CTF and its solutions were part of ChatGPT's training dataset.
For detailed insights and our top tips for using ChatGPT to detect smart contract vulnerabilities, read Halborn’s full report.