ChatGPT always successfully identified the following types of smart contract vulnerabilities:

• Bad randomness
• Use of deprecated functions
• Right-to-left override control character
• Integer overflow
• Missing protection against signature replay
• Typographical error
• Variable shadowing
• Arbitrary jump with function type variable
• Logical error
• Authorization through tx.origin
• Presence of unused variables
• Block values as a proxy for time
• Default visibility
• Asserting EOA from code side
• Numerical Precision/Floating points
• Outdated compiler versions
• Message call with hardcoded gas amount

NO. of vulnerabilities in range of accuracy

100%

Between 75% and 99%

Between 50% and 75%

Between 25% and 49%

Between 0% and 24%

Impressive? Not Really...

In most of these cases, the presence of the vulnerability could have been easily identified by looking for certain functions or code patterns. For example, simply scanning code for the use of tx.origin or a pragma version would identify certain types of vulnerabilities. In general, ChatGPT detection ratio is around 75% with all types of vulnerabilities and in all tested smart contracts. However, we have discovered that ChatGPT can better identify vulnerabilities when prompted if a code sample contained a specific vulnerability (e.g., reentrancy) compared to asking it to find all vulnerabilities in a piece of code.

In fact, the specific vulnerability prompt increased ChatGPT-4's detection accuracy from 76.1% to 86.6%!

while ChatGPT is very effective at finding certain types of vulnerabilities, it struggles with the understanding of how Solidity or how EVM works.

In general, there are certain types of smart contract vulnerabilities that ChatGPT struggles with finding, regardless of the ChatGPT version tested:

Abuse of
Global Semantic

Insufficient
Gas Griefing

Storage
Collisions

Hash collisions with multiple variable length arguments

Reference to an external malicious contract

Furthermore, different versions of ChatGPT struggle with different vulnerabilities.

ChatGPT was able to identify certain vulnerabilities with 100% accuracy — such as variable shadowing or bad randomness— within smart contracts. However, it tends to struggle with different variations of these attacks. For example, most instances of read-only and cross-function reentrancy were not detected during the study. Similarly, ChatGPT overlooked DoS by external calls without gas stipends in 2 out of 3 prompts.

In our study, we found that ChatGPT could completely
solve CTF challenges 43.3% of the time

ChatGPT also offered a partial solution in an additional 20% of cases.

These results depend largely on the complexity of the CTF, with more complex challenges having lower success rates

Ethernaut

• In Ethernaut, ChatGPT was able to solve most of the challenges excelling in the first level with 100% accuracy. Although it was clear that with increasing difficulty, it started to struggle more. 
• With a difficulty level of 4, it’s effectiveness decreased to 25%:

Capture The Ether

• In Capture the Ether, we can observe that ChatGPT is only able to solve correctly 37.5% of the challenges versus 55% in Ethernaut:

Damn Vulnerable DeFi

• This percentage drops even further for Damn Vulnerable Defi’s CTFs, falling to 26.7%. These challenges are often more complex because they typically require an analysis of multiple contracts and how they interact with each other.
• This is different from previous repositories, where challenges usually center on just one contract.

Also, ChatGPT has much higher success rates if the CTF and its solution were published before ChatGPT was trained and, therefore,
were part of the tools’ training data set.