Podcast Summary
AI-generated code maintainability: AI-generated code may run and compile but its long-term impact on productivity and organization health depends on its comprehensibility to future maintainers. Recent studies suggest an increase in copy-pasted code and faster rate of change in AI-generated code, which can be challenging to maintain.
While AI-generated code may be able to run and compile, its long-term impact on productivity and organization health depends on its comprehensibility to future maintainers. Bill Harding, CEO of Get Clear, discussed his background in programming and how his company's research aimed to assess code quality based on signals around how the code is committed and used. The study found that since AI's proliferation, there has been an increase in copy-pasted code, which can be challenging to maintain, and a faster rate of change in recently committed code. These findings suggest that while AI-generated code may be sufficient in the short term, it may not be optimal for long-term maintainability. It's essential to consider these factors when evaluating the quality and impact of AI-generated code.
Code aging and maintenance: The aging of code repositories presents a challenge for developers, with increased code transformation in the initial stages and decreased changes over time. GitClear's code interpretation engine helps save time by analyzing unchanged code during reviews, while observing an increase in copy pasted code and significant updates within the first two weeks.
The aging of code repositories and the maintenance thereof has become a significant challenge for developers, with about 50% higher rates of code transformation within the first two weeks and a decrease in changes after a year or two. This issue is dependent on developers finding time and methods to update legacy code, which has seen less attention since the proliferation of AI. GitClear, a company focused on helping developers write better code and work with less tech debt, has developed a code interpretation engine that allows for granular analysis of changes, such as moved, copied, and find replace code. This information enables developers to save time by minimizing their attention on unchanged code when reviewing. With access to over 153 million change lines of code from both open-source projects and customers' repositories, GitClear has observed an increase in copy pasted code, although the exact origin of that code, such as Stack Overflow, remains unknown. Previous research showed that about one in four people were copying from Stack Overflow answers. However, a noteworthy finding is the significant increase in code that is edited, updated, or removed within the first two weeks.
AI and code additions: AI in coding may lead to more complicated code and subtle bugs due to an increase in code additions, but it's less effective at suggesting code reductions or optimizing existing code. Developers should exercise caution and consider the potential implications before accepting AI suggestions.
The use of AI in coding has led to an increase in code additions, potentially resulting in more complicated code and subtle bugs. This trend was observed in a study that showed an increase from 64% to 74% of commits containing added code. The reasoning behind this is that developers, especially when fatigued, may be more inclined to accept AI suggestions without fully considering their implications. However, AI is currently less effective at suggesting code reductions or optimizing existing code. Additionally, the study identified an increase in copy-pasted code, some of which may be the result of AI suggestions. The study did not have specific data on the origin of these code blocks, whether they were entirely generated by AI or a combination of human and AI contributions. Overall, while AI can be a valuable tool for suggesting additions, it's important for developers to exercise caution and consider the potential implications before accepting these suggestions.
AI impact on code efficiency: AI tools like Copilot offer quick solutions, potentially reducing the need for developers to search for and move existing methods, increasing productivity and developer satisfaction, but may lead to more frequent code churn and potentially lower-quality code. Further research is needed to understand the long-term implications.
The use of drop and move code, or refactoring, is a significant aspect of a developer's daily work, making up around 25% of all changes in 2020. This trend has since decreased to 17% in 2023, which may be due in part to the increasing availability of AI code assistance. Historically, developers have moved code to reuse similar methods and maximize efficiency. However, AI tools like Copilot offer quick solutions, potentially reducing the need for developers to search for and move existing methods. GitHub's study also reported increased productivity and developer satisfaction. Both findings could be true, as AI tools can help complete tasks faster, but the long-term impact on code quality and the frequency of changes remains to be seen. The convenience of AI assistance may lead to more frequent code churn and potentially lower-quality code. It's important to consider the cost-benefit analysis of these changes. While developers may be producing more code and feeling more energized, they may also be reverting or moving code more frequently, which could impact overall code quality. Further research is needed to fully understand the implications of these trends. In conclusion, the integration of AI tools into development workflows is changing the way developers approach their tasks. While these tools offer significant benefits, it's crucial to consider their impact on long-term code quality and efficiency.
Code consolidation: AI-generated code may lead to decreased productivity and test coverage due to lack of consolidation and reuse, requiring teams to actively seek opportunities for code cleanup
While using AI to generate code can save time in the short term, it may lead to a low percentage of similar methods being consolidated and reused, resulting in a decrease in productivity and test coverage over time. The risk is that teams may end up with multiple untested methods, which can slow down development in the long run. Although larger language models with massive context windows may help, there is a need for teams to actively seek opportunities to reuse and clean up their existing code. However, without proper incentives or technical leadership, teams may not prioritize this task, making it a challenging problem to address. It's important for teams to be aware of this issue and consider the long-term implications of adding code without proper reuse and consolidation.
Gen AI method recognition: Gen AI needs to recognize and reuse existing methods in codebase to produce human-level code, but identifying opportunities for method modifications to make them more flexible is the bigger challenge. AI needs to be trained on code evolution and human tweaks to methods.
While Generative AI (Gen AI) has made significant strides, it still needs to be able to recognize and reuse existing methods within a codebase to produce code on par with human developers. The first step is for Gen AI to have a large enough token size to enable it to recognize existing methods. However, the second and more challenging step is for Gen AI to identify opportunities for existing methods to be modified slightly to make them more flexible and applicable to different contexts. This will require an AI that has been trained on how code evolves and where humans often make tweaks to existing methods. While the dream is for Gen AI to be able to cite specific instances of these modifications within a team's codebase, this is currently not a short-term solution. In the meantime, teams can benefit from using tools like Stack Overflow to find and learn from existing solutions, and from research and resources like Get Clear to improve code quality and make pull request reviews faster.