Generative AI and ChatGPT have created a lot of buzz lately, sparking discussions about how and if this technology can replace other tasks or jobs. One field where this topic arises is that of internal documentation for software teams and begs us to ask ourselves: can generative AI replace the need for creating documentation altogether?
Recently, GitHub began to answer this question when it announced Copilot for Docs, a ChatGPT-like interface so devs can get AI-generated responses to questions about documentation 🤯 GitHub’s announcement is clear about the premise of Copilot for Docs; Feed Copilot with your documents, and then query information using natural language.
There are a few directions to explore here. The first involves removing the need for documentation altogether, where AI would help explain the code or task at hand. Secondly, AI has the ability to create and maintain accurate and up-to-date documentation automatically, which reduces the burden on software teams who must manually update docs. Additionally, generative AI can use natural language processing to generate documentation that is easy for developers to understand, making knowledge about code more accessible to team members.
However, there are also potential drawbacks to using generative AI for internal documentation, such as the need for significant training data and the potential for errors in the content generated. Let’s dive in.
Why write internal documentation?
Let’s take a step back. Considering that GitHub’s Copilot for Docs searches the existing documentation, where does this documentation come from?
As a first step, a developer needs to write the documentation. As the code evolves, someone needs to maintain the document, that is, keep it up to date with the current version of the code. Then, when another developer needs the information found in the document, they will hopefully find the document when it’s relevant, and get the information from it.
In general, we can say that the lifecycle of internal documentation goes:
- Creation
- Maintenance
- Discovery
Out of these three steps, ChatGPT assists with the first – creation – while Copilot for Docs focuses on the last – discovery. The potential for improvement in this last step is substantial. One aspect is finding the relevant info without getting to the document where it resides – just ask the question, and Copilot will provide you the answer, that’s it.
What does good documentation achieve?
Before asking ourselves if generative AI could replace documentation, we should consider what good documentation includes, and what it aims to achieve.
When good documentation exists, developers can contribute to new and unfamiliar code with ease. As an Engineering Leader, good documentation should provide the freedom to assign tasks to everyone on your team. To achieve that, documentation should provide the engineers with the high-level overview they need to get the context for the codebase, as well as the specific details required for implementing the task at hand.
Now we can turn to the question: can generative AI create such documentation? Specifically, how will the generative model perform when given access to the entire codebase?
So, will Copilot for Docs or ChatGPT replace internal documentation?
In two words – not really. To explain, let’s break it down in terms of both platforms:
CoPilot for Docs
Copilot for Docs does not try to generate internal documentation, perhaps because meaningful internal documentation also relies on information that is not found in code. For example, one vital role of documentation is to describe the business logic behind the codebase/service. This knowledge cannot be extracted from code, no matter how sophisticated the model analyzing the code is.
Another important aspect of documentation is explaining design decisions. Moreover, important design documents often explain why a certain design was not chosen as a solution, in which case it will obviously not exist in the code. In general, there are some aspects of knowledge about software that cannot be extracted from the source code, so generative AI couldn’t just replace them. Read more in our in-depth post about creating internal documentation using generative AI.
However, when it comes to maintenance, things get interesting. If we provide our model with outdated or inaccurate docs, we may get… Inaccurate answers for our questions. This is a classic case of “garbage in, garbage out”.
ChatGPT
OpenAI’s service can definitely help write documentation… so engineers can write documents that are more helpful, since they wrote the software, and understand the business logic, design decisions and underlying assumptions they made while working on the code.
And there are several other amazing capabilities, including:
- Styling of documents. Generative AI models already do a great job creating introductions, summarizing documents, and helping developers structure their documents by suggesting sections and so on.
- Creation of documents with grammatically correct and coherent language. Not all engineers enjoy writing, and not all are adept at it. IfEnglish isn’t your dominant language, writing technical documents may be daunting. AI models are already embedded in tools and products to help engineers express themselves, and with generative AI models, it may be even easier to interact with them.
- Code insight. AI models can also reflect on the code, and help engineers create the desired documents. As mentioned earlier, a model could, for instance, identify best practices that are common within the repository, and suggest mentioning these practices.
That said, again, there’s no accounting for accuracy…
Beyond creating documentation
Clearly, creating documents is only the first step. To get value from the documents, they need to be up to date, or maintained, and developers need to find the relevant documents when needed.
Even if an engineering team decides to fully rely on generative AI to assist or create document drafts, as the code evolves – it will quickly become obsolete or misleading. To keep documentation up to date, other tools are required.
For example, with Swimm, developers create walkthrough documents – engaging cross-repository docs with content components (e.g., variables, tokens, file/folder paths, code snippets, diagrams) that interact directly with the code and are automatically synchronized as changes are made. Documentation is kept up to date using patented AI that makes recommendations as part of CI/CD workflow. On every PR, the algorithms indicate when everything is okay, alert you when something needs reviewing, give you corrections to approve, and can even automatically correct simple issues for you. This way, the documentation is always up to date, and developers can trust it.
But, what good is a document when you can’t find it? This is another crucial aspect of documentation, regardless of how it was created – using generative AI, a human developer, or a combination. Using Swimm’s IDE plugins, developers don’t have to actively search for documentation. Relevant documents find them.
While working within the IDE, and reading through code, Swimm’s plugin shows a comment from a relevant document that may be helpful for this part of the code. Upon clicking on the comment, the document opens right from the IDE. In addition to the “classic” ways of indexing documents by tagging or organizing them within folders, this new way of finding documentation makes sure you really have the docs as you need them.
Conclusion
Will GitHub Copilot for Docs or ChatGPT replace internal documentation? Well, no. But, Generative AI is here to stay and will disrupt many industries. In this post, we considered its possible implications for internal documentation, particularly in the context of the internal documentation lifecycle. And we are very excited about it 😍
We believe GitHub Copilot for Docs, along with ChatGPT will have a huge impact on the way we discover documentation, and more importantly, the relevant information when we need it. Yet, to really revolutionize how we share knowledge about code, we also need to create useful documents, and then maintain them as the code changes.
While AI can definitely help with these tasks, we still need other tools that will help us create the most effective documentation, and then keep the documents up to date. More to the point, this technology poses an amazing opportunity for many fields, and the ability to create high-quality software documentation, specifically. By utilizing its potential while understanding its limitations with other tools, we can form a new, more efficient reality.
