Improving Results Obtained from AI Assisted Development

Index of All Documentation » Wing Pro Reference Manual » AI Assisted Development »


In order to obtain quality results from AI assisted development, you need to carefully consider the size of the tasks you are asking the AI to perform, the clarity of your instructions to the AI, and whether or not you are providing enough context for it to be able to perform the requested task. If tasks are too large or poorly defined, or if you are not providing the AI with enough contextual information, then you will receive results that are either incorrect or require too much manual effort to work with.

For AI code suggestion, your request is essentially defined by what precedes the current editor insertion point. Adding a comment or the first part of the code you want helps the AI to understand what you expect it to write. Importing modules you expect the AI to use is another way to increase the chances that you will get what you are looking for.

For AI refactoring and description-driven code development, both the code in and around your refactoring target or insertion point and your written instructions are important. Be sure that your request is very specific and complete. The AI will understand more than you might expect. For example, you can ask it to "add documentation strings" but may get results closer to what you're looking for if you instead ask it to "add documentation strings but without documenting arguments" or "add documentation written in French".

However, asking the AI to refactor or write too much code at once tends to exceed the computational limits of generative AI, and won't produce good results. Focusing on a single function, method, or class tends to work well. Trying to refactor a large file all at once often does not work well. You will need to experiment to get a sense for these limits. What works is likely to change over time, as generative AI continues to advance.

For both AI code suggestion and AI refactoring, you will also need to think about whether the AI might need additional context that is not found in the current editor file, as described in detail in AI Request Context.

AI assistants share many characteristics of human software developers: Given an overly large task with too little time to complete it, in the case of AI due to limits on computational complexity, they become overwhelmed and give up before coming up with a good result. A poorly specified task may cause them to go off on a tangent and do the wrong work. Too much irrelevant context can similarly confuse them, wasting their time and attention.

As also for humans, work created by an AI will vary somewhat with each invocation, even if the context and request are exactly the same. If you don't like the results that you've obtained, you may sometimes be able to get what you want just by invoking the same request again. In other cases, you may be better off asking the AI to correct specific mistakes or make further enhancements in subsequent requests, rather than expecting to write all the code with a single request.

Revision Control

We strongly recommend using a revision control system so that you can review and adjust code suggested by your AI provider, before it is accepted into your code base. Wing's AI assisted development features will replace code in your files and if you save them to disk without a revision control system and quit Wing, then you will lose the ability to restore your code to the state it was in before your AI request was made.

Performance

AI supported development is compute intensive. Requests often take 5 or 10 seconds to complete, and sometimes much longer than that. However for well-designed requests, the amount of work being done by the AI is considerable, and it often produces results that would take much longer to implement manually.

Be sure to also read Managing Cost so that you understand the approximate costs involved with your AI assisted development.

Language Model

The default configuration in Wing uses OpenAI's gpt-4-1106-preview (GPT 4 Turbo) language model, which is currently the only model available that is likely to produce good results. The reason for this is that this model has a much larger "Content Window" (128K tokens) than any pre-existing model, which allows for much more context to be provided with requests. Wing depends on this ability when Auto-Context is enabled.

For now, OpenAI is imposing fairly low token rate limits and the model is somewhat expensive to use. Because of this you may need to spend enough to obtain at least a Tier 2 account, and more likely will need to go to Tier 4 for any real production use of AI assisted development in Wing. This requires an initial investment of around US $50 into your OpenAI account, as a credit which is then spent slowly as you issue AI requests.

This is likely to change over time, and other models may appear with similarly large context windows and lower pricing.

In our tests, we did not receive good results with any other OpenAI models, including GPT 4. If you try to experiment with older models, you will need to disable Auto-Context and reduce the Max Local Context configured on the Context page of the AI tool, so that requests sent to the model are smaller. See AI Request Context for details.