An experiment using ChatGPT to take a practice CPA exam produced some unexpected results, Accounting Today found, and it required repeated revisions of its prompts.

Experimenting “extensively” with the tool, the publication determined that “there are ways to use ChatGPT to produce results that are, if not optimal, at least better than a completely unguided prompt.”

Being as specific as possible in a prompt is important. That's because just as someone "who is given vague instructions won't do a good job, neither will ChatGPT.” So, users should be sure to provide a “detailed message applicable for [a] particular situation.”

The user must also recognize that ChatGPT will take a few tries before it comes up with the right answer. “Getting the desired output is often an iterative process,” so the user may have to keep providing more details and instructions as he or she receives each response. It is also important to remember that conversations are not private, so a user should not input any sensitive information.

Setting the right context and scope is crucial to obtaining more accurate and relevant responses. The user needs to provide the tool with information such as telling it to assume the role of a student taking the test.  

Helping ChatCGT to help the user provide the best answers may also require the user to break down questions into simple parts, as one long complicated prompt could result in poor answers. As an example, Accounting Today suggested, instead of writing, "Tell me about climate change,” the user can ask discrete questions such as “What are the sources of climate change?"

ChatCGT’s answers are based on what already exists on the internet, which means that it often incorporates information that is incorrect, or it just fabricates things. Anyone using the tool needs to check facts before accepting the responses. Compounding this issue is that ChatCGT’s knowledge base stops at 2021.

Daniel Street, an accounting professor at Bucknell University, is a co-author of a study that using "an innovative interview method to introduce CPAs to ChatGPT, its development, and the current state of its capabilities." He developed a set of five accounting-specific principles for ChatGPT use. They are:

1. Accountants should use large language models like ChatGPT as an ability enhancer for human thought and judgment, rather than a replacement. They should think of it as more akin to a robotic arm than an autonomous robot.

2. Decompose complex tasks. For instance, instead of just asking it to make a balance sheet, a user should ask it to build first the liability section, then the asset section, then the equity section over the course of three prompts.

3. Exercise professional skepticism.

4. 4. Be wary of LLMs’ quantitative skills. Remember that models like ChatGPT may have trouble doing calculations and so is not very effective at quantitative tasks.

5. Provide reinforcement to ChatGPT to improve its functioning. If users gets an output that is not correct, they should provide feedback by pointing out it was wrong, as well as clarifying what they want to see. ChatGPT runs on a reinforcement learning model, so the more it is told it what was and was not a good answer, the more it will adjust itself to the user's preferences.