Day-3

Part-1

Through trace analysis, we followed the knowledge base retrieval flow to find the root cause. We confirmed that the AI retrieved the correct technical content and the correct article. However, it cited the wrong articles in the reference section. The issue lies in how the email writer extracts IDs. It only extracts IDs that are explicitly mentioned in the email body. If the primary solution was listed only as a URL, the system did not extract it. Instead, it extracted supplementary articles that were mentioned by their ID. This led to a systemic issue where all evaluated cases cited extra or alternative articles.

Part-2

We discovered that forty percent of the cases do not cite the expected primary solution. On average, there are nearly three extra references per case. To fix this, we need to update the email writer prompt immediately. It must always include the primary solution first, followed by any prerequisites. Supplementary troubleshooting articles should come last. We must never list troubleshooting articles without the main solution. For long-term improvement, we should separate reference types and classify them during research. It is also important to validate that the primary article is present before we finalize the response. These changes will ensure accurate and helpful references for the user.