20 Questions — Domain 7 : Troubleshooting and Optimization

Expand any question to reveal the correct answer and explanation.

1. 1 A Claude Project workspace used for monthly campaign summaries starts producing inconsistent results, referencing discontinued product lines and using an outdated brand voice. After reviewing the prompt, which is unchanged, what is the most likely root cause according to Domain 7 diagnostic principles?

   Focus on the maintenance of persistent workspaces and the sources of truth within a Project.

   The project's knowledge base contains superseded files and the instructions have not been updated to reflect current guidelines.

   Configuration maintenance is critical; outdated context in the knowledge base or instructions will override the effectiveness of even a well-written prompt.

   • ✗ The model has reached its internal training data cutoff and can no longer generate accurate brand information.

     Associate-level troubleshooting should first look at user-provided context and project settings rather than assuming model-side training limitations.

   • ✗ Claude is experiencing a temporary 'hallucination spike' due to the complexity of the monthly summary task.

     Hallucinations are typically caused by missing or conflicting context, and diagnosing them as random 'spikes' ignores the actionable fix of updating project files.

   • ✗ The campaign summaries have exceeded the 120-minute session limit for a single project interaction.

     Session limits refer to the time taken to complete an exam or interaction window, not the degradation of output quality over a month.

2. 2 During a long-running research session, Claude begins to lose track of specific constraints established in the first few messages. Which troubleshooting action best mitigates this 'context decay' while preserving essential progress?

   Think about how to refresh the conversation tail without losing the 'essence' of the work.

   Ask Claude to generate a summary of the key decisions and context, then start a fresh chat using that summary as the starting point.

   This strategy clears the 'bloat' of a long conversation history that consumes the context window while retaining the core details needed to continue.

   • ✗ Continue the current session but repeat the original constraints in every third message to keep them 'fresh' in memory.

     Repeating constraints in the same chat only further consumes the context window, eventually worsening the decay issue.

   • ✗ Switch from Claude Sonnet to Claude Opus to utilize a significantly larger context window for the existing history.

     While different models have different limits, switching models mid-session without clearing the history does not address the underlying accumulation of irrelevant conversational tokens.

   • ✗ Delete the early messages in the chat history to manually free up space for new tokens.

     Most Claude interfaces do not allow for the granular deletion of individual historical messages to modify the active context window mid-conversation.

3. 3 An Associate is tasked with automating a high-volume, repetitive workflow that classifies 500 short customer feedback entries per hour. The current workflow uses Claude Opus, but costs are exceeding the budget. What is the most effective optimization?

   Consider the operational trade-offs between the model families for high-volume, rule-based work.

   Switch to a lighter, faster model like Claude Haiku to optimize for speed and cost-efficiency for this straightforward task.

   Haiku is specifically designed for high-volume, lower-complexity tasks where speed and cost are prioritized over deep reasoning.

   • ✗ Instruct Claude Opus to generate shorter responses to reduce the number of output tokens consumed per entry.

     Reducing output length does not address the high per-token cost of the Opus model itself for a task that doesn't require its full reasoning capabilities.

   • ✗ Process entries in larger batches of 50 within a single prompt to save on repeated system instruction overhead.

     While batching can help, using an over-powered model for simple classification remains the primary inefficiency in the cost-to-performance ratio.

   • ✗ Request a project-specific increase in the token limit to allow Opus to process more entries simultaneously.

     Increasing limits does not reduce the cost; it merely allows for higher total expenditure without solving the optimization problem.

4. 4 A manager notices that a newly designed reporting workflow, while accurate, is taking longer than the previous manual process because Claude often generates vague, generic paragraphs. How should the Associate troubleshoot this prompt performance issue?

   Look at the components of 'structured prompting' used to ground a model's output.

   Rewrite the prompt to explicitly state the target audience, specific purpose, required metrics, and expected format.

   Vague outputs are often the result of underspecified prompts; adding structure and constraints helps Claude align with specific business needs.

   • ✗ Ask Claude to 'be more creative' and 'provide more detail' to increase the word count of the reports.

     Subjective instructions like 'be creative' often lead to more 'filler' content rather than the specific, actionable data required for business reporting.

   • ✗ Increase the temperature setting of the model to encourage more diverse and comprehensive vocabulary.

     Temperature affects randomness, which can lead to lower consistency and potential hallucinations in a reporting context.

   • ✗ Upload a larger volume of general business documents to the Project's knowledge base to provide more context.

     Adding unrelated documents can distract the model; the fix for vague output is typically better instruction, not just 'more' unguided data.

5. 5 When validating a complex legal comparison generated by Claude, the model provides a confidence score of 95% and includes specific regulation numbers. What is the correct troubleshooting step before sharing this with stakeholders?

   Recall the primary 'anti-pattern' mentioned in the source material regarding model self-assessment.

   Verify every factual claim and regulation number against primary sources, as self-reported confidence is not a reliable indicator of accuracy.

   Models can produce 'confident' hallucinations; the Diligence Step requires human verification against authoritative sources regardless of model certainty.

   • ✗ Trust the output because the inclusion of specific numbers and a high confidence score indicates the model has 'checked its work'.

     A common misconception is that a model's self-evaluation or its use of specific citations is a signal of truth; both can be fabricated.

   • ✗ Ask Claude in a new chat to verify the accuracy of the regulation numbers it provided in the previous chat.

     Using a model to verify its own potential hallucinations is recursive and unreliable; external verification is the standard for accuracy.

   • ✗ Rerun the prompt three times and only share the output if all three versions are identical.

     Consistent output is not the same as accurate output; a model can consistently generate the same hallucination if the prompt or context is flawed.

6. 6 An Associate is attempting to fix a workflow where Claude repeatedly fails to extract data from a specific, poorly formatted PDF. Which fix remains within the CCAO-F (Associate) scope?

   Identify the boundary between 'using features' and 'engineering systems'.

   Manually clean or reformat the source material into a more structured format before uploading it back to the Project.

   Improving input quality and formatting is a core Associate task for troubleshooting poor data extraction.

   • ✗ Develop a custom Python script using the Claude API to preprocess the PDF's raw binary data.

     Coding and direct API engineering are technical tasks that exceed the non-developer Associate role.

   • ✗ Configure an Agentic Loop to recursively scan the document until the failure rate drops below 1%.

     Designing agentic loops and autonomous multi-agent systems is a task for a Claude Architect, not an Associate.

   • ✗ Modify the underlying Model Context Protocol (MCP) server to better handle varied file types.

     MCP server development and modification require technical engineering skills reserved for the Developer or Architect tracks.

7. 7 A team's workflow involves a manual, six-step prompt sequence every Monday to generate weekly updates. The Associate identifies this is inefficient. What is the best optimization to standardize this process?

   Look for the Claude feature designed specifically for 'recurring workflows' and 'team consistency'.

   Establish a Claude Project with standing instructions and reusable templates in the knowledge base for this specific recurring task.

   Projects allow for persistent configurations that eliminate the need for manual, repetitive prompt sequences and ensure consistent formatting.

   • ✗ Write out the six-step sequence in a shared Word document so team members can copy-paste it into new chats each week.

     Manual copy-pasting is prone to error and doesn't leverage Claude's built-in features for workflow automation and consistency.

   • ✗ Instruct Claude to 'remember' the six steps for the next time the team needs them.

     Claude's memory across distinct, non-project chats is limited; relying on vague 'remembering' leads to inconsistent outcomes.

   • ✗ Assign one team member to be the 'prompt owner' who is the only person allowed to interact with Claude for that task.

     This creates a bottleneck rather than a scalable workflow optimization and doesn't improve the actual AI interaction process.

8. 8 An organization is evaluating its Claude-supported process for drafting client contracts. The evaluation report shows an 'Aggregate Accuracy' of 98%. Why might an Associate still recommend further troubleshooting?

   Think about the limitations of broad statistics versus detailed, segment-based analysis.

   Aggregate metrics can hide specific, critical failure modes in certain document types or fields that require human oversight.

   Optimization requires segmented accuracy (e.g., by document type) because a 2% failure rate in high-risk fields is unacceptable for production use.

   • ✗ A 98% accuracy rate indicates that Claude is likely over-fitting the data and losing its creative flexibility.

     Accuracy in contract drafting is a positive goal; troubleshooting is needed to address the *remaining* errors, not to reduce accuracy for 'creativity'.

   • ✗ The Associate should never trust any metric over 95% without re-running the evaluation on the Claude Opus model.

     This is a procedural guess; the real issue with aggregate metrics is their lack of granularity in identifying high-risk failure patterns.

   • ✗ Organizational AI policy requires 100% accuracy for all tasks before they can be considered 'optimized'.

     100% accuracy is rarely achievable with LLMs; the goal is managed risk, reliability, and knowing where failure occurs.

9. 9 During the rollout of a redesigned research workflow, stakeholders report confusion and gaps in the final outputs. What is the best troubleshooting step to resolve these 'rollout gaps' before a wider deployment?

   Think about a structured way to 'test' a process on a smaller scale.

   Run a limited pilot phase with a small user group to surface practical challenges and refine the workflow checkpoints.

   Pilot testing is a strategic best practice for identifying and resolving gaps in a new AI workflow before broad implementation.

   • ✗ Immediately revert to the manual process and wait for the next Claude model update.

     This is a retreat rather than troubleshooting; practical gaps are usually solved by workflow refinement and iteration.

   • ✗ Increase the prompt complexity to cover every possible edge case reported by the stakeholders.

     Overly complex prompts can lead to 'instruction fatigue' and reduced model performance; iterative refinement is more effective.

   • ✗ Tell stakeholders to simply 'try the prompts again' as LLM output is naturally variable.

     Dismissing feedback ignores the need for workflow optimization and fails to address the root causes of the confusion.

10. 10 A practitioner is troubleshooting a situation where Claude's output violates a strict company privacy policy regarding customer PII. What is the default 'first question' an Associate should ask to fix the workflow?

    Consider the safest method for processing sensitive information without exposing it.

    Can I anonymize or redact this data and still achieve the required business objective?

    Anonymization is the default first step for handling sensitive data safely while maintaining the utility of the AI analysis.

    • ✗ Which Claude model has the highest built-in security for handling PII?

      Security is handled at the platform and policy level; troubleshooting starts with data-handling practices, not model selection.

    • ✗ Can I instruct Claude in the prompt to 'ignore' any PII it encounters?

      Relying on prompt-based instructions for safety or compliance enforcement is a known anti-pattern with a non-zero failure rate.

    • ✗ How can I bypass the filter to ensure my analysis is completed on time?

      Bypassing safety filters violates the 'Governance, Risk, and Responsible Use' principles of the certification.

11. 11 Claude consistently fails to follow a specific output format (e.g., Markdown tables) even when told to do so in the prompt. Which iterative correction is most likely to resolve this?

    Think of the technique that uses 'demonstrations' to guide model behavior.

    Incorporate 2�4 'few-shot' examples in the prompt that show the model both the expected input and the correctly formatted output.

    Few-shot prompting is highly impactful for ensuring format consistency and helping the model understand complex structural requirements.

    • ✗ Use all-caps for the formatting instructions to ensure the model 'pays attention'.

      All-caps is a weak prompting technique compared to structural improvements like few-shot examples or clear task decomposition.

    • ✗ Provide 10�15 simple examples to overwhelm the model with the correct pattern.

      Source material suggests 2�4 *targeted* examples are better than many easy ones, as too many examples can distract the model.

    • ✗ Switch to a different Claude feature like Artifacts, as standard Chat cannot produce Markdown tables.

      Claude Chat is fully capable of producing Markdown; the issue is one of prompt instruction and example guidance, not feature limitation.

12. 12 An Associate is designing a workflow for identifying potential leads from industry news. Claude keeps including companies that are already existing clients. What is the most effective optimization?

    Identify the best way to provide 'negative constraints' using external data.

    Upload a list of existing clients to a Claude Project's knowledge base and update instructions to exclude them from lead generation.

    Using a Project's knowledge base to provide specific exclusion lists is a reliable way to optimize results for specific business contexts.

    • ✗ Tell Claude to 'do its best' to remember who the current clients are based on previous conversations.

      This is an unreliable approach that fails to provide the model with the necessary factual context to succeed.

    • ✗ Increase the 'penalize' parameter in the API settings to discourage repeating known company names.

      The CCAO-F exam does not cover low-level API parameter tuning, and 'penalizing' is not a standard way to manage factual exclusion lists.

    • ✗ Rely on the manual reviewer to filter out existing clients after Claude has finished the task.

      While a reviewer is part of the loop, this does not optimize the AI's performance and wastes human effort on a fixable AI error.

13. 13 A marketing team uses a single long conversation thread for all their brand brainstorming. Claude has started to produce generic ideas and is ignoring recent feedback. What is the diagnostic fix?

    This relates to the technical limitation of the 'context window' in prolonged interactions.

    The conversation history is too large; summarize the best ideas and current brand guidelines, then start a fresh chat.

    Long-running conversations suffer from context decay; summarizing and restarting refreshes the model's focus on the most important information.

    • ✗ The model has become 'bored' with the topic and needs a completely unrelated prompt to reset its internal state.

      Models do not have human emotions like boredom; performance issues in long chats are purely technical results of context window limits.

    • ✗ The team needs to switch from Claude Haiku to Claude Sonnet to handle the 'creative depth' of the brainstorming.

      While model choice matters, the primary issue described is the accumulation of history in a single thread, which affects all models.

    • ✗ The team should use 'Research Mode' instead of standard Chat for brainstorming activities.

      Research Mode is for gathering and organizing external info; brainstorming is a standard Chat or Artifacts activity.

14. 14 A business professional identifies that a Claude-supported workflow for financial forecasting has a high risk of making incorrect recommendations that could lead to financial loss. What is the most important 'Governance' step in this troubleshooting process?

    Determine the necessary safety 'checkpoint' for consequential business decisions.

    Incorporate a mandatory 'human-in-the-loop' expert review gate before any recommendation is finalized or acted upon.

    High-impact or consequential decisions must involve human expertise to mitigate the risks of AI inaccuracies.

    • ✗ Instruct Claude to double-check its math and provide a 'confidence rating' for its forecasts.

      Confidence ratings are not reliable signals of accuracy and do not substitute for actual human accountability in high-risk scenarios.

    • ✗ Switch the workflow to only use Claude Opus as it is the most capable model and therefore 'safe' for financial work.

      No model is 100% accurate; capability does not remove the need for governance standards and human oversight.

    • ✗ Ensure the forecasts are only shared internally to avoid external regulatory scrutiny.

      Internal use does not remove the financial risk or the need for responsible-use practices and accurate results.

15. 15 When an Associate is analyzing why a complex multi-step request resulted in a partial failure, they notice Claude successfully completed step 1 but became confused by step 2 and 3. What is the recommended optimization?

    Consider the strategy of 'breaking down' a large problem into manageable pieces.

    Apply task decomposition by breaking the request into three separate, sequential prompts to ensure accuracy at each stage.

    Breaking complex tasks into smaller, logical steps (decomposition) is a core skill for improving reliability in multi-step AI interactions.

    • ✗ Repeat steps 2 and 3 in the prompt using bold and underlined text to emphasize their importance.

      Formatting emphasis is less effective than structural changes like decomposition when a model is failing to follow a multi-step sequence.

    • ✗ Provide more background information on the project in a single, massive initial prompt.

      Oversaturating the initial prompt with background can actually increase model confusion; step-by-step guidance is the optimized approach.

    • ✗ Switch to a model with a lower temperature to ensure it doesn't 'wander off' during the long prompt.

      While temperature affects variability, the root cause of confusion in multi-step prompts is usually the lack of task decomposition.

16. 16 An organization wants to use Claude to summarize internal meeting transcripts. Troubleshooting reveals that Claude often ignores 'off-the-record' comments even when instructed otherwise. What diagnostic question helps resolve this?

    Consider the hierarchy of instructions within the Claude platform.

    Is the prompt instruction to include 'off-the-record' comments conflicting with a company-wide 'system instruction' or project rule?

    Troubleshooting requires checking for conflicting instructions between the immediate prompt and persistent configuration settings (System/Project instructions).

    • ✗ Has the model's safety filter determined that 'off-the-record' content is inherently dangerous?

      Safety filters trigger refusals or redactions for specific harms (e.g., violence), not for business concepts like 'off-the-record' meeting notes.

    • ✗ Should the Associate use Claude Code instead of Chat to handle meeting transcripts more effectively?

      Claude Code is a tool for developers working on codebases, not for summarizing business meeting transcripts.

    • ✗ Will using the Claude Artifacts feature force the model to include the missing content?

      Artifacts are a way to *present* output, not a mechanism to bypass instructional conflicts or force the inclusion of ignored data.

17. 17 A professional is using Claude to draft responses to client emails. Claude occasionally uses an overly formal tone that doesn't match the company's brand. Which troubleshooting fix is best for correcting brand alignment?

    Identify the technique that uses 'demonstrations' to refine qualitative output like 'tone'.

    Provide a few-shot prompt that includes 3�5 examples of correctly branded email responses to show the model the desired tone and style.

    Few-shot examples are the most effective way to align model output with a specific brand voice or stylistic preference.

    • ✗ Tell Claude to 'write like a friendly human' at the end of every prompt.

      Subjective instructions like 'friendly human' are interpreted differently by the model and lack the precision of brand-specific examples.

    • ✗ Only use Claude for drafting the factual content, and have a human rewrite every sentence for tone.

      While human review is necessary, this is an inefficient process that fails to optimize the AI's ability to learn the brand voice through examples.

    • ✗ Switch to Claude Haiku, as smaller models are generally more 'casual' than larger models.

      Model size does not correlate with 'casualness' or 'formality'; tone is a result of prompting and provided context.

18. 18 An Associate is tasked with finding a way to reduce 'hallucinations' in a Claude-supported research workflow. Which optimization approach is grounded in CCAO-F principles?

    Focus on the source of the 'data' Claude uses to answer a question.

    Ensure Claude only works with primary source documents provided in its knowledge base rather than relying on its internal general knowledge.

    Limiting a model's scope to provided factual context is a key technique for reducing its tendency to fabricate information.

    • ✗ Ask Claude to provide a detailed 'logic chain' for its thoughts, as this automatically prevents it from making up facts.

      A 'logic chain' or 'Chain of Thought' can help reasoning, but the model can still confidently use fabricated 'facts' within that chain.

    • ✗ Configure the Claude Agent SDK to automatically verify every sentence against Google Search.

      Developing systems with the Claude Agent SDK and search integrations is a developer-level technical task, not an Associate-level configuration.

    • ✗ Only use Claude during business hours when its 'contextual awareness' is highest.

      The time of day has no effect on an LLM's technical performance or its 'awareness'.

19. 19 Stakeholders are complaining that Claude's research summaries are 'too long to be useful'. How should the Associate optimize the workflow for executive efficiency?

    Select the feature best suited for 'curating' and 'layering' information for an audience.

    Use the Claude Artifacts feature to present the findings as a structured, interactive briefing with key points at the top.

    Artifacts allow for the creation of separate, structured documents that are easier for stakeholders to digest than long conversational replies.

    • ✗ Instruct Claude to use 'shorter words' and 'fewer sentences' regardless of the complexity of the research.

      Arbitrarily limiting word counts can lead to the loss of critical information; structural curation is a better optimization.

    • ✗ Direct stakeholders to read the full chat history to ensure they don't miss any context.

      This is the opposite of an executive-level optimization and increases the burden on the stakeholders.

    • ✗ Switch the workflow to Claude Haiku, as it is incapable of generating long responses.

      Haiku is fully capable of long responses; the issue is the formatting and presentation of the information, not model capacity.

20. 20 A company is using Claude for internal employee support. Employees report that Claude is giving incorrect advice on the company's travel policy. What is the first troubleshooting step an Associate should take?

    Think about where Claude 'gets' its information for company-specific questions.

    Check if the most recent version of the travel policy PDF is uploaded to the Project's knowledge base.

    Verifying that the model has access to current, accurate reference material is the primary first step for diagnosing incorrect factual advice.

    • ✗ Tell employees to 'be more specific' when asking questions about travel.

      If the model doesn't have the correct source information, specificity from the user will not fix the underlying factual error.

    • ✗ Disable the travel support feature until a newer, more accurate Claude model is released.

      This is a failure to troubleshoot; the issue is almost certainly the provided context, not the underlying model architecture.

    • ✗ Instruct Claude to apologize and admit it doesn't know the policy every time it is asked.

      This reduces the utility of the AI instead of fixing the knowledge gap through proper configuration.