25 Questions — Domain 5 : Model Selection and Optimization

Expand any question to reveal the correct answer and explanation.

1. 1 A developer is using a high-frequency application where 80% of the input prompt consists of a static $50,000$-token documentation set. If standard input costs $X$ and cached input costs $0.1X$, which calculation represents the total input cost for a single request using prompt caching?

   Consider the cost formula where only specific portions of the prompt benefit from the reduced 'cached' rate.

   $(0.2 \times 50,000 \times X) + (0.8 \times 50,000 \times 0.1X)$

   This accurately reflects the breakdown where the $20\%$ uncached portion is charged at the full rate and the $80\%$ cached portion is charged at the reduced rate.

   • ✗ $(50,000 \times X) \times 0.1$

     This mistakenly applies the discount to the entire prompt rather than just the portion designated as a cache checkpoint.

   • ✗ $(0.8 \times 50,000 \times X) + (0.2 \times 50,000 \times 0.1X)$

     This incorrectly applies the full rate to the majority of the prompt which is intended to be cached.

   • ✗ $(50,000 \times 0.9X)$

     This represents a flat $10\%$ discount across the total tokens, which does not align with how individual cache checkpoints are billed.

2. 2 In a complex multi-agent system, when should a developer transition a task from Claude 3.5 Sonnet to Claude 4 Opus according to Anthropic's model selection tradeoffs?

   Think about which model is designated for 'premium' reasoning and high logical capacity.

   When the task requires intricate multi-turn system planning and complex code refactoring.

   Opus is specifically positioned for premium reasoning and complex, multi-layered logical capacity.

   • ✗ When the task involves high-frequency text classification or simple routing steps.

     These tasks are better suited for Haiku due to its low latency and cost-efficiency.

   • ✗ When the primary requirement is the lowest possible latency regardless of logical complexity.

     Latency-first requirements usually favor smaller, faster models like Haiku or Sonnet in 'fast mode'.

   • ✗ When the application needs to maximize throughput for a standard tool-use loop.

     Sonnet is the default production tier for balanced reasoning and high throughput in standard loops.

3. 3 A project requires processing $100,000$ documents for an analytics report that is not time-sensitive. Which API feature provides the most significant cost optimization?

   Look for the mechanism specifically designed for 'non-urgent' or 'overnight' workloads.

   Message Batches API.

   This API is designed for latency-tolerant, high-volume workloads and offers a $50\%$ cost reduction within a 24-hour window.

   • ✗ Realtime Messages API with prompt caching.

     While caching helps, it does not match the deep discount provided by asynchronous batch processing for large volumes.

   • ✗ Claude 3.5 Sonnet in 'fast mode'.

     Fast mode optimizes for latency, but typically does not provide the same per-token cost savings as batch processing.

   • ✗ Implementing aggressive context window compaction.

     Compaction reduces token count but does not change the fundamental billing rate of the API calls themselves.

4. 4 When configuring 'Extended Thinking' for a model, how should a developer determine the 'thinking' token budget?

   Thinking tokens occur internally before the final visible response is generated.

   The budget represents a subset of the total output tokens allocated for internal reasoning before a response.

   Thinking tokens are used for internal reasoning and must be balanced against the desired output length and latency requirements.

   • ✗ The budget should always be set to the maximum context window to ensure the highest quality.

     Setting an excessively high budget can lead to unnecessary costs and increased latency without guaranteed quality gains.

   • ✗ Thinking tokens are free and do not count toward the total token limit of the request.

     All tokens generated, including thinking tokens, typically count toward usage limits and billing.

   • ✗ The budget is automatically determined by the model based on the complexity of the prompt.

     While the model uses what it needs, the developer must define the upper boundary (budget) for the effort level.

5. 5 Which scenario would most likely benefit from using Claude 3.5 Haiku as a 'router' model?

   Consider the model tier that excels at low latency and high-speed classification.

   Classifying an incoming user request to determine if it should be handled by a specific tool or a larger model.

   Haiku is optimized for speed and cost in high-frequency classification and routing tasks.

   • ✗ Generating a detailed architecture diagram from a technical specification.

     This requires high-level reasoning and multi-modal understanding, better suited for Sonnet or Opus.

   • ✗ Evaluating the semantic accuracy of a complex legal contract.

     Legal reasoning requires higher precision and logic than a small model typically provides.

   • ✗ Conducting deep research across $20$ different technical papers.

     Research tasks involve heavy synthesis and context management which favor larger models.

6. 6 What is the primary technical limitation of prompt caching that a developer must manage when designing a stateful application?

   Think about the duration for which a cache remains active without being accessed.

   The cache has a $5$-minute Time-To-Live (TTL) and is evicted if not refreshed by a subsequent request.

   Developers must design their systems to frequently reuse the same cache checkpoints to prevent expiration and re-initialization costs.

   • ✗ Cached tokens cannot be used in conjunction with tool use.

     Prompt caching is compatible with tool use and is often used to cache tool definitions.

   • ✗ Each request can only contain a single cache checkpoint.

     Anthropic allows multiple cache checkpoints (e.g., up to $4$) within a single prompt.

   • ✗ Caching is only available for the Claude 3 Opus model.

     Prompt caching is supported across multiple models in the Claude family, including Sonnet and Haiku.

7. 7 When optimizing token usage, a developer notices 'Context Drift.' What is the most effective architectural fix for this in a multi-step agentic workflow?

   Consider how modularity can prevent information overload in a model's 'memory'.

   Switching from a single long-context agent to a hierarchy of subagents with isolated context windows.

   Isolating subtasks into subagents prevents the 'bloat' of unnecessary history from affecting specific logical steps.

   • ✗ Increasing the total context window to $200,000$ tokens.

     Larger windows can actually exacerbate drift by allowing more irrelevant or conflicting information to accumulate.

   • ✗ Setting the temperature to $0$ to ensure deterministic output.

     Temperature affects token selection probability but does not address the accumulation of irrelevant context.

   • ✗ Using Haiku instead of Sonnet.

     Model choice affects reasoning quality but doesn't solve the structural problem of context management.

8. 8 In the context of LLM mechanics, what does the 'stop_reason' field primarily help a developer optimize?

   This field tells the developer if the model finished naturally or was cut off by a constraint.

   Token budgeting and error recovery by identifying why a generation finished.

   Identifying if a model stopped due to reaching a limit ('max_tokens') versus finishing its thought ('end_turn') is critical for deciding whether to retry or truncate.

   • ✗ Prompt cache hit rates.

     Cache hit rates are determined by input structure, not the reason the model stopped generating.

   • ✗ The frequency of PII detection in the output.

     PII detection is usually a safety filter or post-processing step, not a generation stop reason.

   • ✗ Latency in multi-modal vision tasks.

     While it indicates the end of the process, it does not provide a direct lever for optimizing visual processing speed.

9. 9 How does 'Adaptive Thinking' differ from standard 'Extended Thinking' in Claude 4 models?

   Think about the word 'adaptive'�it suggests a model that changes its effort based on the problem.

   It allows the model to decide autonomously whether to use extra reasoning tokens based on task difficulty.

   Adaptive thinking enables the model to bypass the heavy reasoning overhead for simple tasks while engaging it for complex ones.

   • ✗ It forces the model to use the maximum possible token budget for every request.

     This would be the opposite of adaptive behavior and would lead to high latency and cost.

   • ✗ It is a feature that only works when the model is in 'headless' mode.

     Thinking modes are independent of the interface (UI vs. Headless CLI).

   • ✗ It primarily adjusts the temperature of the model dynamically during generation.

     Adaptive thinking focuses on the reasoning process, not the sampling randomness parameters.

10. 10 A developer needs to implement a 'least-privilege' approach for a tool that handles financial transactions. Which design pattern is most secure?

    Identify the difference between 'requesting' a behavior and 'enforcing' it in code.

    Implementing programmatic 'pre-call' hooks that validate the user's session and permissions before the API call is executed.

    Hard-coded programmatic enforcement (gates) ensures $100\%$ compliance with security policies regardless of model behavior.

    • ✗ Adding a strong warning in the system prompt to only use the tool for authorized users.

      Prompt instructions are non-deterministic and can be bypassed via prompt injection.

    • ✗ Using a larger model like Opus to ensure it follows instructions more reliably.

      Larger models can still be susceptible to injection; security should never rely solely on model compliance.

    • ✗ Restricting the tool to only be used in the 'Batch API'.

      The Batch API changes the timing of execution but does not provide inherent authentication or authorization controls.

11. 11 What is the impact of setting a high 'Temperature' ($T > 0.7$) on token budgeting and cost modeling?

    Think about how randomness affects the consistency of the results.

    It increases the non-determinism of the output, making cost and performance harder to predict across multiple runs.

    Variation in response length and content due to high temperature makes precise token budgeting and latency modeling difficult.

    • ✗ It consistently reduces the total number of tokens generated.

      Higher temperature increases randomness but does not necessarily decrease token count; it may even lead to more verbose outputs.

    • ✗ It disables the ability to use prompt caching.

      Sampling parameters like temperature do not affect the functionality of the prompt cache.

    • ✗ It automatically switches the model to Haiku for faster processing.

      Temperature is a parameter that applies to the selected model, it doesn't trigger a model tier change.

12. 12 When using prompt caching, where should the 'checkpoint' ideally be placed for maximum efficiency in a chatbot that uses a large system prompt and several examples?

    The goal is to cache the parts of the prompt that stay the same throughout the entire session.

    After the static system instructions and few-shot examples, but before the dynamic conversation history.

    This allows the static, expensive part of the prompt to be reused across all turns in the session.

    • ✗ Immediately after the user's latest message.

      Placing it after the user message makes the cache volatile, as it would change with every turn.

    • ✗ At the very beginning of the prompt, before the system role is defined.

      A checkpoint at the start would cache nothing of value, as there is no content before it.

    • ✗ Randomly throughout the conversation history to capture different state changes.

      Random placement is inefficient and would likely lead to frequent cache misses.

13. 13 Which metric is most critical when evaluating whether to use 'Streaming' for a Claude-powered application?

    Consider the user's perception of how quickly the AI 'starts' talking.

    Time to First Token (TTFT).

    Streaming is primarily used to improve the perceived latency (TTFT), allowing the user to see the response as it is generated.

    • ✗ Total Token Count.

      Total tokens remain the same regardless of whether the response is streamed or sent as a single block.

    • ✗ Prompt Cache Hit Rate.

      Caching is an input-side optimization; streaming is an output-side delivery mechanism.

    • ✗ Model Logical Capacity.

      All models support streaming; it does not change the model's ability to solve a problem.

14. 14 A developer is building a 'Code Modernization' agent. Why might they choose Opus 4 over Sonnet 3.5 despite the higher cost and latency?

    Identify the model tier Anthropic suggests for the most logically demanding tasks.

    Opus 4 is designated for 'intricate reasoning' and 'multi-turn system planning' required for deep refactoring.

    Complex code modernization involves high levels of logic and planning that match the intended use case for Opus.

    • ✗ Opus 4 has a much smaller context window and is easier to manage.

      Opus typically has a large window, not a smaller one; context management is similar across tiers.

    • ✗ Opus 4 offers a $50\%$ discount on all input tokens.

      Pricing follows a tiered structure where Opus is the most expensive, not the discounted tier.

    • ✗ Opus 4 is the only model that supports MCP servers.

      All recent Claude models support the Model Context Protocol.

15. 15 In token budgeting, what is a 'Stop Token'?

    Think of this as a 'red light' that the model sees and stops talking.

    A specific sequence that, when generated by the model, signals the end of the response and stops generation.

    Stop sequences allow developers to prevent a model from rambling or generating unwanted sections by cutting off output early.

    • ✗ A token that forces the model to ignore the rest of the system prompt.

      Model execution usually processes the entire prompt; there is no 'ignore' token for the system prompt.

    • ✗ A token used to flag a request for human review.

      Human review triggers are usually based on content evaluation or log analysis, not a specific LLM token.

    • ✗ The token that marks the start of a prompt cache checkpoint.

      Checkpoints are defined in the API request structure, not by special tokens in the text.

16. 16 What is the primary benefit of using a 'Context Window Compaction' strategy?

    Think about 'trimming the fat' from a long chat history to save money and time.

    It reduces cost and latency by removing irrelevant or redundant information from the conversation history.

    By keeping only the essential information, developers stay within token limits and reduce the processing time of the prompt.

    • ✗ It increases the model's logical reasoning capability.

      Compaction manages memory volume but doesn't upgrade the model's fundamental brainpower.

    • ✗ It enables the model to see into the future of the conversation.

      LLMs generate tokens based on past context; they have no predictive sight into future turns.

    • ✗ It replaces the need for a system prompt.

      System prompts provide the 'how-to' instructions, whereas compaction manages the 'data' history.

17. 17 If a developer encounters a 'Rate Limit' error during a large-scale data extraction project, which adjustment is most appropriate for a high-volume, non-urgent task?

    Consider the API designed for 'high-volume' and 'asynchronous' work.

    Switch to the Batch API to utilize separate, larger throughput limits.

    The Batch API has higher rate limits and handles high-volume tasks more gracefully than the realtime Messages API.

    • ✗ Reduce the temperature to $0$.

      Temperature affects output variety, not the speed or quantity of requests permitted by the server.

    • ✗ Increase the 'max_tokens' parameter to allow more data per request.

      Increasing 'max_tokens' might actually worsen rate limit issues by increasing the token-per-minute load.

    • ✗ Switch from Sonnet to Opus.

      Moving to a larger model usually results in lower rate limits and higher costs.

18. 18 Which sampling parameter is primarily used to control the 'diversity' of the model's response by limiting the pool of potential next tokens to a specific cumulative probability?

    This parameter is often called 'Nucleus Sampling' and deals with 'cumulative' percentages.

    Top P (Nucleus Sampling)

    Top P limits the next token selection to the smallest set of tokens whose cumulative probability exceeds the threshold $P$.

    • ✗ Temperature

      Temperature scales the probabilities of all tokens but does not use a cumulative cutoff.

    • ✗ Max Tokens

      Max tokens controls the length of the response, not the statistical selection of individual tokens.

    • ✗ Stop Sequences

      Stop sequences terminate generation when a specific string appears.

19. 19 A developer wants to use Claude to extract data from $500$ different PDF invoice formats. Which technique is most effective for ensuring consistent structured output across all formats?

    Think about how to 'force' the model to use a specific technical format for its response.

    Defining a strict JSON schema in the tool definition and using 'tool_choice' to force the model to use that tool.

    Forcing the model to respond through a specific tool ensures it follows the provided JSON structure for every extraction.

    • ✗ Using a $100$-shot prompt with one example of every possible format.

      This would likely exceed the context window or lead to confusion; a few targeted examples are generally more effective.

    • ✗ Asking the model to 'be very careful' in the system prompt.

      Vague instructions do not provide the programmatic structure needed for reliable data extraction.

    • ✗ Processing the PDFs through Haiku first to categorize them.

      Categorization is a useful step but does not inherently guarantee the quality of the final structured output.

20. 20 When calculating the 'Total Cost' of a Claude application, why must 'Thinking Tokens' be modeled differently than standard 'Output Tokens'?

    Consider that these tokens are generated but are not visible to the end user.

    They represent a significant latency overhead that doesn't contribute to the visible response length.

    Because thinking tokens are 'internal', they consume time and budget without appearing in the final word count, making efficiency modeling tricky.

    • ✗ Thinking tokens are $10$ times more expensive than output tokens.

      Thinking tokens are usually priced at the same rate as standard output tokens.

    • ✗ Thinking tokens cannot be cached.

      Caching applies to inputs, while thinking tokens are part of the generation/output process.

    • ✗ They are only generated when the temperature is set to $0$.

      Thinking tokens are a feature of specific 'extended thinking' models, not a result of temperature settings.

21. 21 In the cost equation $\text{Total Cost} = (U \times R_u) + (C \times R_c) + (O \times R_o)$, what does the variable $R_c$ represent?

    Look at the subscript 'c'�what optimization does it likely refer to?

    The rate for Cached Input Tokens.

    In the standard pricing model, $R_c$ represents the significantly lower cost assigned to tokens retrieved from a cache checkpoint.

    • ✗ The rate for Output Tokens.

      The rate for Output tokens is represented by $R_o$.

    • ✗ The rate for Uncached Input Tokens.

      Uncached tokens are charged at the full rate $R_u$.

    • ✗ The number of tokens in the system prompt.

      The letter $R$ usually denotes a 'rate' or price, while the tokens themselves are represented by other variables.

22. 22 A developer is building a translation app. They want to ensure that the translation always follows a specific tone. Which prompting method is most reliable for this?

    What is the term for providing a 'few' examples to show the model how to behave?

    Multi-shot prompting with $3$ to $5$ examples of 'Input' and 'Target Tone Output'.

    Providing specific examples (few-shot) is the industry standard for teaching a model to follow a specific style or tone.

    • ✗ Zero-shot prompting.

      Zero-shot provides no examples and relies on the model's base training, which may not capture a specific nuance.

    • ✗ Using the Batch API.

      API choice affects timing and cost, not the stylistic behavior of the model.

    • ✗ Reducing the context window.

      Reducing history might help focus the model but doesn't define the tone as effectively as examples do.

23. 23 Which of the following describes 'Adaptive Thinking' model behavior?

    Focus on the word 'dynamically' and how it relates to the 'effort' the model puts in.

    The model dynamically allocates thinking tokens based on its internal assessment of prompt complexity.

    Adaptive thinking allows the model to scale its internal processing power to match the difficulty of the task.

    • ✗ The model uses a fixed $4,000$ tokens of reasoning for every request.

      Fixed reasoning is not 'adaptive'; it's a standard static setting.

    • ✗ The model changes its sampling temperature based on the user's sentiment.

      This would be a custom implementation and is not an inherent feature of 'Adaptive Thinking' modes.

    • ✗ The model switches between Sonnet and Opus automatically.

      Tier switching is usually handled by an application-side 'router', not the model's internal thinking process.

24. 24 A team is worried about 'Prompt Injection' in their customer-facing tool. Which architectural decision provides the best defense-in-depth?

    Think about limiting the 'power' the model has rather than trying to make it 'smarter' at defense.

    Apply a strict 'least-privilege' principle to the tools the model can access, ensuring it cannot perform destructive actions even if injected.

    Enforcing security at the tool-execution level (programmatically) ensures that the damage of a successful injection is limited.

    • ✗ Ask the model to ignore any instructions found in the user input.

      This is a prompt-level instruction which is precisely what injection attacks are designed to circumvent.

    • ✗ Only use the Haiku model, as it is too small to be injected.

      All LLMs are susceptible to prompt injection regardless of their size.

    • ✗ Set the temperature to $2.0$.

      Extreme temperatures lead to gibberish and do not provide security against structured injection attacks.

25. 25 When modeling cost for a RAG (Retrieval-Augmented Generation) system, why is 'Prompt Caching' particularly valuable?

    Think about 'shared' information that many different users might be asking about at the same time.

    It allows the system to cache the 'Retrieved Context' which is often identical across many user queries about the same topic.

    If multiple users ask about the same documentation, caching that documentation in the prompt significantly reduces per-request costs.

    • ✗ It eliminates the need for an external vector database.

      RAG still needs a database to find the relevant documents; caching just handles the LLM input part.

    • ✗ It makes the model's output $100\%$ deterministic.

      Caching is an input optimization and does not change the probabilistic nature of LLM generation.

    • ✗ It increases the maximum context window of the Haiku model.

      Context windows are fixed model limits; caching only changes how you pay for the tokens inside that window.