20 Questions — Domain 4 : Prompt Engineering & Structured Output

Expand any question to reveal the correct answer and explanation.

1. 1 An architect is designing a high-volume document extraction system with a strict 30-hour Service Level Agreement (SLA). The Message Batches API, which takes up to 24 hours to process, will be used. What is the longest submission interval that guarantees compliance while maximizing cost efficiency?

   Calculate the difference between the total SLA time and the maximum processing time of the Batch API.

   Every 6 hours

   Since the max processing time is 24 hours, adding a 6-hour interval results in a worst-case turnaround of 30 hours, meeting the SLA with the lowest submission frequency.

   • ✗ Every 4 hours

     While this meets the SLA, it increases orchestration overhead by submitting more batches than the mathematical requirement for maximum efficiency.

   • ✗ Every 12 hours

     This would result in a worst-case turnaround of 36 hours ($12 + 24$), which exceeds the 30-hour SLA limit.

   • ✗ Every 2 hours

     This is a safe production choice but fails the requirement for maximizing cost efficiency by over-submitting batches.

2. 2 When designing a JSON schema for a data extraction tool, how should an architect handle a field that may not be present in all source documents to prevent model hallucinations?

   Consider how schema definitions impact the model's tendency to satisfy mandatory requirements.

   Mark the field as nullable or optional in the schema

   Designing schema fields as optional or nullable allows the model to return null when information is missing, effectively preventing data fabrication.

   • ✗ Instructional constraints in the system prompt

     Prompt instructions are probabilistic and have a higher failure rate compared to deterministic schema constraints.

   • ✗ Define the field as required but provide an 'N/A' default value

     Making the field required often forces the model to fabricate data if it cannot find the information in the source.

   • ✗ Use a few-shot example that shows a placeholder string

     While few-shot examples help, they do not provide the same structural enforcement as the tool's JSON schema definition.

3. 3 A production pipeline requires extracting technical categories that frequently change. Which schema design pattern is recommended for extensible categorization?

   Think about a pattern that balances strict validation with the need to capture unexpected data.

   Enum fields including an 'other' value paired with a detail string

   Using an 'other' option within an Enum combined with a detailed description field allows for both standardization and extensibility.

   • ✗ A strictly bound Enum containing all known values

     A strict Enum will cause the model to fail or hallucinate when it encounters a valid category not yet present in the code.

   • ✗ A free-text string field with no Enum constraints

     This lacks structural control and leads to inconsistent naming conventions that are difficult to parse downstream.

   • ✗ A multi-pass architecture that generates a new Enum per document

     This adds significant latency and cost without providing a stable structure for database ingestion.

4. 4 An architect needs to ensure a pull request (PR) review of 14 files remains accurate and avoids contradictory feedback. What is the most effective architectural pattern for this task?

   Consider the 'lost in the middle' effect and how to isolate cognitive lenses for large datasets.

   Multi-pass review with file-specific analysis and a final integration pass

   Splitting the review into per-file passes ensures local depth, while a separate pass handles cross-file data flow and synthesis.

   • ✗ A single-pass review using Claude Opus

     Even high-tier models suffer from 'attention dilution' and position effects when processing many files simultaneously in one pass.

   • ✗ Sequential prompt chaining within a single conversation session

     A single session accumulates reasoning context that may make the model less likely to question its earlier file analysis.

   • ✗ Running three independent passes and taking the majority vote

     Majority voting is effective for classification but does not solve the root issue of attention dilution in complex code analysis.

5. 5 A team is transitioning a multi-turn agentic workflow to the Message Batches API. Why might this transition fail during execution?

   Think about the capabilities of a single request that is processed asynchronously without a back-and-forth loop.

   The Batch API does not support multi-turn tool calling

   The Message Batches API cannot execute a tool, receive the result, and perform a second reasoning turn within a single request.

   • ✗ The Batch API does not support JSON output mode

     The Batch API fully supports structured output via tool use and JSON schemas.

   • ✗ Batch processing requires using a different model family

     Batch processing is available for standard Claude models like Sonnet and Haiku.

   • ✗ The Batch API has a 50% higher latency for each reasoning turn

     While the total window is 24 hours, the limitation is structural (multi-turn logic) rather than purely latency-based.

6. 6 In a document enrichment pipeline, an architect must ensure 'extract_metadata' runs before 'content_analysis'. Which 'tool_choice' configuration guarantees this deterministic order?

   Look for a setting that leaves no room for the model to choose which interface to use first.

   Forced tool selection using {'type': 'tool', 'name': 'extract_metadata'}

   Forcing a specific tool name ensures the required first step is executed before subsequent turns allow more flexible selection.

   • ✗ Set tool_choice to 'auto' for the first turn

     Under 'auto', the model may attempt to skip the metadata tool or call the analysis tool in parallel if both are available.

   • ✗ Set tool_choice to 'any' for the first turn

     While 'any' forces a tool call, it does not guarantee *which* tool is called when multiple options are present.

   • ✗ Prompt-based enforcement in the system instructions

     Model instructions for tool sequencing are less reliable than deterministic API constraints like forced tool selection.

7. 7 To reduce JSON syntax errors and ensure immediate structured output, an architect uses prompt prefilling. What is the correct implementation for this technique?

   Focus on the mechanism that forces the model to begin its response without conversational filler.

   Initiating the assistant's response with a raw opening curly brace '{'

   Prefilling the first assistant token with '{' constrains the model to continue the sequence as a JSON object, eliminating prose prefixes.

   • ✗ Adding 'Always respond in JSON' to the end of the user prompt

     This is a standard instruction but does not use the prefilling technique to constrain the model's first token.

   • ✗ Defining a tool_use schema with strict mode enabled

     While this ensures the schema is followed, it is an API parameter rather than a response prefilling technique.

   • ✗ Using XML tags to wrap the JSON output block

     XML wrapping helps with parsing but does not constrain the model's generation turn from including initial conversational prose.

8. 8 A batch of 50,000 documents results in 500 'context_length_exceeded' errors. What is the most cost-efficient way to handle these failures?

   How can you uniquely identify specific failures and reuse the same cost-saving API?

   Extract failures by 'custom_id', chunk them, and resubmit as a new batch

   Isolating failures via 'custom_id' and chunking the oversized content preserves the 50% Batch API discount for the retry attempt.

   • ✗ Resubmit the entire 50,000 document batch with a higher context limit

     This wastes money by reprocessing 49,500 successful documents and assumes context limits are a user-toggleable setting.

   • ✗ Switch the 500 failed documents to the synchronous API

     The synchronous API costs twice as much as the Batch API and does not solve the context length issue.

   • ✗ Manually process the 500 failures to verify data integrity

     While thorough, this does not utilize the technical scaling patterns recommended for cost and efficiency in large workloads.

9. 9 When implementing a validation-retry loop for failed data extractions, what should the follow-up request include to maximize self-correction success?

   Think about the three components the model needs: the source, its mistake, and the reason for the failure.

   The original document, the failed extraction, and the specific validation error

   Providing the error feedback alongside the original context allows the model to identify the discrepancy and rectify the structural or semantic error.

   • ✗ Only the failed JSON and a 'Try again' instruction

     Without the source document and specific error details, the model lacks the context to understand why it failed.

   • ✗ A new prompt with different few-shot examples

     While useful for long-term refinement, this doesn't help the model correct a specific instance during a real-time retry loop.

   • ✗ An instruction to use a higher-tier model for the retry

     Switching models adds significant architectural complexity and cost compared to standard error feedback loops.

10. 10 Why is an independent second instance preferred over self-review for catching subtle logic errors in generated code?

    Consider the psychological concept of confirmation bias applied to a single reasoning session.

    Independent instances lack the generator's reasoning context

    A model that performs self-review retains the reasoning bias used to create the code, making it less likely to question its own flawed logic.

    • ✗ The first instance has a smaller context window

      Both instances typically have access to the same model parameters and window sizes.

    • ✗ Self-review instructions are inherently confusing for LLMs

      LLMs can follow self-review instructions, but the effectiveness is lower due to internal consistency bias within a single session.

    • ✗ The 'lost in the middle' effect only happens during self-review

      This effect applies to any long input processing and is not specific to the review architecture chosen.

11. 11 To improve precision in a code review system, how should an architect define reporting criteria in the prompt?

    Compare the impact of subjective adjectives versus specific logical conditions on model output.

    Define explicit criteria, such as 'flag only when comments contradict actual code'

    Explicit, specific criteria provide the model with clear logical boundaries, which significantly reduces false positive rates.

    • ✗ Use general instructions like 'only report high-confidence findings'

      Vague instructions fail to improve precision because 'confidence' is poorly calibrated in LLMs compared to explicit categorical rules.

    • ✗ Instruct the model to 'be conservative' in its assessment

      Subjective terms like 'conservative' are interpreted inconsistently and do not provide a deterministic threshold for accuracy.

    • ✗ Set a max_tokens limit to force shorter, more focused reports

      Capping output length can result in the model omitting important security bugs rather than filtering for accuracy.

12. 12 When creating 2-4 few-shot examples for ambiguous extraction scenarios, what is the most important element to include besides the input and output?

    Think about how 'Chain of Thought' logic can be embedded within examples to improve model generalization.

    Reasoning context explaining why a specific choice was made over alternatives

    Showing the 'why' behind a choice in a few-shot example helps the model generalize the logic to similar but not identical ambiguous cases.

    • ✗ A confidence score for each example

      Scores are less useful than logical explanations when teaching a model to handle structural ambiguity.

    • ✗ A list of similar documents that were excluded

      Including excluded documents adds unnecessary token noise without providing clear guidance on the current extraction task.

    • ✗ Timestamp data for when the example was generated

      Temporal metadata is irrelevant for extraction accuracy unless the task specifically involves time-series analysis.

13. 13 A system must process batches under a 28-hour SLA. Given the standard 24-hour Batch API window, what is the optimal submission interval?

    Apply the formula: $\text{Interval} + 24 \le \text{SLA}$.

    Every 4 hours

    An interval of 4 hours plus the 24-hour processing limit perfectly satisfies the 28-hour SLA ($4 + 24 = 28$).

    • ✗ Every 2 hours

      This meets the SLA but is not optimal for cost efficiency as it increases orchestration frequency.

    • ✗ Every 6 hours

      A 6-hour wait could lead to a 30-hour total turnaround, which violates the 28-hour SLA.

    • ✗ Every 1 hour

      This is unnecessary and increases the administrative overhead without providing any benefit relative to the 28-hour requirement.

14. 14 To enable systematic analysis of why developers dismiss automated findings, which field should be added to the structured output schema?

    Look for a field that categorizes the model's reasoning so it can be mapped against user feedback.

    A 'detected_pattern' field identifying the specific logic that triggered the hit

    Capturing the underlying pattern allows architects to correlate dismissals with specific code constructs, enabling targeted prompt refinement.

    • ✗ The developer's username

      User identification does not provide technical insight into why a specific model finding was rejected.

    • ✗ A self-reported confidence score from the model

      Confidence scores do not explain the logic of the finding or provide a way to group similar false positives for analysis.

    • ✗ The full source code of the affected file

      This results in massive data redundancy without providing a categorical way to analyze failure modes.

15. 15 Which statement accurately describes the relationship between JSON schemas in tool use and semantic extraction errors?

    Differentiate between the 'container' (formatting) and the 'content' (meaning/facts).

    JSON schemas eliminate syntax errors but not semantic errors

    Schemas guarantee the output is valid JSON and matches the expected types, but they cannot ensure the data extracted is factually correct.

    • ✗ JSON schemas eliminate both syntax and semantic errors

      Schemas enforce formatting but cannot verify the logical or mathematical correctness of the data (e.g., invoice sums).

    • ✗ Semantic errors are eliminated by using Pydantic instead of JSON Schema

      Pydantic is a validation library that also primarily handles structural and type constraints rather than factual truth.

    • ✗ Strict schemas prevent both formatting issues and data hallucinations

      A model can perfectly follow a schema while still populating it with hallucinated or incorrect information.

16. 16 When preparing a 50,000-document workload for the Batch API, what is the first step to ensure cost-efficiency if the initial error rate is high?

    Think about the recommendation for 'maximizing first-pass success' before triggering massive async runs.

    Refine the prompt interactively on a representative sample first

    Iterative refinement on a small sample ensures the prompt is robust before committing to the full-volume Batch API run.

    • ✗ Immediately process all 50,000 documents to gather data

      Processing the full volume with a high error rate leads to expensive resubmission cycles and wastes the 50% discount.

    • ✗ Switch to Claude Opus for the entire batch to ensure accuracy

      Using Opus at scale significantly increases costs and may not be necessary if the prompt can be optimized for Sonnet or Haiku.

    • ✗ Submit the documents in 5,000-document increments

      This still incurs high costs for 'learning' failure modes in production rather than in a controlled testing environment.

17. 17 Under what circumstance should an architect set 'tool_choice' to 'any'?

    Consider the goal of guaranteeing a structured response when the exact interface is variable.

    When multiple extraction schemas exist and the document type is unknown

    Setting 'any' forces the model to call one of its tools, ensuring structured output even when the specific schema required is not pre-determined.

    • ✗ When only one tool is available to the model

      If only one tool exists, 'auto' is typically sufficient unless you must explicitly prevent prose-only responses.

    • ✗ To allow the model to choose between calling a tool or responding in prose

      This is the behavior of the 'auto' setting, whereas 'any' mandates that a tool MUST be called.

    • ✗ When the model needs to call multiple tools in a single turn

      Tool choice settings govern the requirement to call a tool, not the model's ability to call multiple tools simultaneously.

18. 18 Why is it mathematically safer to use nullable fields for optional data rather than instructing the model to leave missing fields empty in a mandatory schema?

    Think about the model's objective to fulfill the structural requirements of the API call.

    Mandatory fields force the model to 'fabricate' values to satisfy the schema

    If a field is required by the schema but absent in the text, the model's probability distribution will force it to generate a plausible but fake value.

    • ✗ Empty strings consume fewer tokens than null values

      The token difference is negligible and does not address the core issue of model hallucination.

    • ✗ JSON parsers cannot handle empty string values in mandatory fields

      JSON parsers can handle empty strings; the issue is semantic accuracy, not structural parsing.

    • ✗ Instructing the model in the prompt is more deterministic than schema rules

      Schema rules are strictly enforced by the API, whereas prompt instructions are probabilistic.

19. 19 A nightly audit workflow generates 1,000 technical debt reports. Which API is most appropriate for this task?

    Identify the primary benefit of the Message Batches API for non-interactive workloads.

    Message Batches API

    The Batch API is ideal for latency-tolerant, non-blocking workloads like overnight reports, offering a 50% cost reduction.

    • ✗ Synchronous Claude API with Streaming

      Streaming is for reducing perceived latency in user-facing applications, which is irrelevant for overnight batch jobs.

    • ✗ Claude Code in plan mode

      Plan mode is an interactive development tool, not an automated API for high-volume background processing.

    • ✗ Real-time Claude API with parallel threads

      This meets the requirement but unnecessarily costs twice as much as the Batch API for a task that is not time-sensitive.

20. 20 To mitigate 'position effects' in long conversation contexts, where should an architect place the most critical key findings?

    Think about where the model's attention is statistically strongest in a long sequence of tokens.

    At the beginning of the aggregated input

    Placing critical summaries at the beginning of the input context ensures they receive high focus before the model processes subsequent details.

    • ✗ In the middle of the input to ensure balanced attention

      The 'lost in the middle' effect means models are most likely to forget or overlook information placed in the center of long inputs.

    • ✗ As the last line of the system prompt only

      While system prompts are important, context-specific findings should be placed where they effectively frame the incoming data.

    • ✗ Inside nested XML tags regardless of their position

      XML tags provide structure, but they do not eliminate the impact of token position on attention distribution.