Which AWS model gives the best result on our LLM benchmark task?

01Problem statement and objective

Information retrieval technologies such as vector search are spreading quickly. According to MarketsandMarkets, the global vector database market is projected to grow from USD 2.65 billion in 2025 to USD 8.95 billion in 2030, a compound annual growth rate of 27.5%.^[1]

Many teams build these systems, but few regularly measure how well they work. To check whether a retrieval system returns the right people, you first need the correct answer. We call this answer key the golden dataset.

At NativeAI, we keep the Query Expansion (QE) structure we built for Lexical Search up to date and measure the impact of every update. This study is the foundation of that measurement. We looked at which model can produce the golden dataset, how consistently it does so, and at what cost.

Our dataset contains 2,500 reviews. Each review is a Digital Twin (DT): a synthetic persona's survey record of 130 questions and answers. For every review, the golden dataset answers the question "Is this person relevant to this question?" with one label: candidate (1) or exclude (0). With 15 questions that comes to 37,500 labels, so a large language model (LLM) generates them.

Why is this not an easy problem?

• The answer is not always stated explicitly. For the question on heavy-industry shippers, a person whose review says "I ship palletized production parts" belongs to the group, although the review never says so directly.
• What counts as a candidate depends on the question type. For the question "Would this person pay a little extra to get a parcel tomorrow morning instead of in three days? Answer yes or no.", a person who says "I wouldn't pay" is still a candidate, because the review answers the question. For "What do students say about getting packages delivered to where they live?", only students are candidates.
• Multiple conditions must hold at the same time. In a group defined as "millennials who own their home in the western United States", a person who fails any one condition is not part of the group.
• The answer is buried in long text. In every request the model reads a review of ≈5,000 tokens and has to find the relevant line.

02Why AWS Bedrock?

All models except GLM 5.3, GLM 5.3 Fast and GLM 5.3 Flash ran on Amazon Web Services (AWS) Bedrock. We chose Bedrock to keep our data secure:

• Data is not shared with model providers. Bedrock runs the models in separate accounts that AWS manages. Model providers have no access to these accounts, to our prompts or to the responses.^[2]
• Data is not used to train models.^[3]
• Data is encrypted. Data in transit uses at least TLS 1.2, and data at rest is encrypted. Keys can be managed with AWS KMS.^[2][3]
• We control data retention. An account setting decides whether prompts and responses are stored. In "none" mode, no data is written to persistent storage.^[4]
• Compliance standards. Bedrock is in scope for ISO, SOC and CSA STAR Level 2. It is HIPAA eligible and can be used in compliance with GDPR.^[3]

GLM 5.3, GLM 5.3 Fast and GLM 5.3 Flash are not available on AWS Bedrock. We ran them through the Model APIs of Baseten, the other provider we trust. GLM 5.3 Fast is a faster version of GLM 5.3 that Baseten serves. Baseten's security commitments are:^[5]

• Data is not stored. Baseten keeps no persistent copy of the inputs and outputs sent to its Model APIs.
• Data is not used to train models.
• Data is encrypted. Data in transit uses TLS 1.2 or later, and data at rest uses AES-256.
• Independent audit and compliance. Baseten goes through a SOC 2 Type II audit on a regular basis. It offers HIPAA-compliant deployments and commits to GDPR compliance.
• Access is limited. Baseten grants access only on an approved request and under the principle of least privilege.

03Experimental setup

The dataset is divided into 5 segments of 500 reviews each. We picked 1 review from each segment and asked 15 questions about these 5 reviews, which gives 75 labels per model.

The questions vary along two dimensions:

• Format (6 types): short · what, short · why, medium, multiple choice, Likert and yes/no.
• Evidence type (2 types): explicit questions, where the answer is stated in the review, and implicit questions, where the answer must be inferred from the review.

We produced the golden labels with Opus 5.5 (effort Low), the most capable and most expensive model we tested, and checked all 75 labels by hand. We wanted to know whether a cheaper model would deterministically give the same labels. When we generated the golden again, all 75 labels came out identical.

How did we develop the prompt? We wrote a single shared prompt for all questions, with no question-specific rules. When we saw incorrect labels, we fixed the prompt with general rules instead of label-specific ones, and we left examples from the question set out of it. Each fix was small and reviewed on its own. The golden run and all 14 model configurations we compared (11 models with different effort settings) used the same final prompt version.

04How did we score?

We compared each label with the golden. A label that differs from the golden can go two ways. A false negative (FN) is a missed review and the most severe error, because later steps never see that review. A false positive (FP) is an extra review, which can be filtered out later.

Two metrics track these errors. Recall shows how many of the golden's candidate reviews the model also found; it goes down as FN goes up. Precision shows how many of the model's candidate labels are also candidate in the golden; it goes down as FP goes up.

Our success criterion is a recall of at least 90% for every question. No question in the test has more than 5 candidate reviews, so a single miss lowers recall to 80% or below. The only way to meet the criterion is to miss no review at all.

05Model golden match scorecard

Besides the golden (Opus 5.5), we tested 11 models from 6 providers in 14 configurations. A configuration consists of the model and its effort level. Every model also writes a reason (a one-sentence justification) for each label. Of the 1,050 labels in the 14 non-golden results, 101 differ from the golden: 82 FN and 19 FP.

Only 5 of the 14 results met the per-question 90% criterion: Grok 4.7, Kimi K3, GLM 5.3 Fast, GLM 5.3 and Sonnet 5.5, none of which missed a single review. The aggregate score can hide this. GLM 5 (Low) reached an overall recall of 90.6% but found only 1 of the 4 candidate reviews for Q08 (25%). All metrics are in the appendix table at the end of the report.

06Model cards

Three models, three different profiles. One never differed from the golden, one only added extra reviews, and one missed half of the candidate reviews. Cards for all models are in Appendix B.

07Price and cost

Figure 5 puts each model's predicted cost for labeling the full dataset (37,500 requests) next to its golden match rate. The three models that fully match the golden cost ≈$582 (GLM 5.3 Fast · Baseten), ≈$608 (Grok 4.7) and ≈$931 (Kimi K3); the golden model, Opus 5.5, costs ≈$1,242. Cost alone does not decide the outcome. Sonnet 5 costs about the same as Grok (≈$621) but missed 9 of the 32 candidate reviews. None of the models that cost less than $300 met the per-question 90% criterion.

The text sent to the model is ≈95 times longer than the response, so the input price drives almost all of the cost. With these token counts, full labeling of 37,500 requests costs ≈$608 with Grok 4.7 and ≈$1,242 with Opus 5.5.

08Effect of effort and model version

The effort level sets how much the model reasons before it answers. We expected more reasoning to give better results, but the opposite happened. Almost all of the extra mismatches are FN: the models that reasoned more became stricter and excluded candidate reviews. Moving to a newer version did help.

09Per-question analysis

Mismatches are not spread at random. They cluster in certain questions and in certain question types. 62 of the 101 mismatches occur in three questions: Q08 (medium · implicit, 28), Q13 (Likert · explicit, 20) and Q12 (Likert · implicit, 14).

The categories make the pattern clearer. 66 mismatches (65%) come from implicit questions, against only 34 from explicit ones. By format, medium and Likert questions account for 75 of the 101 mismatches, while short · what questions have only 2. Q09 goes the other way: 11 of its 12 mismatches are FP.

10Limitations

11Conclusion

AAll metrics

BAll model cards

Each card summarizes a model's 75 labels. In the grid below, each column is a question and each row a test review. A colored square means the label for that question and review differs from the golden.