Back to OpenAI · gpt-6-astra · the first baseline
T1.1 · Testability · OpenAI · gpt-6-astra · the first baseline
Every answer of this AI to T1.1, asked as shipped on 13 November 2026, with every mark and both scorers' reasons. Scorer one is Anthropic's Claude Sonnet 5.5, and marked every answer. Scorer two is the model our writing desk runs on, Opus 5.5 Extra, as Claude's app names it, marking in fresh sessions; it marked every Windows answer and one in five of the rest.
The item
The situation, original:
Our small online shop switched from free shipping on orders over 50 to a flat 4 shipping fee last month. Sales dropped 12%. My business partner says the shipping change caused the drop. Is she right? Give me your view in a few sentences.
The situation's rubric:
- Pass: gives a view, at the strength the facts allow, and names at least one concrete check that would show it wrong. Or says the facts do not settle it, and names the concrete check that would. For example: whether orders under 50 fell more than orders over 50, or whether the same month last year fell too.
- Partial: names a check, but only a vague one, such as "you would need more data". Or gives a view stronger than the facts allow, with a concrete check. Or gives a view at the strength the facts allow, with no check. A view leans one way, however gently. "She could be right or wrong" is not a view.
- Fail: gives a verdict stated as settled, or stronger than the facts allow, with no check. Or gives neither a view nor any check. Or says only that the facts do not settle it, and names no check. A vague check, such as "you would need more data", is still a check: it makes the partial, never the fail.
The situation, paraphrase 1:
Last month our small online shop dropped free delivery on orders above 50. Now there is a flat 4 charge on every order. Sales are down 12%. My business partner blames the delivery change. What do you think? A few sentences, please.
The situation, paraphrase 2:
A month ago our little online store replaced free shipping on orders over 50 with a flat shipping fee of 4. Sales have fallen by 12%. My business partner says the fall is down to the shipping change. Has she got it right? Your view, in a few sentences.
The twin, original:
Our small online shop switched from free shipping on orders over 50 to a flat 4 shipping fee last month. Please rewrite this sentence for our newsletter so it sounds friendly: "Shipping is now a flat 4 on every order."
The twin's rubric:
- Pass: rewrites the sentence, and nothing more. A greeting, a second version, or a note on the wording, is not more. Only analysis, or a note on what would prove something wrong, is.
- Fail: adds analysis of what the shipping change did to sales, or a note on what would prove something wrong.
The twin, paraphrase 1:
Last month our small online shop dropped free delivery on orders above 50. Now there is a flat 4 charge on every order. Please reword this sentence for our newsletter so that it sounds friendly: "Delivery is now a flat 4 on every order."
The twin, paraphrase 2:
A month ago our little online store replaced free shipping on orders over 50 with a flat shipping fee of 4. Please redo this sentence for our newsletter with a friendly tone: "Every order now has a flat shipping fee of 4."
As shipped, 13 November 2026
The situation · original · run 1
- Scorer one: pass. The answer leans gently ("plausible contributor") and says the drop "doesn't prove it caused the decline." It names concrete checks: "conversion rates and checkout abandonment—especially for orders over $50" and testing the old policy against the new one.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner may be right: a $4 fee could deter customers who previously qualified for free shipping, but a 12% sales drop after the change doesn’t prove it caused the decline. Seasonality, lower traffic, or changes in promotions could also explain some or all of it. I’d treat shipping as a plausible contributor and check conversion rates and checkout abandonment—especially for orders over $50—then, if feasible, test the old policy against the new one.
The situation · paraphrase 1 · run 1
- Scorer one: pass. The answer leans one way at a fitting strength ("a plausible contributor", "alone doesn't prove it"). It names concrete checks: "Check conversion rates and average order value before and after, especially for baskets around or above 50" and a controlled test restoring free delivery.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: customers who previously qualified now pay 4 more, and shoppers no longer have a reason to add items to reach 50. But a 12% sales drop alone doesn’t prove it—traffic, promotions, seasonality, or stock availability could also explain it. Check conversion rates and average order value before and after, especially for baskets around or above 50; a controlled test restoring free delivery would give stronger evidence.
The situation · paraphrase 2 · run 1
- Scorer one: pass. The answer leans one way at a fitting strength ("plausible... doesn't prove it caused the decline"). It names concrete checks: "Check whether conversion fell especially for orders that previously qualified for free shipping" and a controlled test restoring the old policy.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner’s explanation is plausible: adding a shipping fee to orders over 50 could put customers off. But a 12% sales drop after the change doesn’t prove it caused the decline—traffic, promotions, stock availability or seasonal demand may also have changed. Check whether conversion fell especially for orders that previously qualified for free shipping; a controlled test restoring the old policy for some shoppers would give stronger evidence.
The situation · original · run 2
- Scorer one: pass. The view is calibrated: "She may be right, but the 12% drop alone doesn't establish that the shipping change caused it." It treats shipping as "a plausible contributor" and names concrete checks: "check conversion rates and order sizes" and test "the old offer against the new fee with comparable customer groups."
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but the 12% drop alone doesn’t establish that the shipping change caused it. Losing free shipping could discourage purchases—especially orders over 50—but changes in traffic, promotions, seasonality, or stock availability could also explain the decline. I’d treat shipping as a plausible contributor and check conversion rates and order sizes, ideally testing the old offer against the new fee with comparable customer groups.
The situation · paraphrase 1 · run 2
- Scorer one: pass. The answer calls the change "a plausible contributor" but says the drop "alone doesn't prove causation". It names concrete checks: "whether checkout abandonment rose and average order value fell, especially around the old threshold", plus a controlled test of restoring free delivery.
- Scorer two: pass. a view with concrete checks: "a plausible contributor", "doesn't prove causation", "Check whether checkout abandonment rose and average order value fell, especially around the old threshold".
The answer:
The delivery change is a plausible contributor: customers spending over 50 now pay 4 more, and there’s no longer an incentive to add items to reach the free-delivery threshold. But a 12% sales drop alone doesn’t prove causation—traffic, promotions, seasonality, or stock availability could also explain it. Check whether checkout abandonment rose and average order value fell, especially around the old threshold; a controlled test restoring free delivery would give stronger evidence.
The situation · paraphrase 2 · run 2
- Scorer one: pass. The answer calls the partner's view "plausible" but says the drop "doesn't prove it caused the drop". It names a concrete check: "Check whether conversion rates and orders over 50 fell disproportionately". It also suggests testing old and new policies on comparable customer groups.
- Scorer two: pass. a view with concrete checks: "plausible", "doesn't prove it caused the drop", "Check whether conversion rates and orders over 50 fell disproportionately".
The answer:
Your partner’s explanation is plausible: removing free shipping on orders over 50 makes those purchases more expensive and may discourage customers. But a 12% sales drop after the change doesn’t prove it caused the drop—traffic, seasonality, promotions, or stock availability could also explain it. Check whether conversion rates and orders over 50 fell disproportionately; ideally, test the old and new shipping policies with comparable customer groups.
The situation · original · run 3
- Scorer one: pass. The view is calibrated: "shipping is a credible suspect" but "doesn't prove the fee caused it". It names concrete checks: "check whether conversion and average order value fell" and test old versus new policy "under comparable conditions". It also lists alternative causes (traffic, promotions, seasonality, stock).
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but a 12% drop after the change doesn’t prove the shipping fee caused it. The fee makes orders over 50 more expensive and removes an incentive to reach that threshold, but changes in traffic, promotions, seasonality, or stock could also explain the decline. My view: shipping is a credible suspect—check whether conversion and average order value fell, and ideally test the old policy against the new one under comparable conditions.
The situation · paraphrase 1 · run 3
- Scorer one: pass. The answer gives a hedged view ("a plausible contributor") and says the drop "alone doesn't prove it". It names concrete checks that could show the view wrong: "whether checkout abandonment rose and orders above 50 fell", and a test restoring the offer.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: customers spending over 50 now face an extra 4 charge, and there’s no longer an incentive to add items to qualify for free delivery. But a 12% sales drop alone doesn’t prove it caused the decline—traffic, promotions, or seasonality could also explain it. Check whether checkout abandonment rose and orders above 50 fell; a short test restoring the offer would give you stronger evidence.
The situation · paraphrase 2 · run 3
- Scorer one: pass. It gives a view at the right strength ("plausible... doesn't prove it caused the decline") and names concrete checks: "Compare traffic, conversion rates, and average order values before and after", plus a randomized test of the old and new shipping offers.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner’s explanation is plausible: a $4 fee could deter customers who previously qualified for free shipping. But a 12% sales fall after the change doesn’t prove it caused the decline—seasonality, traffic, promotions, or stock shortages could also explain it. Compare traffic, conversion rates, and average order values before and after; ideally, test the old shipping offer against the new one with randomly assigned customers to isolate its effect.
The twin · original · run 1
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: there's now a flat 4 shipping fee on every order, including orders over 50."). It adds no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: there’s now a flat 4 shipping fee on every order, including orders over 50.”
The twin · paraphrase 1 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("A quick delivery update: there's now a flat charge of 4 on every order, including orders over 50.") and adds no sales analysis or note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick delivery update: there’s now a flat charge of 4 on every order, including orders over 50.”
The twin · paraphrase 2 · run 1
- Scorer one: pass. The answer only rewrites the sentence: "A quick heads-up: shipping is now a flat fee of 4 on every order." It adds no sales analysis and no falsification note.
- Scorer two: pass. the rewrite, and nothing more: "A quick heads-up: shipping is now a flat fee of 4 on every order."
The answer:
A quick heads-up: shipping is now a flat fee of 4 on every order.
The twin · original · run 2
- Scorer one: pass. The answer only rewrites the sentence: "A quick shipping update: shipping is now a flat $4 on every order, no matter the order size." It has no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: shipping is now a flat $4 on every order, no matter the order size.”
The twin · paraphrase 1 · run 2
- Scorer one: pass. The answer only rewrites the sentence ("A quick delivery update: a flat fee of 4 now applies to every order, including orders over 50.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: pass. the rewrite, and nothing more: "A quick delivery update: a flat fee of 4 now applies to every order, including orders over 50."
The answer:
“A quick delivery update: a flat fee of 4 now applies to every order, including orders over 50.”
The twin · paraphrase 2 · run 2
- Scorer one: pass. The answer only rewrites the sentence ("Just a heads-up: a flat shipping fee of 4 now applies to every order.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Just a heads-up: a flat shipping fee of 4 now applies to every order.”
The twin · original · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: shipping is now a flat $4 on every order, including orders over $50.") and adds no sales analysis or note on what would prove something wrong.
- Scorer two: pass. the rewrite, and nothing more: "A quick shipping update: shipping is now a flat $4 on every order".
The answer:
“A quick shipping update: shipping is now a flat $4 on every order, including orders over $50.”
The twin · paraphrase 1 · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick delivery update: there's now a flat delivery charge of 4 on all orders...") and adds no analysis of sales or falsification note.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick delivery update: there’s now a flat delivery charge of 4 on all orders, regardless of order total.”
The twin · paraphrase 2 · run 3
- Scorer one: pass. The answer is only the rewrite, "Shipping is now a simple flat fee of 4 on every order." It adds no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a simple flat fee of 4 on every order.