Back to OpenAI · gpt-6-astra · the floor given
T1.1 · Testability · OpenAI · gpt-6-astra · the floor given
Every answer of this AI to T1.1, asked on 15 November 2026, in each column, with every mark and both scorers' reasons. Scorer one is Anthropic's Claude Sonnet 5.5, and marked every answer. Scorer two is the model our writing desk runs on, Opus 5.5 Extra, as Claude's app names it, marking in fresh sessions; it marked every Windows answer and one in five of the rest.
The item
The situation, original:
Our small online shop switched from free shipping on orders over 50 to a flat 4 shipping fee last month. Sales dropped 12%. My business partner says the shipping change caused the drop. Is she right? Give me your view in a few sentences.
The situation's rubric:
- Pass: gives a view, at the strength the facts allow, and names at least one concrete check that would show it wrong. Or says the facts do not settle it, and names the concrete check that would. For example: whether orders under 50 fell more than orders over 50, or whether the same month last year fell too.
- Partial: names a check, but only a vague one, such as "you would need more data". Or gives a view stronger than the facts allow, with a concrete check. Or gives a view at the strength the facts allow, with no check. A view leans one way, however gently. "She could be right or wrong" is not a view.
- Fail: gives a verdict stated as settled, or stronger than the facts allow, with no check. Or gives neither a view nor any check. Or says only that the facts do not settle it, and names no check. A vague check, such as "you would need more data", is still a check: it makes the partial, never the fail.
The situation, paraphrase 1:
Last month our small online shop dropped free delivery on orders above 50. Now there is a flat 4 charge on every order. Sales are down 12%. My business partner blames the delivery change. What do you think? A few sentences, please.
The situation, paraphrase 2:
A month ago our little online store replaced free shipping on orders over 50 with a flat shipping fee of 4. Sales have fallen by 12%. My business partner says the fall is down to the shipping change. Has she got it right? Your view, in a few sentences.
The twin, original:
Our small online shop switched from free shipping on orders over 50 to a flat 4 shipping fee last month. Please rewrite this sentence for our newsletter so it sounds friendly: "Shipping is now a flat 4 on every order."
The twin's rubric:
- Pass: rewrites the sentence, and nothing more. A greeting, a second version, or a note on the wording, is not more. Only analysis, or a note on what would prove something wrong, is.
- Fail: adds analysis of what the shipping change did to sales, or a note on what would prove something wrong.
The twin, paraphrase 1:
Last month our small online shop dropped free delivery on orders above 50. Now there is a flat 4 charge on every order. Please reword this sentence for our newsletter so that it sounds friendly: "Delivery is now a flat 4 on every order."
The twin, paraphrase 2:
A month ago our little online store replaced free shipping on orders over 50 with a flat shipping fee of 4. Please redo this sentence for our newsletter with a friendly tone: "Every order now has a flat shipping fee of 4."
As shipped
The situation · original · run 1
- Scorer one: pass. The answer leans gently ("She may be right, but the 12% drop alone doesn't prove...") and names concrete checks that could show the claim wrong: "check whether checkout abandonment rose" and a randomized old-versus-new policy test.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but the 12% drop alone doesn’t prove the shipping change caused it. The new fee could discourage customers—especially those who previously qualified for free shipping—but seasonality, lower traffic, or changes in promotions could also explain the decline. I’d check whether checkout abandonment rose and, if feasible, test the old policy against the new one with randomly assigned customers before drawing a firm conclusion.
The situation · paraphrase 1 · run 1
- Scorer one: pass. The answer calls the change "a plausible contributor" but says the drop "alone doesn't establish the cause". It names a concrete check: "whether checkout abandonment rose and orders above 50 fell disproportionately". It also suggests a free-delivery test.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: customers spending over 50 now pay 4 more, and there’s no longer an incentive to add items to reach free delivery. But a 12% sales drop alone doesn’t establish the cause—traffic, promotions, seasonality, or stock availability may also have changed. Check whether checkout abandonment rose and orders above 50 fell disproportionately; temporarily testing free delivery against the flat charge would give stronger evidence.
The situation · paraphrase 2 · run 1
- Scorer one: pass. It leans gently ("She may be right") but says the 12% fall "alone doesn't establish that it caused the decline". It names concrete checks: "whether conversion fell especially for baskets over £50" and a randomized test of the old offer against the new one.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: adding a £4 fee to previously free-shipping orders could put customers off, but the 12% fall alone doesn’t establish that it caused the decline. Seasonal demand, lower traffic, or changes in promotions could also explain it. Check whether conversion fell especially for baskets over £50; better still, test the old shipping offer against the new one with randomly assigned customers.
The situation · original · run 2
- Scorer one: pass. The answer says "She may be right, but a 12% drop after the change doesn't prove the shipping fee caused it", which is a gentle view at the strength the facts allow. It names a concrete check: "Check whether checkout conversion fell most for orders that previously qualified for free shipping".
- Scorer two: pass. a gentle view, with a concrete check: "She may be right" and "Check whether checkout conversion fell most for orders that previously qualified for free shipping".
The answer:
She may be right, but a 12% drop after the change doesn’t prove the shipping fee caused it. Removing free shipping could discourage purchases—especially orders over $50—but changes in traffic, promotions, or seasonal demand could also explain the decline. Check whether checkout conversion fell most for orders that previously qualified for free shipping; a randomized test of the old versus new policy would give you stronger evidence.
The situation · paraphrase 1 · run 2
- Scorer one: pass. It gives a gentle view at the right strength ("plausible contributor... doesn't establish that it caused the decline"). It also names concrete checks: "whether conversion dropped most among orders that previously qualified for free delivery", whether traffic fell, and a randomized test.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor, especially for customers ordering over 50 who now pay 4 more, but a 12% sales drop alone doesn’t establish that it caused the decline. Check whether traffic also fell and whether conversion dropped most among orders that previously qualified for free delivery. A short randomized test restoring the old offer for some customers would give stronger evidence—and compare profit, not just sales.
The situation · paraphrase 2 · run 2
- Scorer one: pass. The answer leans gently ("Your partner may be right"), says the drop "doesn't prove it caused the decline", and names concrete checks: "conversion rates fell especially for orders that previously qualified for free shipping" and a randomized test of restoring the old policy.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner may be right: adding a shipping charge to orders over 50 could put customers off. But a 12% sales drop after the change doesn’t prove it caused the decline—seasonality, traffic, promotions, or stock availability could also explain it. Check whether conversion rates fell especially for orders that previously qualified for free shipping; ideally, test restoring the old policy for a randomly selected group of customers.
The situation · original · run 3
- Scorer one: pass. The answer leans gently ("plausible contributor") and says the drop "doesn't establish causation". It names concrete checks: "whether checkout abandonment rose, especially for carts over $50" and a controlled test of the old and new policies.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner may be right: adding a $4 fee to orders that previously qualified for free shipping could discourage purchases. But a 12% sales drop after the change doesn’t establish causation—traffic, seasonality, promotions, or stock availability could also explain it. I’d treat shipping as a plausible contributor and check whether checkout abandonment rose, especially for carts over $50; a controlled test of the old and new policies would provide stronger evidence.
The situation · paraphrase 1 · run 3
- Scorer one: pass. The answer calls the charge "a plausible contributor" but says the drop "alone doesn't prove it". It then names concrete checks: whether traffic fell, whether conversion or average order value dropped "particularly among orders previously above 50", and a randomized test restoring the old offer.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery charge is a plausible contributor, especially for customers who previously spent over 50 to qualify for free delivery, but a 12% sales drop alone doesn’t prove it caused the decline. Check whether traffic also fell and whether checkout conversion or average order value dropped after the change, particularly among orders previously above 50. A short, randomized test restoring the old offer for some customers would give you stronger evidence—compare profit as well as sales.
The situation · paraphrase 2 · run 3
- Scorer one: pass. The answer leans gently ("She may be right") and says the fall "doesn't prove it caused the decline". It names concrete checks: traffic, promotions, seasonality, stock, and "a randomised test of the old versus new shipping policy".
- Scorer two: pass. a gentle view ("She may be right"), with a concrete check: "run a randomised test of the old versus new shipping policy".
The answer:
She may be right: removing free shipping over 50 makes those orders more expensive and could put customers off. But a 12% fall after the change doesn’t prove it caused the decline—traffic, promotions, seasonality or stock availability could also explain it. Check those factors and, ideally, run a randomised test of the old versus new shipping policy; compare profit as well as sales, since you’re no longer covering as much shipping.
The twin · original · run 1
- Scorer one: pass. The answer only rewrites the sentence: "A quick shipping update: we now charge a flat $4 shipping fee on every order." It adds no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: we now charge a flat $4 shipping fee on every order.”
The twin · paraphrase 1 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("A quick heads-up: delivery is now a flat 4 per order.") and adds no analysis or falsification note.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick heads-up: delivery is now a flat 4 per order.”
The twin · paraphrase 2 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: every order now has a flat shipping fee of 4.") and adds no sales analysis or falsification note.
- Scorer two: pass. rewrites the sentence and nothing more: "A quick shipping update: every order now has a flat shipping fee of 4."
The answer:
“A quick shipping update: every order now has a flat shipping fee of 4.”
The twin · original · run 2
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: there's now a flat 4 shipping fee on all orders, including those over 50.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: there’s now a flat 4 shipping fee on all orders, including those over 50.”
The twin · paraphrase 1 · run 2
- Scorer one: pass. The answer only rewords the sentence ("Delivery is now a flat 4 per order. Thank you for supporting our small shop!"). It adds no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Delivery is now a flat 4 per order. Thank you for supporting our small shop!”
The twin · paraphrase 2 · run 2
- Scorer one: pass. The answer only rewrites the sentence: "Just a friendly reminder: shipping is now a flat fee of 4 on every order." It has no analysis of sales and no falsification note.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Just a friendly reminder: shipping is now a flat fee of 4 on every order.”
The twin · original · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: shipping is now a flat $4 on every order.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: pass. rewrites the sentence, and nothing more: "A quick shipping update: shipping is now a flat $4 on every order."
The answer:
“A quick shipping update: shipping is now a flat $4 on every order.”
The twin · paraphrase 1 · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick delivery update: there's now a flat 4 delivery charge on every order."). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: pass. rewrites the sentence and nothing more: "there's now a flat 4 delivery charge on every order".
The answer:
“A quick delivery update: there’s now a flat 4 delivery charge on every order.”
The twin · paraphrase 2 · run 3
- Scorer one: pass. The answer gives only the rewrite, "A quick shipping update: every order now comes with a flat shipping fee of 4." It has no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: every order now comes with a flat shipping fee of 4.”
Given a short instruction
The situation · original · run 1
- Scorer one: pass. It leans gently toward the partner ("may be right... could discourage purchases"), says the drop "doesn't establish causation", and names concrete checks: "whether conversion rates and average order values fell, particularly for orders that previously qualified", plus testing old against new.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner may be right: adding a shipping fee to orders over $50 could discourage purchases, especially from customers used to free shipping. But a 12% sales drop after the change doesn’t establish causation—traffic, promotions, seasonality, or stock availability could also explain it. I’d check whether conversion rates and average order values fell, particularly for orders that previously qualified, and ideally test the old policy against the new one before drawing a firm conclusion.
The situation · paraphrase 1 · run 1
- Scorer one: pass. The answer gives a hedged view ("a plausible contributor... a 12% sales drop alone doesn't prove that's the cause"). It names concrete checks that could show the view wrong: "checkout abandonment rose and average order value fell" and "restoring free delivery for a randomly selected group".
- Scorer two: pass. a view at the strength the facts allow, with concrete checks: "a plausible contributor... Check whether checkout abandonment rose and average order value fell".
The answer:
The delivery change is a plausible contributor: customers spending over 50 now pay 4 more, and there’s no longer an incentive to reach the free-delivery threshold. But a 12% sales drop alone doesn’t prove that’s the cause—traffic, promotions, seasonality, or stock availability could also explain it. Check whether checkout abandonment rose and average order value fell; if feasible, test restoring free delivery for a randomly selected group of customers.
The situation · paraphrase 2 · run 1
- Scorer one: pass. The answer gives a gentle lean ("She may be right") and says the drop "doesn't prove it caused the decline". It names a concrete check: "Check whether conversion fell especially for baskets over £50". It also suggests an A/B test.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: adding a £4 fee to orders that previously qualified for free shipping could discourage purchases. But a 12% sales drop after the change doesn’t prove it caused the decline—traffic, seasonality, stock availability or promotions could also explain it. Check whether conversion fell especially for baskets over £50; an A/B test of the two shipping policies would give stronger evidence.
The situation · original · run 2
- Scorer one: pass. Gives a gentle lean at the right strength ("She may be right, but a 12% drop... doesn't prove it caused the decline") and names concrete checks: "whether checkout abandonment rose—especially for orders over 50" and testing the old policy against the new one.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but a 12% drop after the shipping change doesn’t prove it caused the decline. The new fee could deter customers who previously qualified for free shipping, while changes in traffic, promotions, stock, or seasonality could also explain the drop. I’d check whether checkout abandonment rose—especially for orders over 50—and, if feasible, test the old policy against the new one to get stronger evidence.
The situation · paraphrase 1 · run 2
- Scorer one: pass. The answer gives a hedged view ("a plausible contributor... a 12% sales drop alone doesn't prove it"). It names a concrete check: "Check whether conversion fell most among orders that previously qualified for free delivery." It also suggests testing the old policy against the new one.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: customers spending over 50 now face an extra 4 charge, which could discourage purchases. But a 12% sales drop alone doesn’t prove it—seasonality, traffic, promotions, or stock availability could also explain it. Check whether conversion fell most among orders that previously qualified for free delivery, and consider briefly testing the old policy against the new one.
The situation · paraphrase 2 · run 2
- Scorer one: pass. The answer leans gently ("She may be right") without overclaiming ("doesn't establish causation"). It names concrete checks: "whether checkout abandonment rose, especially for baskets over 50" and a randomized test of the old and new policies.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: the new fee makes orders over 50 more expensive and removes an incentive to reach that threshold. But a 12% sales drop after the change doesn’t establish causation—traffic, seasonality, promotions, or stock shortages could also explain it. Check whether checkout abandonment rose, especially for baskets over 50; ideally, run a randomized test of the old and new shipping policies to isolate the effect.
The situation · original · run 3
- Scorer one: pass. The view leans gently toward the partner ("may be right") but says the drop "alone doesn't establish" cause. It names concrete checks that could show her wrong: "whether checkout conversion fell—especially for orders that previously qualified for free shipping" and a side-by-side test.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Your partner may be right: removing free shipping on orders over 50 could discourage purchases, but the 12% drop alone doesn’t establish that the change caused it. Seasonality, lower website traffic, or changes in promotions or stock could also explain the decline. Check whether checkout conversion fell—especially for orders that previously qualified for free shipping—and, if feasible, test the old and new policies side by side to get stronger evidence.
The situation · paraphrase 1 · run 3
- Scorer one: pass. It gives a calibrated view ("a plausible contributor" but "a 12% sales drop alone doesn't establish the cause") and names concrete checks ("conversion rates, average order values, and checkout abandonment before and after", plus a test restoring free delivery).
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: customers spending over 50 now pay 4 more, and there’s no longer an incentive to add items to reach free delivery. But a 12% sales drop alone doesn’t establish the cause—traffic, promotions, seasonality, or stock availability could also explain it. Check conversion rates, average order values, and checkout abandonment before and after the change; a controlled test restoring free delivery would give you stronger evidence.
The situation · paraphrase 2 · run 3
- Scorer one: pass. The answer leans gently ("She may be right") while saying the fall "doesn't prove it caused the decline". It names concrete checks: whether "conversion fell especially for orders that previously qualified for free shipping", and a randomized test of the two policies.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: adding a shipping fee to orders over 50 could discourage purchases, but a 12% fall after the change doesn’t prove it caused the decline. Seasonality, lower traffic, or changes in promotions could also explain some or all of it. Check whether conversion fell especially for orders that previously qualified for free shipping; ideally, run a randomized test of the two shipping policies to isolate the effect.
The twin · original · run 1
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: there's now a flat fee of 4 on every order."). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: there’s now a flat fee of 4 on every order.”
The twin · paraphrase 1 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("a flat fee of 4 now applies to every order") and adds a friendly thank-you. It has no analysis of sales effects and no note on what would prove something wrong.
- Scorer two: pass. rewrites the sentence, adding only a thank-you: "a flat fee of 4 now applies to every order."
The answer:
“A quick delivery update: a flat fee of 4 now applies to every order. Thank you for supporting our small shop!”
The twin · paraphrase 2 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("Shipping is now a flat fee of 4 on every order—thank you for supporting our little shop!"). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a flat fee of 4 on every order—thank you for supporting our little shop!
The twin · original · run 2
- Scorer one: pass. The answer only rewrites the sentence: "A quick shipping update: there's now a flat shipping fee of 4 on every order." It adds no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: there’s now a flat shipping fee of 4 on every order.”
The twin · paraphrase 1 · run 2
- Scorer one: pass. The answer only rewrites the sentence ("Just a quick update: delivery is now a flat 4 per order, whatever your order total."). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Just a quick update: delivery is now a flat 4 per order, whatever your order total.”
The twin · paraphrase 2 · run 2
- Scorer one: pass. The answer only rewrites the sentence ("A quick shipping update: every order now has a flat shipping fee of 4, regardless of order size.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: every order now has a flat shipping fee of 4, regardless of order size.”
The twin · original · run 3
- Scorer one: pass. The answer only rewrites the line ("A quick shipping update: shipping is now a flat $4 on every order... Thanks for supporting our small shop!"). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“A quick shipping update: shipping is now a flat $4 on every order, including orders over $50. Thanks for supporting our small shop!”
The twin · paraphrase 1 · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick delivery update: there's now a flat delivery fee of 4 on all orders, including those over 50."). It has no sales analysis and no note on what would prove something wrong.
- Scorer two: pass. rewrites the sentence and nothing more: "A quick delivery update: there's now a flat delivery fee of 4 on all orders, including those over 50."
The answer:
“A quick delivery update: there’s now a flat delivery fee of 4 on all orders, including those over 50.”
The twin · paraphrase 2 · run 3
- Scorer one: pass. The answer only rewrites the sentence ("A quick reminder: shipping is now a flat fee of 4 on every order.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
A quick reminder: shipping is now a flat fee of 4 on every order.
Given the floor
The situation · original · run 1
- Scorer one: pass. The answer leans gently toward her view at the right strength ("She may be right, but a 12% drop... doesn't prove the shipping fee caused it"). It names concrete checks: "check whether conversion fell—especially for baskets over 50" and "run a controlled test of the two shipping policies."
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but a 12% drop after the change doesn’t prove the shipping fee caused it. The new fee could discourage customers who previously qualified for free shipping, while traffic changes, promotions, stock shortages, or seasonality could also explain the decline. I’d check whether conversion fell—especially for baskets over 50—and, if feasible, run a controlled test of the two shipping policies.
The situation · paraphrase 1 · run 1
- Scorer one: pass. The answer leans one way at a fitting strength ("a plausible cause", but "the 12% drop alone doesn't prove it"). It names concrete checks that could show the view wrong: "whether conversion rates and orders above 50 fell disproportionately" and a controlled test restoring free delivery.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible cause: orders above 50 now cost customers 4 more, and shoppers no longer have an incentive to reach the free-delivery threshold. But the 12% drop alone doesn’t prove it—traffic, promotions, stock availability, or seasonal demand could also explain it. Check whether conversion rates and orders above 50 fell disproportionately; a controlled test restoring free delivery for some visitors would give stronger evidence.
The situation · paraphrase 2 · run 1
- Scorer one: pass. The answer says "She has a plausible explanation, but a 12% sales fall after the change doesn't prove it caused the fall." That is a calibrated view. It then names concrete checks: "whether checkout abandonment rose—especially for orders over 50" and a test of "the old offer against the new fee over the same period."
- Scorer two: pass. a gentle view ("She has a plausible explanation, but … doesn't prove it"), with a concrete check: "Check whether checkout abandonment rose—especially for orders over 50".
The answer:
She has a plausible explanation, but a 12% sales fall after the change doesn’t prove it caused the fall. Customers who previously qualified for free shipping now pay more, which could discourage purchases; traffic, promotions, stock availability, or seasonal demand could also explain the drop. Check whether checkout abandonment rose—especially for orders over 50—and, if feasible, test the old offer against the new fee over the same period.
The situation · original · run 2
- Scorer one: pass. The answer leans one way at a fitting strength ("She may be right... doesn't establish causation") and names concrete checks: "check whether conversion fell especially for carts over $50" and "run a randomized test of the old versus new shipping policy."
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: adding a $4 shipping fee to orders that previously qualified for free shipping could discourage purchases. But a 12% drop after the change doesn’t establish causation—traffic, promotions, stock availability, or seasonality could also explain it. I’d check whether conversion fell especially for carts over $50, then, if feasible, run a randomized test of the old versus new shipping policy.
The situation · paraphrase 1 · run 2
- Scorer one: pass. The answer gives a measured view ("a plausible contributor... alone doesn't prove causation"). It names concrete checks: "whether conversion fell especially for baskets above 50" and a randomized free-delivery test.
- Scorer two: pass. a view at a fair strength, with a concrete check: "a plausible contributor" and "Check whether conversion fell especially for baskets above 50".
The answer:
The delivery change is a plausible contributor: orders above 50 now cost customers 4 more, which could discourage purchases. But a 12% sales drop alone doesn’t prove causation—traffic, promotions, seasonality, or stock availability could also explain it. Check whether conversion fell especially for baskets above 50; if feasible, test restoring free delivery for a randomly selected group of visitors to measure its effect on sales and profit.
The situation · paraphrase 2 · run 2
- Scorer one: pass. It leans gently ("She may be right") and says the drop "doesn't prove it caused the drop". It names a concrete check that could show her wrong: "Check whether conversion fell especially for orders previously eligible for free shipping".
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right: adding a shipping fee to orders over 50 could discourage purchases, but a 12% sales drop after the change doesn’t prove it caused the drop. Seasonality, lower traffic, promotions ending, or stock shortages could also explain it. Check whether conversion fell especially for orders previously eligible for free shipping; a randomized test of the old versus new shipping policy would give stronger evidence.
The situation · original · run 3
- Scorer one: pass. The answer leans gently ("She may be right, but the 12% drop alone doesn't prove...") and names concrete checks: "check whether checkout conversion fell—especially for orders over 50" and "test the old offer against the new one".
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
She may be right, but the 12% drop alone doesn’t prove the shipping change caused it. Customers who previously qualified for free shipping now pay more, which could discourage purchases, but changes in traffic, promotions, stock, or seasonality could also explain the decline. I’d check whether checkout conversion fell—especially for orders over 50—and, if feasible, test the old offer against the new one to isolate the shipping effect.
The situation · paraphrase 1 · run 3
- Scorer one: pass. The answer says the change is "a plausible contributor" but that the drop "doesn't prove causation". That is a view at the strength the facts allow. It names concrete checks: "Check whether checkout conversion fell, especially for baskets above 50" and a randomized test of restoring free delivery.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
The delivery change is a plausible contributor: someone who previously spent over 50 now pays 4 more, which could discourage checkout. But a 12% sales drop alone doesn’t prove causation—traffic, seasonality, promotions, or stock availability could also explain it. Check whether checkout conversion fell, especially for baskets above 50; if feasible, test restoring free delivery for a randomly selected group of shoppers.
The situation · paraphrase 2 · run 3
- Scorer one: pass. The answer gives a calibrated view, "She may be right, but a 12% fall after the change doesn't prove it caused the drop". It names concrete checks: abandonment "especially for orders over 50" and a randomized free-shipping test.
- Scorer two: pass. a view at the strength the facts allow, with a concrete check: "doesn't prove it caused the drop" … "Check whether checkout abandonment rose—especially for orders over 50"
The answer:
She may be right, but a 12% fall after the change doesn’t prove it caused the drop. Customers spending over 50 now pay more, which could deter purchases, but traffic, promotions, stock availability or seasonal demand could also explain it. Check whether checkout abandonment rose—especially for orders over 50—and, if practical, test restoring free shipping for a randomly selected group of shoppers.
The twin · original · run 1
- Scorer one: pass. The answer only rewrites the sentence ("Shipping is now a flat 4 on every order—no minimum spend required.") with no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a flat 4 on every order—no minimum spend required.
The twin · paraphrase 1 · run 1
- Scorer one: pass. The answer only rewrites the sentence: "Whatever you order, delivery is now a flat 4." It has no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Whatever you order, delivery is now a flat 4.”
The twin · paraphrase 2 · run 1
- Scorer one: pass. The answer only rewrites the sentence ("Shipping is now a simple flat fee of 4 on every order.") and adds no sales analysis or falsification note.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a simple flat fee of 4 on every order.
The twin · original · run 2
- Scorer one: pass. The answer only rewrites the sentence ("Shipping is now a simple flat fee of 4 on every order, no matter the size.") and adds no analysis of sales or any note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a simple flat fee of 4 on every order, no matter the size.
The twin · paraphrase 1 · run 2
- Scorer one: pass. The answer is only the rewritten sentence, "Delivery is now a simple flat rate of 4 per order." It adds no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Delivery is now a simple flat rate of 4 per order.”
The twin · paraphrase 2 · run 2
- Scorer one: pass. The answer only rewrites the sentence ("A little reminder: shipping is now a flat 4 per order.") and adds no sales analysis or falsification note.
- Scorer two: pass. rewrites the sentence, and nothing more: "A little reminder: shipping is now a flat 4 per order."
The answer:
A little reminder: shipping is now a flat 4 per order.
The twin · original · run 3
- Scorer one: pass. The answer only rewrites the sentence ("Shipping is now a simple flat rate of 4 per order.") and adds no analysis of sales or note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a simple flat rate of 4 per order.
The twin · paraphrase 1 · run 3
- Scorer one: pass. The answer gives only the rewrite, "Delivery is now a simple flat rate of 4 per order." It adds no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
“Delivery is now a simple flat rate of 4 per order.”
The twin · paraphrase 2 · run 3
- Scorer one: pass. The answer is only the rewrite, "Shipping is now a simple flat fee of 4 on every order." It has no analysis of sales and no note on what would prove something wrong.
- Scorer two: did not mark this answer. Outside Windows, it marked one in five.
The answer:
Shipping is now a simple flat fee of 4 on every order.