Last week, as part of our Outdated Answers project, we ran a test on Stack Overflow to see what would happen if we stopped pinning the accepted answer to the top of the list of answers. As expected, there were no negative repercussions when we made this change. So, as promised, today we are making this change permanent.
By default, answers are now sorted strictly by Votes (descending order by highest score), and the accepted answer’s order in the list is based on its score. If you prefer to sort by Active (descending order by answer’s created or edited timestamp) or Oldest (ascending order by answer’s created timestamp), the accepted answer is also unpinned.
Read on if you’re interested in details on what we learned from the experiment.
How we conducted the test
For questions where the highest scored answer was not the accepted answer and the user’s preference was the default Votes sort, 50 percent of users saw the accepted answer pinned to the top and the other 50% saw answers sorted by the highest score with the accepted answer unpinned. For both sorts, our success metric was the rate at which users either upvoted or copied.
Findings
We analyzed the data in a number of ways.
Users copied or voted on any answer. We looked at users who copied all or part of any answer, or users who took any voting action (upvote, downvote, etc.) on any answer. Our hypothesis was that there would be no statistically significant difference between the two sorts, but we found that when the accepted answer was unpinned there was a 4% increase in copying and voting. Success rate was 20.3% when the accepted answer was pinned, and 21.1% when it was unpinned. (In case you are wondering, upvotes far outnumber downvotes, so we weren’t terribly concerned that the two types of votes were mixed together in this analysis.)
Users copied the first answer. There was a whopping 61.6% increase in users copying from the top answer when the accepted answer was unpinned and the highest-scoring answer was first in the list of answers. Success rate was 6.9% when the accepted answer was pinned, and 11.2% when it was unpinned.
Users upvoted the first answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 90.5% increase in users upvoting the top answer when the highest-scored answer was shown first. Upvote rate was 0.5% when the accepted answer was pinned, and 1.0% when it was unpinned.
Users copied the accepted answer. There was a 43.6% decrease in users copying from the accepted answer when the highest-scored answer was shown first. Copy rate was 6.9% when the accepted answer was pinned, and 3.9% when it was unpinned.
Users upvoted the accepted answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 39% decrease in users upvoting the accepted answer when the highest-scored answer was shown first. Upvote rate was 0.5% when the accepted answer was pinned, and 0.3% when it was unpinned.
Users copied an answer that was neither the accepted nor highest scored answer. There was an 8% increase in users copying from an answer that was lower down in the list of answers when the highest scored answer was shown first. Copy rate was 5.4% when the accepted answer was pinned, and 5.8% when it was unpinned.
Users upvoted an answer that was neither the accepted nor highest scored answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 3.5% decrease in users upvoting an answer that was lower down in the list of answers when the highest scored answer was shown first. Upvote rate was 0.48% when the accepted answer was pinned, and 0.46% when it was unpinned.
Feedback
Please leave any bugs related to unpinning the accepted answer as answers below this post. We will monitor this post until Wednesday, September 15. Report any further issues after September 15 as new questions on Meta.