Some of the most important decisions coaches make during competition include deciding which quizzer starts the game on the bench and when to put them in the game. Sure, at times their teammates help make that decision by erring- or quizzing-out. But what if you are trying to get that 3rd- or 4th-quizzer bonus? What if your Memory Verse quizzer has 2 errors with 10 questions and 1 more memory verse question to go? Do you take them out now or wait until the next memory verse question is asked?
Being able to predict the next few question types – or knowing how much longer until the next Memory Verse question will be asked – allows you to make strategic choices to put your team in the best possible position for success.
Impossible, you say? What if I told you there are patterns inherit in the algorithm that generates competition questions? For certain conditions, it is not the totally random selection process as I had believed.
Being able to predict the next few question types – or knowing how much longer until the next Memory Verse question will be asked – allows you to make strategic choices to put your team in the best possible position for success.
Impossible, you say? What if I told you there are patterns inherit in the algorithm that generates competition questions? For certain conditions, it is not the totally random selection process as I had believed.
After researching this topic using the entirety of the scripture material, I found that some of the results align with beliefs held by the majority of the quizzing community. Other analyses revealed new, surprising details. And one finding in particular debunked a theory most of us were convinced to be true.
Here are the answers to the 6 questions I asked to better understand how the question set generator works:
Here are the answers to the 6 questions I asked to better understand how the question set generator works:
DO QUESTIONS REPEAT AT A QUIZ MEET?
As it turns out, the answer is “it depends”.
If all of the question sets for the event were created as a single file download, then the first 150 sets or so contain very few – if any – questions that repeat. Quiz directors will do this to ensure all of the quizzers in all of the rooms hear the same questions over the course of the event.
However, if sets of competition questions are generated separately for each room then questions are repeated between the rooms. This happens because the questions chosen for Room 1 are put back in the question pool used to generate questions for Room 2. While Room 1 will not likely ask any question twice, Room 1 may ask questions asked in Room 2 (and vice versa).
In my study, 14% of the questions between 2 file downloads containing 10 competition sets each are repeats. With 5 files (or Rooms) of 10 competition sets each, 33% of all questions are repeats (or about 6 per game). Just keep in mind that your team will not hear all of these questions that get repeated because they will be competing in 1 room while the other rooms are using different questions. As you change rooms throughout the event, if each room is using a separately-generated batch of questions you can expect to hear 1-2 questions per game that were asked in one of your previous games.
So, if you have a bye round cheer on a fellow team and pay attention to the questions – you may hear some of them again!
As it turns out, the answer is “it depends”.
If all of the question sets for the event were created as a single file download, then the first 150 sets or so contain very few – if any – questions that repeat. Quiz directors will do this to ensure all of the quizzers in all of the rooms hear the same questions over the course of the event.
However, if sets of competition questions are generated separately for each room then questions are repeated between the rooms. This happens because the questions chosen for Room 1 are put back in the question pool used to generate questions for Room 2. While Room 1 will not likely ask any question twice, Room 1 may ask questions asked in Room 2 (and vice versa).
In my study, 14% of the questions between 2 file downloads containing 10 competition sets each are repeats. With 5 files (or Rooms) of 10 competition sets each, 33% of all questions are repeats (or about 6 per game). Just keep in mind that your team will not hear all of these questions that get repeated because they will be competing in 1 room while the other rooms are using different questions. As you change rooms throughout the event, if each room is using a separately-generated batch of questions you can expect to hear 1-2 questions per game that were asked in one of your previous games.
So, if you have a bye round cheer on a fellow team and pay attention to the questions – you may hear some of them again!
BY WHEN WILL THE FIRST ‘ACCORDING TO’ OR MEMORY VERSE QUESTION APPEAR IN A GAME?
Interestingly, Memory Verse questions rarely occur on the first question. If it was statistically random, 15% of the games should start with a Memory Verse (3 memory verses divided by 20 available questions). Instead, Memory Verses only started 3% of the games. As the game progresses, the odds increase for the appearance of the first Memory Verse question. After question #5, there is about an 80% chance that question #6 will be the first Memory Verse. In none of the games I studied did the first Memory Verse question show up after question #10.
Conversely, According To questions represent 30% of all #1 questions (instead of the expected 20%). The first According To question of a game has a 78% of appearing on or before question #5.
Interestingly, Memory Verse questions rarely occur on the first question. If it was statistically random, 15% of the games should start with a Memory Verse (3 memory verses divided by 20 available questions). Instead, Memory Verses only started 3% of the games. As the game progresses, the odds increase for the appearance of the first Memory Verse question. After question #5, there is about an 80% chance that question #6 will be the first Memory Verse. In none of the games I studied did the first Memory Verse question show up after question #10.
Conversely, According To questions represent 30% of all #1 questions (instead of the expected 20%). The first According To question of a game has a 78% of appearing on or before question #5.
 When the 1st Memory Verse question appears in a game (Click to enlarge) |  When the 1st According To question appears in a game (Click to enlarge) |
Questions 1 & 20. (Click to enlarge) BONUS STATISTICAL INSIGHT #1:
Memory Verses, Book & Chapter, and Context questions are strangely under-represented on questions 1 and 20. Thus, there is a 96% chance that Question 1 is either an According To or General question and a 91% chance that Question 20 is an According To or General question.
Memory Verses, Book & Chapter, and Context questions are strangely under-represented on questions 1 and 20. Thus, there is a 96% chance that Question 1 is either an According To or General question and a 91% chance that Question 20 is an According To or General question.
Questions 1-10 vs 11-20. (Click to enlarge) BONUS STATISTICAL INSIGHT #2:
Book & Chapter questions tend to occur in the first half of the game (61% of the time). Further, they appear between questions #5 and #9 more than half of the time.
Book & Chapter questions tend to occur in the first half of the game (61% of the time). Further, they appear between questions #5 and #9 more than half of the time.
HOW LONG UNTIL THE NEXT ACCORDING TO OR MEMORY VERSE QUESTION APPEARS IN A GAME?
During the game, for the purposes of planning substitutions you may want to know when the next According To or Memory Verse question will appear.
In the totality of this study, there were absolutely no back-to-back Memory Verse questions. Nor did Memory Verses appear with 1 question between them. Instead, 2 questions is minimum and most frequent gap between Memory Verses. The average gap between Memory Verses is 4.3 questions and 90% of the time the gap between Memory Verse questions is 7 or less.
About once a season (or 1 out of every 100 games), a team may encounter a game where two According To questions are asked back-to-back. More likely, however, According To questions are separated by up to 12 questions with an average gap of 3.7 questions. 91% of the time the gap between According To questions is 7 or less.
During the game, for the purposes of planning substitutions you may want to know when the next According To or Memory Verse question will appear.
In the totality of this study, there were absolutely no back-to-back Memory Verse questions. Nor did Memory Verses appear with 1 question between them. Instead, 2 questions is minimum and most frequent gap between Memory Verses. The average gap between Memory Verses is 4.3 questions and 90% of the time the gap between Memory Verse questions is 7 or less.
About once a season (or 1 out of every 100 games), a team may encounter a game where two According To questions are asked back-to-back. More likely, however, According To questions are separated by up to 12 questions with an average gap of 3.7 questions. 91% of the time the gap between According To questions is 7 or less.
 Gaps between Memory Verse questions (Click to enlarge) |  Gaps between According To questions (Click to enlarge) |
WHAT TYPE OF QUESTION WILL BE ASKED NEXT IN A GAME?
For General questions, this one is just math. Simply divide the number of General questions remaining by the number of questions remaining in the game. If the previous question was a Memory Verse or According To question, the chances that the next question is a General Question increases.
The higher the current gap since the last Memory Verse or According To question, the more likely those types of questions will be asked next.
Here is an example:
What will Question 5 most likely be?
For General questions, this one is just math. Simply divide the number of General questions remaining by the number of questions remaining in the game. If the previous question was a Memory Verse or According To question, the chances that the next question is a General Question increases.
The higher the current gap since the last Memory Verse or According To question, the more likely those types of questions will be asked next.
Here is an example:
- Question 1 was a General question.
- Question 2 was an According To question
- Question 3 was a General question
- Question 4 was an General question
What will Question 5 most likely be?
- Memory Verse = 69% probability (from the previous “First MV Question” chart)
- General = 50% probability (8 General questions remaining/16 questions available)
- According To = 41% probability (from the “AccTo Gap” chart)
- B&C = 30% probability (from the previous “Book & Chapter Location” chart)
- X = 21% probability (from the “Context Location” chart)
Screen shot from "Naz Quiz Scoresheet" header. (Click to enlarge)
DO SOME QUESTIONS GET REPEATED MORE THAN OTHERS?
No, the question set generator does not show favoritism. In repeated trials, the questions that get repeated between the files vary. While this is good news, it does in effect debunk this persistent myth believed by quizzers and coaches alike.
ARE CHAPTERS EQUALLY REPRESENTED IN EACH GAME?
No, the question set generator does not show favoritism. In repeated trials, the questions that get repeated between the files vary. While this is good news, it does in effect debunk this persistent myth believed by quizzers and coaches alike.
ARE CHAPTERS EQUALLY REPRESENTED IN EACH GAME?
Chapter representation per game (Click to enlarge) Yes, within a game no chapter was represented more than once when the algorithm is selecting from 20 or more chapters. But when you look at all of the question sets in a single file download, it appears the question set generator favors the chapters that contain more verses when it has to choose a chapter to leave out. The less verses a chapter has, the less that chapter is represented overall:
CONCLUSION
Having completed this study, I am pleased that the data indicates the question set generator is working as intended to “balance” the game: Questions within a file of question sets do not tend to repeat, According To and Memory Verse questions are purposely spread throughout the game, and specific questions do not appear more than others when multiple files are generated.
As a Quiz Director, I now know how to minimize the number of repeat questions during a tournament.
As a Coach, the timing and effectiveness of my player substitutions will be greatly improved now that I have a better feel regarding what types of questions will be asked soon.
Having completed this study, I am pleased that the data indicates the question set generator is working as intended to “balance” the game: Questions within a file of question sets do not tend to repeat, According To and Memory Verse questions are purposely spread throughout the game, and specific questions do not appear more than others when multiple files are generated.
As a Quiz Director, I now know how to minimize the number of repeat questions during a tournament.
As a Coach, the timing and effectiveness of my player substitutions will be greatly improved now that I have a better feel regarding what types of questions will be asked soon.
Want to give it a try? Please do and let me know what you think, if it was useful, or if you have any questions.
Naz Quiz Scoresheet for iOS Numbers (2-team quizzing)
Naz Quiz Scoresheet for iOS Numbers (3-team quizzing)
On an iPad, after the link opens choose the export button (the button to the right of the address bar) and select "Copy to Numbers". If you have trouble, try the Excel link below and export that to Numbers.)
Naz Quiz Scoresheet for MS Excel (2-team quizzing)
Naz Quiz Scoresheet for MS Excel (3-team quizzing)
Naz Quiz Scoresheet for iOS Numbers (2-team quizzing)
Naz Quiz Scoresheet for iOS Numbers (3-team quizzing)
On an iPad, after the link opens choose the export button (the button to the right of the address bar) and select "Copy to Numbers". If you have trouble, try the Excel link below and export that to Numbers.)
Naz Quiz Scoresheet for MS Excel (2-team quizzing)
Naz Quiz Scoresheet for MS Excel (3-team quizzing)
RSS Feed