2.) Accuracy Testing For Season Simulation Settings (Sliders):
To confirm the accuracy and precision of my season sliders, I compared the statistics from select quarterbacks and running backs to those that were produced in the simulated season. The goal of this portion of my research is to confirm that Madden statistics are as accurate as possible to real life so that ensuing Madden Wins Above Replacement (MWAR) can retain high levels of credibility, and also so that they may be comparable to the Positional Wins Above Replacement Values conducted by Andrew Hughes, Cory Koedel, and Joshua Price. The tables below depict a couple examples of NFL player’s statistics over five seasons versus their statistics over five simulated seasons.
Once I was happy with the statistics that individual players were earning throughout the simulated season, I wanted to make sure that the simulations accurately represented the displacement of wins throughout the entire NFL. Since the primary focus of my project revolves around wins, it is important that the value of a win in the simulation is as close as possible to real life so that my MWAR values do not need to be adjusted in any fashion. In order to collect this data, I performed a two variable t-test in Matlab comparing the average number of wins from each NFL teams over the past five seasons to that of five season simulations in Madden. Although the NFL data is collected from 2011-2015 and the Madden data replays the 2015 season five times in a row, I believe that the results of this test will still be significant because the average wins in the NFL has remained extremely consistent over the past five season.
For this Matlab code, we hypothesize that the average number of wins in the NFL and Madden simulations are equal. If the result from Matlab states that h=0, then we can confidently affirm that the previous notion is true. If it returns h=1, however, then there is statistical evidence to reject the notion. Additionally,the p-value (or the probability in which are observing the notion to be true) must be below 0.05 in order to reject the null hypothesis.
| Null Hypothesis (what we are assuming) | h = | p-value |
| AWnfl = AWmadden | 0 | 0.9694 |
Since h=0 from the table above, we cannot reject the notion that the average number of wins in the NFL through five seasons is similar to five simulated seasons. Although a high p-value cannot entirely affirm that this notion is 100% true, it provides a very strong justification that the contrary is most likely untrue. Now that I have mathematically justified that Madden simulations are (in terms of individual player statistics and league wins), it is now time to collect a series of “control” data that will be used to compare with that of the replacement player.