August 25, 2018, 8:40 am
The only way to truly refine a process is to be extraordinarily thorough when looking at the results.
That’s been a cornerstone of the Bruski 150 since I started doing it and it’s always one of my most anticipated days of preparation for the next season.
Editor’s Note: You can get the Draft Guide, including free early access to the Bruski 150 on October 1, right here. You can also get the Bruski 150 a la carte, without the Draft Guide, right here. Or, if you like saving money on a season-long bundle, we got your Season Pass right here. And if you like FREE, we got the whole thing FREE right here.
It’s a truly grueling and somewhat inane exercise. After a three-month break of not looking at the results, I take my rankings and I compare them to what happened and then to my old friends at Rotoworld.
If I’m beating them, I figure we’re on the right path.
The good news is that once again this year we did just that.
The methodology for figuring out how we did ranges from simple (heads up who had the better prediction) to complex (how much impact did a decision have and how does one weight that).
Simply speaking, in 8-cat leagues we were better on 56 percent of our top-200 plays compared to Rotoworld at 44 percent (81-61). In 9-cat formats, we did even better with 60 percent wins against Rotoworld on our top-200 (87-58). Both of us crushed ADP.
If we were being bet on, the 9-cat rankings would put the Bruski 150 odds at (-150) and the RW odds at (+150).
Anytime one can have that type of an advantage against one of the best in the industry, it’s a great year.
Trying to take a deeper look (nerd alert), I created a methodology for determining how good or bad a recommendation was by assigning it a scale of impact (1-5).
I also used color schemes to add another concurrent element of the differentiation in recommendations on a given higher-impact prediction.
The color schemes were:
• Dark Green (massive win, easily had opportunity to draft a player relative to ADP)
• Green (solid win, likely to have had opportunity to draft a player relative to ADP)
• Blue (Just means the prediction was better)
• Yellow (painful loss, prediction put owner in likely position to move the needle backward)
• Red (brutal loss, prediction put owner in likely position to move needle back at significant level)
All the while I’m using a common sense approach to evaluating picks. If there was an uncontrollable event not tied to obvious injury risk, such as Gordon Hayward’s broken leg, then there might not even be an evaluation of predictions.
As we go further down in the draft, when player values start to bunch up, the impacts won’t be as great and we won’t be as concerned with a 1-2 rank difference. At the same time a sleeper that can crawl up into early round value will have tremendous impact.
If a player got extremely lucky due to unforeseen injuries ahead of him, we’re not trying to reward or punish predictions as much as we would a prediction that’s based on known variables — one that reflects greater understanding of stat sets, usage rates and the like.
The key to this, for me, is to be brutally harsh with myself and give my competition benefit of the doubt when evaluating these predictions.
Still, there’s a fine line between going over one’s work to make sure we’re constantly improving, and being obsessive over results analysis that’s taking away from my ability to research something else.
It’s entirely possible I have screwed up on a piece of logic in an example in an attempt to be expedient. I’m pretty sure any shifting results will be within a reasonable margin of error and not take away from the findings.
For the new impact analysis, I didn’t even measure the 9-cat predictions because the 8-cat analysis tallied up below was already way too tedious of a task. I assume that since we performed better in 9-cat that we wouldn’t glean anything new.
If you see anything hugely off, just let me know and I’ll make adjustments, but I doubt it’s going to matter.
For what it’s worth, there is a much more detailed results analysis that’ll never see the light of day because it’s too proprietary and way too inane to write about.
Again, looking at it simply, the biggest measure of how the B150 is doing in relation to a great site in Rotoworld, or against ADP, is simply measuring how much the list is winning head-to-head on the predictions themselves.
That evaluation is simply ‘did my pick beat their pick’ and was I able to be in position for a gain, avoid a loss and to what degree.
As for the new wrinkle in my analysis this year, the impact analysis is very qualitative.
I use the aforementioned color scale and then measure impacts from 1-5. You’ll see a column on the spreadsheet that identifies impacts. Depending on how a prediction did (good or bad), the spread on a prediction can be really big, and in the case of Kawhi Leonard it really stands out.
I got crushed by Kawhi this year. I’ll wear the L and those are the breaks when you’re trying to go after distressed assets. In the end, I should’ve recognized that the juice wasn’t worth the squeeze, or more specifically the goal (first round value) wasn’t worth the second round price considering the risk.
Of course, we didn’t know that there would be an unprecedented injury/holdout situation, depending on whose doctors one believes. That situation ended up making Leonard the worst fantasy pick of all time (send your entries if you disagree).
Owners had to hold him virtually all season because of a hypothetical return. So not only did you get screwed on total value but you also couldn’t replace him with a player on the wire.
How much should something like that either help or hurt a predictor in the ratings? After all, it’s only one prediction out of over 200.
That’s where the impact analysis tries to create a methodology for understanding how impactful the predictions are.
So I created a simple integer system associated with each of the aforementioned colors:
• Dark Green (+6)
• Green (+4)
• Blue (+2)
• Yellow (-4)
• Red (-6)
I can pick a million holes in this system but what it’s essentially saying is a good or bad decision on these impactful players can be worth 2-4 or even 60 times more than (Kawhi) what your run of the mill ‘push’ on a player prediction is.
Most big, impactful predictions in which one site is really high on a guy and the other site is low – and something good or bad happens that is really impactful — the kind that puts all of your readers on one side of the line vs. the other … those are checking in at 10-40 times more impactful than a ‘push.’
The high-end of the scale is rare.
Think Joe Ingles, Taurean Prince, Victor Oladipo, Myles Turner, Josh Richardson, Will Barton and Donovan Mitchell rare.
Many teams are comprised of 13-15 players and leagues have anywhere between 150-200 players. What this impact scale is doing is saying a prediction with an impact rating of 3-4 is 10-40 times more impactful than being one slot ahead on Nikola Vucevic when he finished right where both sites predicted. The latter ‘push’ isn’t helping, nor hindering you in your climb up the ladder.
But getting Ingles in the later rounds — that bought you about a half-draft of value. You’d be doing great if six of your picks each etched away a round’s worth of value in your favor, but you got a guy that did that in one swoop.
Again, I mostly wanted to understand if the big needle movers were going in my favor.
Because the colors were often influenced by the reality of a prediction situation, there are cases when a color rating has been upgraded or downgraded to better reflect that tension.
To counter that and give more credit for more impactful predictions, which is another way of saying ‘who cares about the qualitative adjustments, the results are the results,’ we basically use the size of the win or loss both in the impact rating and also in the color rating.
In the end, it’s the players that are hopping entire segments that are moving the needle in wins and losses. That would be your late or last-round guy returning early round value or vice-versa. Those are your 4s (Kawhi was the only 5).
Guys that are hopping or costing 3-4 rounds or more as we get into the middle rounds are your 3s, and players that moved the needle for a few rounds are 2s.
Everywhere else it’s assumed that if a side won or lost a prediction that it has an impact of 1 and weren’t included in the final tally. Because our predictions were 56 or 60 percent more accurate, I felt it would be creating more noise than I wanted and I wanted to isolate the needle movers in this discussion.
It’s assumed that everybody understands that just because you ranked a guy highly doesn’t mean you’re drafting him way ahead of ADP. That’s where the color scale comes in and assesses the likelihood that a prediction resulted in drafting a player for the associated loss/gain.
And that’s starting to get at the core of these ratings. Not all prediction wins are created equally. Some are dumb luck and have massive impact, which isn’t the sign of a good prediction, and other great predictions have smaller impacts but deserve more credit.
The equation takes the differentiation of predictions on the color scale, assigns that a numeric value, multiplies it by the impact rating, and does that for every prediction that had an impact of 2 or higher.
So as an example, if a dark green prediction was made by one predictor (massive win, easily had opportunity to draft a player relative to ADP), and the other predictor had a yellow prediction (painful loss, prediction put owner in likely position to move the needle backward), we’re using the following equation:
Absolute Value of Dark Green (+6) + Absolute Value of Yellow (-4) = Color Impact
Color Impact * Impact Rating (positive/negative) = Total Impact for the Prediction
Again – gibberish!
But I can assure you that having Kawhi on squads virtually killed them. I know because I had squads scoring in the 50s and 60s (in 8-cat, 12-team Roto formats) that had a ton of huge wins but could never get over the hump.
If the Hayward injury chopped a leg out from under the chair on Day 1, the Kawhi pick chopped one leg out and slowly sawed at another.
If his rating was a -60, which is appropriately high since it’s the worst fantasy pick of all time, then Rotoworld getting aggressive on Mike Conley, a known injury risk playing the first years of a massive deal for a team that’s tanking, is only impacting a team to the tune of -12 (4 for the spread of the bad prediction compared to mine times 3 for the overall impact rating).
To me that makes sense. Again, most impactful ratings including large differentiation between Rotoworld and the B150 – where readers’ squads showed that differentiation the most – those predictions are measured somewhere between 10-30 times more impactful than your everyday ‘push.’
See if you agree with the color ranks, the impact ratings and even the overall count.
In the end it looks like my predictions carried about 400 more rating points (Total Impact) than Rotoworld’s and we both crushed ADP.
I’ll do my best to detail those big wins and losses in a player comments section below, and then you can look through this doc to see how things stacked up for yourself.
If you want to try and duplicate or double-check the impact ratings, by all means be my guest but again, for me I just wanted to confirm what my gut felt like – and that was that we had a ton of strong wins that moved the needle and we avoided a bunch of losses in the process.
Also, for a link to last year’s B150 you can click here.
PLAYER COMMENTARY (also, see notes in the doc)
Stephen Curry’s ankle was a bummer as he was doing fairly well for me as my No. 1 ranked player. Anthony Davis would eventually run away with the top slot behind his own good health and Boogie’s injury. Karl-Anthony Towns worked out for RW. LeBron decided to have a ‘remember me’ season … Paul George didn’t slip at all … Myles Turner was brutal on a team that otherwise saw a ton of success in fantasy leagues … Kemba Walker and Kyle Lowry weren’t sexy picks but they got the job done … DeMarcus Cousins was gobbling up all of the numbers before he got hurt … Rudy Gobert was a predictable overdraft as his explosion is fading … Kristaps Porzingis was not … he was running well before the knee injury, which was a freak thing … Eric Bledsoe held on … Paul Millsap looked extremely old out there … I really wanted to fade him more than I did but I got caught up on the stat set upside … Khris Middleton was a beast … so was Bradley Beal … Otto Porter was on quite a few championship squads … Victor Oladipo’s season might have been the most impressive of anybody’s … unreal stat set enhancement … Taurean Prince hit my early 40s prediction … Joe Ingles might have been the best prediction of the year … late, late-round prices for a top-40 guy and it was right there in front of everybody’s faces … Josh Richardson had an equally sizable profit but the Dion Waiters injury really helped him from getting snipped … still, Richardson was right up there at the top of the best predictions list … James Johnson was pretty brutal so hopefully your ADP was in the right spot. He and Richaun Holmes brought up the rear in the Hoop Ball Six, but Ingles and Prince helped carry it along with Kelly Olynyk, who brought back top 70-80 value with even lower ADP than Ingles. The last member of the crew, Justin Holiday, was also on pace for a big win until the Bulls decided to creatively tank. As for Holmes, I had a bad vibe when the Sixers brought on Colangelo guy Amir Johnson. Then Aron Baynes putting him out for three weeks essentially handed the minutes to Johnson and Holmes never recovered. Of course, with Colangelo we know how that organization was setup and Brett Brown moved Holmes away this summer so he could continue to bring in veterans. Now Holmes is in Phoenix and they’re terrible, but we’ll get to see if my assessment of his game holds water. He was available for a last round pick so owners didn’t lose much even if my ranking was way off. Gary Harris was a nice win for owners … D’Angelo Russell is not good at NBA basketball … Clint Capela was a big win for us … We edged out RW on the Jrue Holiday gain, which was really big, and we had the rights to Robert Covington, which is fair since we found him way back when he was a D-Leaguer on the rise. Nerlens Noel and Willy Hernangomez stung us, as did Jusuf Nurkic to some degree, not that we were extraordinarily high on any of them … we just got caught up in the risk/reward quotient and with Willy he just fell off the face of the planet right away … Nikola Mirotic should have been such a huge win for us, maybe even the biggest, but who gets their face punched out like that … Steven Adams was quietly a solid win for us … George Hill made me look stupid and I wasn’t even that high on him in the first place … just a dumb rank … glad we only got caught up in the Marquese Chriss waiver wire situation and not the whole thing like the RW guys did … ouch. Thadditude was Dan Besbris’ guy and he was outstanding this season. Greg Monroe landed on way too many of our teams and probably started eating depressively once he figured out he was traded to Phoenix. Blake Griffin was an easy fade that hurt so many predictors … yikes. Kent Bazemore was solid … Andre Drummond nearly doubled his free throw percentage and became a dribble handoff guy … one of the better (and more lucrative) stories of the fantasy season. Lou Williams just kept his foot on the gas all year and finally owners got to see what it would be like with him let totally loose. Carmelo Anthony predictors have nobody to blame but themselves … same with Jeremy Lin … and to a lesser degree Danilo Gallinari. The Avery Bradley prediction shows who is doing hand crafted work and who is not … his big value in Boston was the boards and one had to know that was going away next to Drummond. Enes Kanter was an easy low-end play that ended up having all sorts of upside … Jeremy Lamb finally hit for us … If Kawhi was the worst pick of all time then LaMarcus Aldridge was the luckiest pick of all time … Jamal Murray also got very lucky … throughout draft season he was part of a three-headed monster with Jameer Nelson and lottery pick Emmanuel Mudiay. Nelson asked to be waived and the Nuggets made a hard cut with Mudiay, leaving Murray as the only guy and then Millsap got hurt. Murray made his shooting stat set sing and the rest was history. Dennis Smith Jr. predictions were brutal this year – a lot of folks got hurt on that. Donovan Mitchell, on the other hand, was great in just about every area and I didn’t get in on any of that … ouch. Buddy Hield drafters got lucky and many of them dropped him after Joergerball took full effect. But because of his sweet shooting stat set he finished strong and was our best buy low recommendation of the year when folks were legitimately panicked. Spencer Dinwiddie was actually ranked on our list, which is why we had all of the Dinwiddie. Jaylen Brown couldn’t break out despite the first game season-ender for Gordon Hayward, but Jayson Tatum took full advantage and avoided the early rookie lull that would have made him fight his way into standard leagues. Kyle Kuzma and John Collins made it worth the Summer League hype, though it wasn’t always pretty. Boban never happened.