PECOTA captures your 2021 White Sox dreams and nightmares

After the White Sox shut out the Reds 5-0 on Sept. 19, they led the AL Central by three games with eight to play. Here’s a look at the divisional standings afterward:

  1. White Sox, 34-18
  2. Twins, 32-22
  3. Indians, 28-24

Then the White Sox wrapped up the season by losing seven of those eight games, including a four-game sweep by the Indians that tilted the season series and tiebreaker in Cleveland’s favor.

  1. Twins, 36-34
  2. Indians, 35-25
  3. White Sox, 35-25

On one hand, the White Sox handily beat their 60-game PECOTA projection of 31-29. Yet they couldn’t outrun the estimation of the divisional order, which had the Sox finishing in third. Perhaps the system couldn’t predict Rick Renteria’s team melting down and losing composure, but one could argue that’s one way a talent shortage manifests itself.

Anyway, I bring that up because PECOTA projected the White Sox for a third-place finish yet again, with a marginal improvement over their original 162-game projections in 2020.

Team20202021
Twins93-6991-71
Indians86-7686-76
White Sox83-7983-79

The White Sox are improved, but not outside of a rounding error in the win column (82.5 to 83.1 wins). Surprised? I am. But rather than focusing on one number — even if it’s the result of 1,000 numbers — I find PECOTA’s accompanying win curves more illuminating. Sure enough, a map of the White Sox’s simulated results shows how the big picture has them running in place, because the individual simulations have them running all over the place.

I’ve circled two areas. The front shoulder is much healthier than the one from last year’s simulations, which dropped off a table well before the Twins started peaking. Here they are side-by-side for comparison.

That area captures all the ways the tumblers can click into place and turn the season into something special. The White Sox’s 50th-percentile individual projections have a lot of underwhelming forecasts that can be beat. A few examples:

  • Yoán Moncada hitting .243/.334/.426, with the algorithm understandably missing on the unique challenges posed by COVID-19.
  • Tim Anderson hitting .271/.306/.438, which is below his career line despite a .331/.357/.514 line the last two years.
  • Andrew Vaughn being worth -0.6 WARP.
  • Lucas Giolito projecting as the 19th-best pitcher in baseball.
  • Dallas Keuchel barely being average (2.1 WARP).

BP has acknowledged the built-in volatility of this talent by centering an entire article on Michael Kopech’s wildly erratic projections, and he’s far from the only one on this roster who might swing from world-beater to world-weary in the same season. Maybe the same game.

The other circle covers the greater number of (under)achievable pessimistic outcomes. The curve’s initial ascent is steeper than the other AL Central teams, which shows there are a greater number of scenarios where a critical amount of Plan A’s fall through.

The White Sox have a lineup that’s generally walk-averse with inconsistent-at-best defensive showings, which makes them hard to individually project for stardom. Beyond the stylistic objections, Vaughn hasn’t competed against other teams above A-ball, and he’s getting the bulk of the DH plate appearances, with no clearly better options ahead or behind him. Michael Kopech is the only MLB-ready starter in reserve. One or two targeted injuries can send this whole roster calving into the sea, and the Sox might regret spending the most money on a closer with few games to save.

You could probably distill a lot of the systemic disagreement over how the White Sox built their roster down to settling for in-house solutions at DH, with Vaughn leading the way. PECOTA is the low man, but even the most optimistic projection on his FanGraphs page doesn’t paint a great picture.

  • PECOTA: .220/.289/.335
  • ZiPS: .235/.311/.374
  • THE BAT: .249/.321/.405

Plug that into a bat-only spot in the lineup, and it’s going to sink the readings. Imagine this dialogue:

WHITE SOX: We’re confident in Andrew Vaughn as our primary DH for most of the season.

PECOTA: Are you sure?

WHITE SOX: Yes.

PECOTA: Because you might not like–

WHITE SOX: Do it.

PECOTA: OK, but only because you said so.

[PECOTA PROJECTS 83 WINS FOR WHITE SOX]

WHITE SOX: I’d like to speak to the manager.

Two of those lines represent improvement over 2020’s Edwin Encarnación-led group, but then again, said Encarnación-led group was supposed to be an easy upgrade over the previous season. Instead, this happened:

  • 2019: .205/.285/.356
  • 2020: .148/.238/.350

A similar fate befell right field, where Nomar Mazara, despite all his flaws, couldn’t be worse than what the Sox trotted out in 2019. And indeed, he wasn’t …

  • 2019: .220/.277/.288
  • 2020: .230/.284/.324

… but it illustrates the gap between an improvement and an upgrade. Going from “brutal” to “bad” qualifies as the former, but doesn’t register as the latter on a meaningful scale. It left the White Sox seeking their third new right fielder in as many years, and they again didn’t set their sights much higher on paper.

It wouldn’t surprise me if Vaughn beat those projections handily, because his plate discipline is MLB-ready, and nobody really knows how to account for seasons spent at alternate training sites. It also wouldn’t surprise me if he struggled, because it’s unfair to expect immediate, quality MLB production from a guy with limited high-minors experience.

All in all, the White Sox have championship upside. Championship depth has yet to materialize. The PECOTA win number sounds low, but the curve helpfully illustrates why one chunk of the fan base is arguing with another. For me, it’s not too cynical to make the White Sox show their work, because they couldn’t quite outrun the system even over 60 games. Adding 102 more games ups the degree of difficulty, which is why I hoped the White Sox would make more meaningful additions to make everything a little easier on themselves.

Take a second to support Sox Machine on Patreon
54 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
metasox

Sox brass arguing among themselves why they are in 3rd when they were trying for 2nd.

I still wonder to what degree the Sox are sold on Vaughn vs talking themselves into it. And how DH’ing would affect his bat.

Last edited 3 years ago by metasox
MrStealYoBase

I wonder how you can look at projected standings that have Cleveland scoring just 10 fewer runs than the Sox and take them seriously.

MrStealYoBase

If a bookie gave me a 83-win over/under on the Sox, pretty much every free nickel I could find would get put on the over.

I get the argument: there’s a lot that can go wrong with this team. But a lot went wrong last year too. Below replacement for DH and RF. Moncada and Robert going on simultaneous month-long slumps. Multiple long-term injuries to the starting pitchers.

Still finished with the highest position player WAR and a top 7 offense. Even without McCann and without Abreu playing out of his mind, there is a pretty high floor with this group.

Eloytes

And there’s always the trade market if any position is absolutely hemorrhaging…assuming they care.

Eagle Bones

Early season trades are fairly rare. If Vaughn bombs or someone gets hurt early, they’re likely waiting multiple months if they want to trade for anyone of consequence.

Josh Nelson

Great point about Championship Upside and Championship Depth, Jim. Probably why for the Padres it wasn’t just good enough to get Blake Snell. They also got Yu Darvish and followed that up by acquiring Joe Musgrove.

We may not think much of it now, but the Mets signing Albert Almora and Jonathan Villar as bench players are better talents than many competitive teams 25th and 26th guys.

When I look at the American League, man, the rosters appear to be pretty thin compared to the NL contenders. I think Minnesota has good depth, but not to the level of LA Dodgers, Padres, or Mets. So I don’t think the White Sox are necessarily in the same boat, but I think this image sums it up:

HallofFrank

Brad Miller would help the Sox depth problem a lot. He can cover 1B, 2B, and 3B—and corner OF or SS in a pinch. And he’s a decent bat. 120+ in wRC+ over the last two seasons. He could either bridge to Vaughn or be Vaughn’s safety net.

asinwreck

This. The Brad Millers and Jonathan Viillars of the world will not be core pieces of a championship roster. They will, however, prove remarkably useful over a long season and give the GM the ability to optimize the postseason roster.

Valuing depth is yet one more contrast between the World Champion Los Angeles Dodgers and the White Sox. It’s not just a matter of payroll, though a higher payroll helps.

I am slightly less irate at the Sox’ weird philosophy now that Lucroy’s in the fold. Though he has a Best Shape of His Life narrative, his best days are far behind him. Even with those eroded skills, I feel much more comfortable with him as the backup catcher than just giving the job to Collins. Andrew Vaughn is a better player than Collins, but why not sign Miller or Adam Duvall or somebody to start the year at DH? If the veteran actually hits, it’s not a problem!

metasox

Isn’t Madrigal expected to start the season on the injured list? If so, a guy like Miller is depth the Sox could use on Day 1, even without considering spots like DH

Last edited 3 years ago by metasox
joewho112

Why? What’s wrong with Madrigal?

asinwreck

Madrigal’s post-season shoulder surgery was supposed to take a bite out of his spring training, but he’s expected to be ready for the start of the season.

ParisSox

Ha! this was the pitch to Machado. KW: “We offered more money. He just had to stay healthy.”

Last edited 3 years ago by ParisSox
vanillablue

Counting on Vaughn to be the primary DH really seems crazy. Let’s optimistically assume that his alternate site experience last summer was the equivalent of AA ball. How often are players able to be productive in the majors after a couple of months of AA?
“Hey Andrew, for us to contend, we’ll need you to be Juan Soto. No problem, right?”

HallofFrank

It’s fair to be concerned about Vaughn starting at DH, but the comparison of Vaughn’s case to normal prospect development needs to stop. At least recognize the disparities: Vaughn was an advanced, *elite* college bat, we have almost no idea what his development was like in 2021, and he’ll only be hitting. Again, no problems with concerns or wanting a different solution, but most of the criticisms of this move ignore the context of Vaughn’s situation.

And no one is asking him to be Juan Soto, but literally to just be decent at the one thing he’s always excelled relative to his peers: hitting.

Right Size Wrong Shape

I think we also need to stop acting like the guy is Babe Ruth. It’s not like he laid waste to the minor leagues his first year. I don’t think the list of guys going from okay in A ball to good major leaguer in one season is very long.

Last edited 3 years ago by Right Size Wrong Shape
HallofFrank

Who’s acting like that, though? Even adjusting for the obvious hyperbole, I don’t think I’ve seen a single poster on this site set unreasonably high expectations for Vaughn. For the crowd (like myself) who are pro-Vaughn starting at DH, the consensus seems to be: it’s not ideal; it’s been a weird year; Vaughn is talented and an advanced hitter; he’ll only have to hit; he’s got more upside than whatever temp they’d add in the meantime; and if it doesn’t work you can pivot midseason. I mean, I’ve even said he would probably hit 8th in the order all year.

Your final sentence shows the disconnect because you’re continuing to appeal to normal prospect development. You’re right, that list isn’t long. But show me list of healthy prospects who had to spend a year at an alternate site due to a pandemic and how they performed the next season.

The fact is: prospects aren’t on a normal timeline anymore. I can all but guarantee you that teams won’t operate like that, either. In fact, the Sox have already proved this: show me the list of players with 0 professional IP before they are put in the back of a bullpen for a playoff race.

Right Size Wrong Shape

Eloy Jimenez destroyed every minor league level and still sucked for half a season. I have no expectations that Vaughn will be good this year. I hope I’m wrong.

HallofFrank

Eloy arrived when he was younger than Vaughn and with scouting reports that he was less polished… and he still managed an .828 OPS and 116 wRC+. So if he sucks for half a season and produces those results, I’m in.

Right Size Wrong Shape

I don’t think it’s reasonable to expect Vaughn to be at the same level that Eloy was entering his rookie year. And what if he is terrible? Where do you send him? Winston-Salem?

HallofFrank

Sure, I wouldn’t say I *expect* Vaughn to be at the same level the Eloy was his rookie season. But it also wouldn’t at all surprise me if Vaughn bested Eloy’s 116 wRC+. If Vaughn struggles, send him to AAA.

I’m not sure what I expect from Vaughn. I grant there is a high variance in outcomes—higher than ideal for a team with playoff hopes. But, the drum I keep beating: fans need to (a) stop pretending like prospects are on a normal timeline and (b) stop appealing to Vaughn’s projections that simply cannot factor in Vaughn’s last 18 months. I’m not one for trusting this FO, but the fact is they are in a *much* better position than normal to asses Vaughn’s readiness compared to fans or projections. The pandemic just makes this an absolutely unprecedented situation. And there are good reasons for thinking that, unless you’re going to go out and spend $12m+ on a DH, Vaughn is the best available option—even if it’s less than ideal.

texag10

I don’t think that’s the expectation but Vaughn supposedly has more room for error given he is touted as having much better plate discipline.

dongutteridge

Thanks for this, Jim. It’s very interesting. Though, I have to say that even as down as I am on the Sox offseason, I’m quite confident that they will surpass 83 wins – especially playing in the AL Central.

They could still finish in 3rd but that’s pretty doubtful. My best guess is that they’ll finish in second and just miss the wild card.

Root Cause

They would have spent more if they really wanted 2nd place.

I think it was Dale Earnhardt’s quote that most accurately describes the WS brain trust.
“It doesn’t matter if they like you or hate you, just so long as they are talking about you.

Oddvark

I’m happy to see that Adam Eaton will get 567 plate appearances and will have the 3rd highest OBA on the team, behind only Nick Madrigal and Yasmani Grandal. So his signing won’t be the downfall of the team (at least it isn’t projected to be here).

And Carlos Rodon will pitch 82 innings of 4.59 ERA ball, and along with similar performances from Reynaldo Lopez (72 IP/4.56 ERA) and Michael Kopech (89 IP/4.56 ERA), will produce reasonable results from the 5th starter spot. True, it’s not as good as the 140 innings with a 3.88 ERA that the Angels will get from Quintana, but it’s OK for 5th starter depth.

Zach Collins’ numbers on the other hand look very bad (87 DRC+/-0.4 WARP). But he’s only in for 220 plate appearances, and replacing him with someone like Curt Casali (91 DRC+/0.6 WARP with the Giants) probably only gives the team 1 more win. Marginal wins matter, but that still leaves us in 3rd place.

The biggest mistake this offseason — if we’re looking to PECOTA to inform the analysis — is not signing a proven DH. Vaughn is projected to get most of the DH bats, and he’s expected to suck at it (73 DRC+/-0.8 WARP). He’ll have a lower DRC+ than literally any hitter on the Twins. Or the Indians. Or the Tigers (except that he’ll tie with their back-up catcher Grayson Greiner). Vaughn will have the lowest WARP of any batter on any team in the entire MLB. Those 60FV prospect rankings really don’t give you much.

HallofFrank

If we’re looking to PECOTA to inform the analysis, then the Sox would have almost certainly been better off trading Moncada and Anderson and looking for upgrades elsewhere.

lifelongjd

Agreed. They’ve never valued Anderson’s “lucky” stats, but he always seems to out perform projections. The Moncada prediction is surprising.

The PECOTA just doesn’t pass the eye test to me. Half of the team should get better with the extra year of experience and we made improvements to RF and added a solid #3 starter, which were two positions of weakness last season. Them projecting Cleveland and Minnesota in roughly the same manner as last season despite some significant changes makes me want to ignore the analysis altogether.

dwjm3

Zips projects Eaton to be replacement level in right. Models generally aren’t going to reward you for simply being less worse. Two fifths of our rotation is a complete question mark even with Lynn in the rotation. Furthermore, our DH spot is a complete unknown as well. The model seems with in the realm of possibility to me.

Given this organization’s track record I’m not sure why anyone would dismiss a model for being pessimistic. If anything pessimism should rule the day with a Reinsdorf organization.

lifelongjd

My overall point was these projections mirror what they were last year for the AL Central, despite the Sox making assumed improvements in areas ranked deficient last year. Plus Minnesota and Cleveland lost some major pieces and still remained nearly unaffected in the same analysis.

Eagle Bones

I’m not saying I agree with PECOTA, but you can’t just take their 2020 prorated win total and add the projected wins for the players they acquired and call it a day. They’re obviously projecting regression for several players. So yeah that’s certainly possible.

patrick

I mean, no one is using PECOTA to make player personnel decisions (I hope), but it does give you some idea of comparisons of variance in outcomes across players and teams. The Twins have less variance around their win totals because they have more players that can take in slack if 1-3 others hit a wall and they’re high WAR guys have a longer more consistent history of success (i.e. the priors around Sano are much tighter around Moncada since he’s been more consistent). But, it ignores things like depth in the minors, certain prospects playing above their projections, etc. To me that’s even more reason to make Vaughn prove he is a DH rather than have him be plan 1a. If he doesn’t work, then what? If I hear DH by committee again I’m going to barf. You can’t put Eloy there full time because you have no real outfield depth. And Zack Collins isn’t good at baseball.

So, the argument isn’t so much that I think Vaughn will be bad. My argument is that they should assume he is bad until he proves he’s good. Which is how other teams seem to handle their prospects when actively trying to win a World Series.

HallofFrank

I agree with your first paragraph entirely—my comment was more to point out that “looking to PECOTA to inform analysis” isn’t a great way to evaluate moves.

As to your second, I don’t necessarily disagree, but I (as I said above) wish people would stop comparing Vaughn to “how other teams seem to handle their prospects.” I’m confident that in 2021 teams won’t handle their prospects like they always have, because prospects simply are not on their normal timelines anymore. Any team that acts like they are will make mistakes.

Oddvark

I don’t think the subtle sarcasm in my original post came through.

PECOTA’s “prediction” of 83 wins no more proves/confirms that the offseason has been a complete failure than it’s “prediction” of 567 plate appearances for Adam Eaton and 82 innings of 4.59 ERA pitching from Carlos Rodon proves/confirms that those signings were good/decent. If you look at the details, they don’t obviously support the narrative about how bad the offseason moves have been.

And my noting, in the context of “if we’re looking to PECOTA to inform the analysis”, that they “predict” Vaughn to end up with the lowest WARP of any batter in all of baseball was supposed to suggest that we shouldn’t be relying on PECOTA too much at all. Regardless of whether you think Vaughn can/will be a contributor in 2021, it seems really far-fetched to me that he would provide that much negative value to the team (in part because even if he really struggles they can easily send him down to the minors since he’s not an established veteran on a multi-million dollar contract).

patrick

I think that’s definitely true, but in relations to that and @HallofFrank’s comment, no we shouldn’t treat all prospects like we would have if things were more normal. Sure, he may “break the mould” of other prospects due to his elite college profile. But that is still ignoring the highly variable outcomes possible for an offense-first position. If he doesn’t hit, he provides absolutely no value. And given the position, he really needs to hit, not just be serviceable, to be replacement-level. Good teams account for this variance by building depth. The Sox didn’t, consciously I might add. If you want ot win the WS, it isn’t just about making sure your median projection is above the rest of your division, but also making sure your variance is constrained. The Rays always seem to make up for this by building a roster full of high variance guys, with a significant depth of other high variance guys to sub in if some don’t work out. Vaughn is a high variance guy, but his replacement is a known commodity of bad or non-existent.

Oddvark

I don’t disagree with almost anything you say. Except with respect to potential replacements for Vaughn. Mercedes and Collins are also high variance because of their limited MLB exposure, but Mercedes definitely exists and has demonstrated hitting abilities at the lower levels.

And I just don’t think it’s fair to say that Collins is a known commodity of bad — I assume that’s whom you are referring to — especially if we’re talking as a potential DH and not his catching abilities. From 2016-2019 as he worked his way through the minors from high-A to AAA (not counting 3 games in Rookie ball), his wRC+ was consistently good at each and every level, with his worst performance being 128 in 2018 in AA and his most recent being 140 in 2019 in AAA. In the pros, he’s had limited and often sporadic opportunities, and the one time he was given more of an extended look in September 2019, he finished strong with a .380 OBP and 2 HR in his last 10 games. There is some cherry picking here, but there is enough for me to not conclude that he is a definitely “bad”. Unproven at this level and a high variance option, sure. Would a proven, stable option have been better — sure (though we know that doesn’t always work out either). Overall, the offseason choices with respect to DH are more than fair to criticize, but I don’t see the team as hopeless there, with a few options that have at least some realistic chance to be good.

Eagle Bones

It would be cool if we could get a look at the ZiPS outcome distributions for Vaughn, Collins and Mercedes. I obviously haven’t seen them myself, but my guess is people would be surprised by how likely it is that these guys will not be good. I hate to bug Dan, but in case he has a spare minute, Szymborski, Szymborski, Szymborski…

Last edited 3 years ago by Eagle Bones
oldtimer

I’m in the Vaughn will be OK camp. With the addition of Lucroy, I see him taking the 2nd/3rd catcher spot freeing up Collins to get in AB’s at DH while Vaughn starts at AAA. After Vaughn gets in 100 or so at bats (service time) in AAA getting his feet under him he will replace one of Collins or Lucroy. I’m obviously hoping Collins does well and Vaughn replaces Lucroy. Vaughn will have his ups and downs but I do see him taking over the DH and succeeding. Certainly not Juan Soto but who would argue with a rookie slash of 245/315/450. I’m an optimist (glass half full guy).

asinwreck

Old Friends in New Homes: Nate Jones heads to Atlanta and Steve Cishek isn’t done yet! He’s going to Houston.

MrStealYoBase

Steve Cishek escapes the crowded graveyard of veterans who bomb with the Sox and disappear from baseball. An impressive feat.

tommytwonines

Good luck, Nate Jones.

Good guy, hope he can stay healthy.

mikeyb

No backup catcher. A DH who hasn’t played above A Ball. A RF who hasn’t been healthy or good in the recent past. A CF who cratered down the stretch last year. A LF who can’t stay healthy because he can’t play LF. A 1B coming off a wild career year at the age of 33. A 2B who couldn’t hit the ball out of the infield who is now coming off shoulder surgery. A SS who has never seen a pitch he won’t swing at. A 3B coming off covid. A back end of the rotation full of guys we thought would be looking for new homes this year. A new manager who can’t be trusted not to end up in jail.

And it’s the most excited I’ve been heading into a season in a decade. There are plenty of holes that I wish had been filled. There are concerns about the leadership coming from the Owner/Manager. It’s highly likely I won’t even be able to attend a game this year due to covid. And the projections hate us. But dammit, this team is going to be fun. If PECOTA nails it and we end up finishing in 3rd place, I feel confident this will be the most fun 3rd place team I’ve ever watched. It’s an exciting group of young guys at key positions who seem to truly have fun playing the game. I cannot wait for this season to finally get started.

denny

Does anyone know just how often has PECOTA been correct and just how often have they missed? IIRC, they often miss….sometimes by a lot.

mikeyb

They’ve been dead on every year. They’ve always predicted that the average record across the league will be .500, and so far, they’ve hit that number right on the money.

HallofFrank

But that’s not a good way to evaluate PECOTA’s success. Projecting how many wins get distributed is a lot less impressive (and a lot less interesting) than projecting *how* they get distributed. In other words, fans don’t care whether they’ve correctly picked the average record, but how good they are at evaluating particular teams. By your metric, PECOTA could project 95 wins for the White Sox and 70 for the Cubs, but be “dead on” if the Cubs won 95 and the White Sox 70.

This is about as unscientific as it gets, but I did some quick back-of-the-napkin math for PECOTA’s success in 2018 & 2019 in the AL (those parameters because I didn’t care to spend more time looking into this, so maybe this isn’t representative). If PECOTA’s projection was inside a 5 game swing (4 or less) of being correct I called it a “hit” and if it was outside 10 games (11 or more) I called it a “miss.” If the projection was within 1 or 2, I called it a “critical hit” and outside 15 I called it a “critical miss.” The critical are counted in both columns.

2018-2019 AL PECOTA Projections (Critical Hits/Misses in Parenthesis)
Hits: 8 (2)
Misses: 9 (6)

That means their projection for the remainder of the teams in this window (13 teams) PECOTA’s projections were somewhere between 5-10 of the teams final win total.

I’m not sure what we can really glean anything from this, however. Projections are just difficult. I’d guess PECOTA outperforms Vegas and probably is up there with other projection systems, too. And this exercise highlighted (for me) that the critical misses are because of the inherently conservative nature of projections. PECOTA, like other projection systems, stay within a pretty confined space of 62-94 wins. It’s rare for a team to project outside of that even though teams fall outside of that window almost every season.

mikeyb

Ha I was just making a silly joke.

HallofFrank

I’ve been very bad at reading jokes today! Anyway, I was curious myself how PECOTA had done. I’m sure there’s a better analysis out there somewhere.

denny

What did PECOTA predict for 2005 World Champ Sox ?

Eagle Bones

It’s not really the purpose of projection systems to be “right”. They’re predicting the likelihood of a myriad of different outcomes (hence the win curve). If the Sox win 100 games, PECOTA wasn’t “wrong”. Nor would they be “right” if they win 83.

patrick

We’re approaching “the polls were wrong” territory

Right Size Wrong Shape

The way Jim evaluated it is absolutely correct, look at the range of probabilities and what factors lead to the results on both ends of the spectrum.

Last edited 3 years ago by Right Size Wrong Shape
HallofFrank

This seems like semantics. Maybe projections can’t be “right” or “wrong” but they can be “good” or “bad,” right?

Eagle Bones

Good and bad are probably the wrong words, but yeah some projection systems obviously perform better than others when measured against the actual outcomes they are designed to project.

HallofFrank

Are they? Supposing good in this context means “relatively successful in its aims” and bad its opposite, I’d say it’s entirely reasonable to label projection systems good or bad.
(to be clear, I’d say PECOTA is more good than bad as far as projection systems go, but I still think the labels apply)

patrick

Accuracy vs. precision

HallofFrank

Sure – again, semantics. It seems to me the question is: can we make qualitative judgments about projections or projection systems? Put another way: can projections be successful or unsuccessful? The answer seems to me: “obviously, yes.”