Question: How do to measure how well a manager manages?
Why I asked the question: There has always been a debate on how much of an impact a manger has on a team's performance and a game's and season's outcome. I wanted a thorough, yet relatively easy way to measure a manager usefulness. My main focus now with the initial release is to try to find any huge glaring errors so it won't have too many mid-season changes.
Analysis:
General overview
Finding a way to measure the many different aspects of a manager's job is close to impossible, but this my best shot. I am calling it, the Manager Scorecard and am hoping that others would join me (if not, oh well) We could start getting some measurable impact of mangers over an entire season, especially with the number of games the team wins.
The Scorecard will actually look at all of the team's managers. If the pitching coach leaves a pitcher in too long or a 3rd base coach sends someone home and they get thrown out, the entire staff will take the hit. I am not looking at any decisions that would be made by the general manager (trades, player call ups). The General Manager Scorecard is something on my to do list and should be ready by the beginning of this upcoming off season.
Finally, some of the data I plan on collecting can easily be mined from old game logs to determine how a manager performs. At the end of the season, I plan on comparing how a quick sample of data (IBB, CS/SB, Sacrifice Buts, pitcher abuse, etc) to the results from the Scorecard and see how the numbers correlate.
I was able to find some way to measure the manger's effect in following aspects of a game:
Theoretical runs gained or lost
Lineup utilization - This area is probably the most over analyzed while having the least impact on the outcome on the game.
IBB - Looking to see if managers intentionally walking someone is a good or bad decision.
Fielding – UZR/150 is going to be used to see if a player is obviously out of position.
CS/SB – Does the advantage of the extra base out weigh the disadvantage of causing an out?
Effectiveness of hit and run – How much this call helps or hurt in getting the batter safe and the runner safely advanced to another base?
Sacrifice bunts – Does the giving the out up, justify moving the running up another base?
Pinch hitter/pinch runner - Are they effectively utilized later in a game to generate more run scoring opportunities?
Bullpen management – Are pitchers used correctly as the game goes on? How long is the starting pitcher used compared to when they usually begin to break down? Also, is the bullpen used so that the pitchers face the batters when they have a distinct advantage?
Bullpen usage – Are the best relief pitchers being used in situations when the game is at the most critical moments?
Bullpen Fatigue - Looks to see if certain relievers get over used and their performance declines because of this over usage.
Starting Pitcher Abuse - Looking to see if some managers over use their starters and the effects on them later in their career. The up to date stats can easily be found at Baseball Prospectus website . Also the number of innings pitched will be tracked for the Verducci effect. The rule states that young pitchers that increase significantly the number of pitches thrown from one year, take a major step back the next year.
Finally, there are other aspects that a manager does that are not easy to measure such as team chemistry, player development and scouting. These might be added later as I can find a way to measure them.
Source of statistics:
I was wanting to limit the number of locations to visit to get player projections and the season's stats. I have it so the user only has to go to three web sites:
BaseballProspectus.com - stater abuse points
Baseball-Reference.com - starting pitching breakdown points
Fangraphs.com - everything else
It will take a little daily updating, but once the spreadsheet is filled out for the first game, it will be fairly easy to maintain. Also, game data should be drawn from any source possible, articles, TV, radio, MLB games, game box scores and public questioning in order to get all the facts from the game.
I am not for sure exactly how much of an effort it will take to keep a scorecard for an entire season, so I am only going to following two managers this season, Trey Hillman's of the Royals and Ron Gradenhire's of the Twins. I feel this first season, there will be several changes to the system due to unforeseen game situations and some aspects being incorrectly weighted, but I will need to begin the evaluation.
The initial scorecard is available for download from here: Link I will update the scorecard with more detailed instructions closer to the beginning of the season. Also, a website will be set up to display the various final scores of Hillman and Gardenhire for comparisons and other individuals, that want to, can post the individual game scores of the managers they track.
Basic Instructions for the Manager Scorecard
Important: Even tough I have have written some rules/guidelines, the user of the scorecard need to use good judgment when making filling it out. It might be ideal for a player to be in a lineup everyday for maximized run generation, but players need breaks too. An example would be if Trey Hillman plays Mike Jacobs at 1B for more than 10 games during the season. I can see a few games to give others a break, but his defense is so bad, he should only be used as a DH. The decisions are totally up to the users.
The entire Manager Scorecard is a spreadsheet with 5 main tabs (Seasons_Total Score, Starting_Pitcher_Abuse, Bullpen_Fatigue_and_Abuse,Roster_Projections and Additional_Comments) and two tabs for each game (Gameday_Sheet_XX_XX_XX and Gameday_Sheet_XX_XX_XX_copy) . Here are the instructions on what each sheet is looking for and how to set each one up.
Note – I use the “Calc” program from OpenOffice when creating the spreadsheet and save it out as an .xls. If there are any problems with it working in Excel or other spreadsheet programs please let me know and I will try to resolve the issue.
Tab 1: Seasons_Total_Score
This tab contains totals from the rest of the tabs for easy access. I am hoping to have each section weighed somewhat evenly (I can see some multiples being added as the season goes on). The only work that needs to be done on this sheet is to copy over the daily Bullpen Management and Runs Gained/Lost from the daily copy of the Gameday sheet (more on why a copy of each Gameday sheet is used in its section). The rest of the data should update automatically.
Tab 2: Starting_Pitcher_Abuse
The top half of the page checks for the Verducci effect. The number of innings pitched in the previous year in the major and minor leagues and the total number of innings pitched during the current season will be inserted into this page. For every five innings over the threshold, add one point to the Points of Abuse column. The threshold for the number of innings the pitcher is not allowed to go over is set to 30.
On the bottom half of the page, the numbers are inserted from the Baseball Prospectus web page on pitcher abuse. The data can be entered into the section by coping and pasting into the correct place or entering data by hand. The final column labeled Stress will be divided by 10 and they the totals will then be added up for the team.
Tab 3: Bullpen_Fatigue_and_Abuse
This has been the toughest to figure out and I have found very little information on it on the web. I basically have the numbers set up so that when a pitcher pitches for 3 days in a row, the manager will get “awarded” with an abuse point. For each day I am taking .7 times the previous days value and adding the pitches thrown that day to come up with that day's value. If the number exceeds 90, the pitcher gets an abuse point. If anyone knows a better method please let me know. All the relievers will need to be entered by hand and the number of pitchers per game will need to be copied over from each Gameday Sheet.
Tab 4: Roster_Projections
Initially filling out this tab will be the most time consuming event and it will need to be updated daily also. There are 3 distinct areas, Hitters, Bullpen and Starting Pitchers.
Hitters – A weighted average of the 4 prediction (25% Marcels 25% Chone 25% Bill James 25% Oliver) from Fangraphs.com is used. Feel free to use what ever projection system or combination you feel appropriate. For determining the players ability, the yearly predicted numbers will be added to the year-to-date numbers to get the player's current ability/stats.
Also, I am attributing the lifetime UZR/150 from FanGraphs.com for each position the player has ever played (I have recently come upon another method, besides life time numbers and need to test it first. If it is easy to use and understand. If it is easy, I will enter it in the final version).
Bullpen – using The weighted averages of the different available projection systems from FanGraphs.com is used to determine the pitchers FIP. I am using FIP as it is the best measure of pitching performance available from projection data.
Starting Pitchers – In this area, the lifetime splits from baseball-reference.com are entered to see if there is point during the game when the pitcher really breaks down. The three areas that can be measured are:
Inning
Total pitches thrown
Time through the lineup
These breaks aren't that precise, but as of right now it is the best available information I can find.
Finally, I am really torn here on including all available player information, while still making it easy to do a assessment of each game in under 5 minutes. As of right now, I am including just the player's projected line and calculating their OBP, SLG and FIP.
Tab 5. Additional_Comments
This tab contains the factors that aren't measurable, but might have a major effect on the team's performance (e.g. kicked out of game, called out player/team in public, etc). If there are enough of these comments/factors not checked in other categories, they could be added together in a new area or they could give a point total for each event.
Tab 6: Gameday_Sheet_XX_XX_XX (where XX_XX_XX is the date)
Areas where run expectancy can change
Lineup – All batters will need to be imported and put in their actual lineup spot. Then using a lineup analysis tool, such as this one at Baseball Musing , the run expectancy of the lineup is generate. The OBA and SLG will need to be adjusted depending on if they are facing RHP or LHP. The user needs to determine if the lineup could be improved in any way (e.g. different order of players or different players starting). If there is a better way to construct the lineup, put the run differential in to this category.
Defensive Placement – Use UZR/150 to determine if the correct people are positioned in the field. If there should be an adjustment, take the difference in UZR/150 and divide by 150 to get the runs lost for incorrect positioning. Make sure if you add a batter for his offensive ability, adjust for his defensive ability also.
Pinch hitter usage - To determine if a pinch runner should be used, subtract .074 from OBP and .042 from SLG along with the correct left or right hand splits. Compare the pinch hitter to players already in the lineup and see if he should insert as a batter late in the game. If the bench player is better than one in the game, insert the player in to the on-line lineup run predictor calculator to get the difference in runs after they are in the lineup. Take the difference and divide it by 38 PA/G (~ average PA per game over the last 3 years) to get the amount of runs lost.
Other - Any situations that arise in the season where any of the preceding situations don't apply.
Manager Game Time Decisions
The various situations that manager has control over that can be measured with a run value are:
Intentional Base on Balls
Sacrifice bunts
Stolen bases
Base running – The user might need to watch/ listen to the game see if it was the manager or the player's decision to take an extra base. If there wasn't a play at the base, it should not count towards or against the manager
Hit and run
When the situation happens, take the run expectancy before and after the event from the Fraphgraphs game log, and insert the difference. Late in the game the Win Expectancy might increase even though the Runs Expectancy didn't. These events should not be counted against the manager.
Pitching Staff Usage
Bullpen – This area is where to rank which relief pitchers to use verses right and left hand batters. The league average splits for FIPS are located in this area. There is also a chart from Between the Numbers that shows the situations when the team's ace reliever show be used. Besides the chart on the Ace reliever, the usage of which reliever to use when is not very precise. Previous usage, game score, match ups, etc will come into effect to determine who should. What the user should be looking for here is see if the obvious case when a pitcher should not be in the game. To measure incorrect usage, the number of batters faced incorrectly with be counted..
Also in this area is where the number of pitches are tracked to help measure the each reliever's fatigue. I used the “pitch one inning for 2 days, one day off, repeat and you won't be getting fatigued” mantra as my measure stick. Also, a pitcher that pitches 3 days in a row, but to only 1 batter each time won't also get fatigued. I assumed it takes the average 24 (12 full speed and 36 not at full speed – divided 36 by 12) pitches to warm up from reading various articles. If it is determined by observation that each individual pitcher uses a different number of pitches, feel free to use the actual number the pitcher uses. Once the number of pitches are calculated, they will be link to Tab 3 (Bullpen_Fatigue_and_Abuse).
Starting Pitcher – This area shows when the starting pitcher begins to fatigue. These values are taken from the Roster_Projections tab and are the time when a pitcher breaks down. Once 2 of the 3 situations occur, each batter the pitcher pitches beyond that point is added up and added to count of batter not faced in optimal conditions. A batter can only be incorrectly counted once, not once for each the bullpen and the starter.
Tab 7: Gameday_Sheet_XX_XX_XX_copy
Make a copy of the each Gameday_Sheet before starting a new and paste it into this sheet using Paste Special (in OpenOffice Calc). The regular Gameday_Sheet has values that are linked to other parts of the scorecard and these values on the Gameday_Sheet will change due to other parts of the Scorecard changing. This copy is created to preserve the data exactly as it was on the day it happened.
Additional comments
Here are some Possible other additions to the scorecard. Currently, I believe they take up too much time for not extra information, but that could always change:
Look at LH and RH splits greater than or less than the league average
Other splits - power/finesse/average, fly ball vs ground ball,
Rotation management – rest between starts for starting pitching
As always, I will gladly accept suggestions (too much data, too little data, too hard to implement,etc) on ways to add to or improve the Manager's Scorecard.
No comments:
Post a Comment