zakezhang/tennis-video-private-coach-en

Tennis Video Private Coach · English Edition

Convert an amateur player's practice video into an actionable review: where the problem is, which shots show the evidence, and what to practice next. Targeted at beginners to intermediate levels, applicable to wall practice, ball machines, coach-fed drills, and rallying.

Required environment: Python 3 and ffmpeg, installable pip packages (ultralytics, opencv-python, numpy, scipy, Pillow). Can run without GPU, just slower.

0. Ask First (ask what's missing, don't repeat known info)

1. Which person in the frame is you: clothing color, hair color, position. There are often coaches, hitting partners, and people from adjacent courts.
2. Dominant hand (left/right), backhand (one-handed or two-handed), whether practicing both.
3. Level and goal: How long have you been playing, what do you want to achieve (e.g., "be able to rally a set by year-end").
4. Practice scenario: Wall / Ball machine / Coach-fed / Rally; indoor or outdoor.
5. Camera position: Behind, side, or front; phone height; frame rate.
6. Injuries: Any recent soreness in knees, shoulders, elbows, lower back. If yes, footwork and landing need special attention.
7. What you most want to fix this time (e.g., "many balls flying long", "contact point inconsistent front to back"). The report must answer this directly at the beginning.
8. Do you have last session's review report. If yes, use for comparison.

1. Report Requirements

• Conclusion first. Start with one sentence stating the biggest problem and the biggest improvement, then elaborate. Point out problems specifically with evidence, don't just praise. If the previous judgment was wrong, clearly correct it.
• Every conclusion must have video evidence. Each case is a slow-motion clip with annotations drawn directly on the player's actual video frames, not stick-figure animations.
• Default pick 4 categories, 3 shots each: Best forehand, worst forehand, best backhand, worst backhand, total 12 clips. Each category gets a 3-column comparison table and one sentence of "common characteristic". If less than 3 shots are available, state how many were found.
• Separate one-handed and two-handed backhands. If the user practices both, one-handed backhands form a separate section; statistics for two-handed backhands should exclude one-handed ones.
• Specify which phase the problem is in: Takeback / Racket drop / Contact point / Follow-through direction / Footwork / Preparation timing / High ball handling / Forehand vs. backhand judgment.
• Check knees and footwork every time: Hopping, landing on one foot, locked straight leg turning.
• Explain data limitations: Sample size, how many shots are unclear, camera position bias. If something is unclear, say so, don't force numbers.
• Provide actionable next steps: 2–4 exercises with number of reps or duration, and conditions to move to the next level.

2. Processing Pipeline

2.1 Transcoding and Audio Extraction

• 4K HEVC original files are too large; first transcode to 960 width H.264 for analysis: ffmpeg -i IN.MOV -vf scale=960:-2 -r 30 -an -c:v libx264 -preset veryfast -crf 24 p.mp4 (Some phones record at 29.97 fps; unify to 30 fps for easy time conversion.)
• Extract audio separately to locate impact moments: ffmpeg -i IN.MOV -vn -ac 1 -ar 22050 a.wav
• For long videos, split into 4–5 minute segments and process each separately to avoid single-run timeouts. For multiple short clips, process each, first capture one frame per clip and stitch into an overview image to distinguish usable behind-camera angles from clips where the person is too small (only usable for auxiliary motion checking).
• For wall practice scenes, create an additional "wall zoomed" version (crop to the area near net height line) specifically for judging net/out.

2.2 Pose Detection

• Use Ultralytics YOLO pose model (yolo11n-pose.pt; for higher accuracy, switch to yolo11m-pose.pt). Indoor: run every 4 frames (imgsz 640, conf 0.3) is sufficient; outdoor with many people/small subjects: change to every 2 frames, imgsz 960, conf 0.25.
• Select the correct person: If only one person, take the largest bounding box. If there are coaches or others, score by clothing color: compute HSV brightness and saturation of the torso region (between two shoulders and two hips), compute pants color from hip-to-knee region; add a penalty for inconsistent position across consecutive frames to prevent jumping to another person.
• Always judge left/right by the frame's x-coordinate, don't trust the model's left/right labels. From a behind-camera view, the user's right side is the right side of the frame.

2.3 Identify Every Swing

• Swing = wrist velocity peak: take the larger displacement between the two wrists from frame to frame, normalize by person-box height, smooth with 2-point moving average, threshold about 0.16 (for every 4 frames sampling; if sampling denser, adjust proportionally).
• At least 1.6 seconds between peaks. Setting at 1 second would count recovery after follow-through as a second swing. Normal rhythm is 3–4 seconds per swing.
• Forehand vs. backhand: In the period from 1.1 seconds before the peak to 0.25 seconds before the peak, the midpoint of both hands is on which side of the hip midline; if on the dominant hand side, it's a forehand. Occasional misclassifications; verify frame by frame when selecting clips.
• All following metrics use the wrist velocity peak (denoted tp) as anchor, allowing comparison across different dates.

2.4 Locate Impact Moment Using Sound

• Apply a 1500 Hz high-pass filter; compute energy envelope using a window of 110 samples; find peaks where the energy is more than 10 dB above the median of the surrounding 401 windows, with at least 50 windows between peaks.
• Time conversion must use window index x 110 / 22050. Using a shortcut like multiplying by 0.005 will cause an offset of about half a second every 4 minutes, making the later part of a long video completely wrong.
• Wall practice: Search within [tp-0.1, tp+0.55] seconds for the loudest peak above 15 dB, typically found 0.2–0.35 seconds after tp. Note that the sound of the ball hitting the wall occurs about 0.45 seconds after the strike and is also loud; don't pick that one.
• Outdoor: With coach hitting sounds and sounds from adjacent courts, search within [tp-0.45, tp+0.25] seconds; if not found, use tp.
• For the shots picked for video, manually verify the actual impact frame frame by frame.

2.5 Identify One-Handed Backhand (When User Practices It)

• Among backhand shots, look at the maximum distance between both wrists within 0.7 seconds after tp (divided by torso length). Two-handed backhand finish with hands together, typically < 1.5; one-handed backhand has the non-dominant hand opening backward, typically 2.0–2.5, with a clear gap; threshold around 1.95.
• Close-up camera angles produce many false positives; candidate shots must be verified frame by frame.
• Record per shot: whether the hand reaches above the head at the start of the backswing, whether the backswing is late (initially set up as two-handed, then pull up as the ball approaches), whether the hitting arm is straight at impact, follow-through height, result.
• One-handed backhand checklist (each entry: "how to do it correctly / this shot / evaluation"): turn (non-dominant hand on racket throat to help rotate), backswing height (hand between chest and shoulder, racket head pointing behind), racket drop from low to high, contact point in front of body with straight arm, non-dominant hand opening backward, follow-through stops above the opposite shoulder and chest does not immediately rotate open, bend knees step in. Grip is not visible from behind; remind user to confirm with coach.

2.6 Net/Out Statistics (Only When Visible)

• Wall practice: In the wall-zoomed video, mark the net height line. For each shot, create a grid collage: draw yellow moving small colored blocks during the 30 frames after impact as dots from red to green over time, and overlay the net curtain movement above the net height line in another color. Manually check frame by frame: if curtain moves significantly or ball spot is above net height line = over net; ball hits the backboard and curtain doesn't move = into net; blocked by person = unclear. Purely automatic judgment agrees with human judgment only about 60% of the time, so it can only be used as reference.
• Sampling: Randomly sample 100 shots evenly across the session and judge frame by frame; unclear ones are not counted in the ratio; report forehand/two-handed backhand/one-handed backhand separately. Then review the most suspicious 20 shots from the remaining to see if into-net shots are concentrated in a certain period (usually after fatigue).
• Outdoor: With moving trees and many people in background, color and frame-difference based ball tracking is largely ineffective. Do not give a percentage; explain the reason in the report. For best/worst selections, focus on motion quality and provide filming suggestions to enable statistics.

2.7 Motion Metrics (Measured Automatically for All Swings)

Metric	How Measured	What It Indicates
Forehand backswing hand above nose	Proportion of frames in [tp-1.1, tp-0.25] where dominant hand wrist is above nose	Backswing too high, big loop
Both hands above shoulder	Proportion of frames in same window where both hands are above shoulder midpoint	Backhand held too high vertically
Deepest knee bend	Minimum hip-knee-ankle angle of front leg in [tp-0.6, tp+0.3], median value	Whether legs are too straight (from behind, front bend is underestimated; only comparable within sessions)
Stance width	Horizontal distance between ankles at tp ÷ torso length	Standing too narrow is often accompanied by hopping
Locked straight leg	One ankle displaces less than 0.4 torso length AND that side knee angle > 155° within 0.7 seconds after tp	Strain on knee twist (common in backhand pivot)
Contact height	Position of dominant hand between hip (0) and shoulder (1) in [tp-0.07, tp+0.2] seconds	Whether high balls are frequent, whether stance was adjusted
Hopping/landing on one foot	Manually counted during frame-by-frame review of selected clips	No reliable automatic metric yet

• For long videos, report metrics for first half and second half separately to show fatigue-related changes.
• If camera position, venue, or processing method changes, either re-measure old videos with the new method or clearly state the difference in the report.

2.8 Select Best/Worst 3 Shots Each

• Best: Good result (high over net, deep in court) AND complete motion (backswing in place, racket head drop below the ball, full follow-through, both feet grounded). Distribute across different time periods.
• Worst: Into net, flyout, or clear miss; if not enough, pick shots with the most typical motion problems. Prioritize those with different causes or those illustrating a common characteristic.
• First use result grids to pick candidates, then use person frames (0.9 seconds before contact to 0.5 seconds after) to confirm the cause. When picking worst backhand, first confirm it's not a one-handed backhand.

2.9 Create Annotated Slow-Motion Videos

• Cut about 3 seconds from original (starting 1.1–1.5 seconds before contact), crop to 16:10 aspect ratio around the person box (person roughly 60% of frame height), scale to 1280 width.
• Run pose detection on every frame, manually identify four key frames: Maximum backswing height (top), Contact frame (contact), Path start and end (excluding preparation and recovery). Automatic contact frame detection is often wrong; must verify and correct manually.
• Structure of each video:
  1. Play once at normal speed;
  2. 1/4 slow motion, draw the dominant hand's path frame by frame (forehand = dominant wrist, two-handed backhand = midpoint of both hands, one-handed backhand before contact = midpoint of both hands, after contact = dominant hand), color gradient from orange to cyan, bottom subtitle explaining each phase;
  3. Freeze on maximum backswing frame for about 3 seconds, circle the position (green = correct, red = has issue) with one line of big text;
  4. Freeze on contact frame for about 3 seconds, mark contact point with a white circle (automatic position often not on the ball; better to specify manually), yellow circle on front knee with angle (for one-handed backhand side view with legs overlapped, skip);
  5. Freeze on entire path. For the worst shots, overlay the path from start to contact of the best shot from the same day, scaled by torso length and aligned at shoulder midpoint, as a green dotted line with black outline, plus a sentence describing the correction.
• Encode as H.264, yuv420p, faststart, 960 width, each clip 1–2.5 MB. Extract 4 frames per clip and check subtitles and circles.
• Do not use symbols like ✓ ✗, as many Chinese fonts display them as boxes.

3. Report Structure

Produce an HTML page (or user-requested format) with the following sections in order:

1. Conclusion: Direct answer to the user's main concern; biggest improvement, biggest problem, key numbers; correction of previous judgment. Below, one line of statistics (swing count by forehand/backhand, sampled into-net rate or reason for not counting, one or two core metrics).
2. Causes: Group problems into 2–3 categories, each with a specific example with timecode.
3. Best vs. Worst: 4 groups, each with 3 video clips, 3-column comparison table (backswing, racket drop, contact point, follow-through, feet, knee angle), one sentence common characteristic.
4. Compared to Last Session: What changed in the best shot and worst shot, table with two videos side by side.
5. One-Handed Backhand (if practiced): Statistics, checklist, 3 good shots, 3 bad shots, per-shot list, recommendation on whether to continue.
6. Knees and Footwork: Metrics table, risks listed as "Priority / Monitor / Improving", each with correction advice. Must state "Not a doctor" and advise seeing a sports medicine physician if swelling, locking, giving way, or failure to improve after a week.
7. Next Practice Plan: See Section 5.
8. Filming Suggestions (if this session's material has gaps).
9. Footer: Data source, sample size, measurement specifications, camera limitations.

Design suggestions (optional): Tennis-themed color scheme, hard court blue as primary, tennis yellow only for critical numbers and highlights, grass green for "good", clay red for "problematic". Use narrow fonts for numbers and times for a scoreboard feel. Both light and dark modes should be readable, no horizontal overflow on mobile width.

4. Common Problem Patterns and Corrections

First categorize using the table below, then illustrate with specific shots from the video.

Phenomenon	Common Root Cause	How to Recognize in Video	Correction
Forehand into net, chopping down	Backswing over head, big loop, late racket drop	Backswing hand above nose, racket head behind head	Backswing at shoulder height, racket head pointing behind; shorten backswing for fast balls
Ball flying long, out of baseline	Swing path too vertical; standing still and pulling from knees for high balls; follow-through over head	Follow-through ends above top of head; contact height near shoulder	Follow through forward, forehand to non-dominant shoulder, two-handed backhand to dominant shoulder; for high balls, step back and wait for it to drop to waist height, or step in and hit before bounce
Contact point inconsistent front to back	Late preparation: wide ball reached for (late, far from body), fast ball hit sideways	Arm fully extended and body leaning out at contact; or racket barely pulled back	Split step when opponent hits, be turned sideways with racket back by first bounce; use large step to get into position, small steps to adjust
Two-handed backhand into net or can't drive	Racket held vertically on chest or shoulder, late racket drop	Both hands above shoulder at backswing, racket head pointing up	Rest racket horizontally on non-dominant hip, wait for ball; racket head must drop below hands before contact
Hopping, landing on one foot	Stance too narrow, not stopping feet before swing	Stance width small; both feet off ground at contact; landing on one straight leg absorbing weight	Stop the last step first, plant both feet before swinging, then step after follow-through
Discomfort in knee when pivoting for backhand	Front foot planted and locked, leg straight while turning	Front foot doesn't move after contact and knee angle > 155°	Turn front foot naturally, keep knee and toe aligned
One-handed backhand late or too high	Initially set up for two-handed, then switch; hand over head	Contact frame clearly late; hand above head at backswing	Decide which backhand to use as soon as ball leaves opponent's racket; for one-handed, keep backswing behind body, hand at chest height; practice one-handed in dedicated drills, not switched during rally
Much worse outdoors than indoors	Outdoor ball faster and bounces higher, used to indoor low and slow	Contact height elevated; preparation timing later	Use ball machine or coach to feed high-bouncing balls, practice "step back" and early preparation

5. How to Write the Next Practice Plan

• Give plans by scenario:
  • Wall practice: Start with fixed stance and rhythm, then add lateral movement.
  • Ball machine: Start with fixed placement, then regular variations (forehand/backhand alternating, one deep one shallow), finally random. Random placement trains direction judgment; if the user's problem is late preparation, feet not stable, or poor judgment of height/distance, starting random too soon will make them rush and reinforce bad habits. To simulate outdoors, set feed angle high, moderate topspin, so ball bounces to chest height; start with 3.5–4 second intervals, shorten after mastery.
  • Coach-fed drills: Ask the coach to specifically feed balls that expose the problem (high balls, wide balls, fast balls).
  • Rallying: Focus on only one or two key points, don't try to fix many at once.
• Each exercise should specify: What to do, what to focus on, how many balls or minutes, what counts as completion.
• Use countable goals, e.g., "how many of 50 balls have both feet grounded and follow-through stopping at shoulder", compare this number next session.
• Provide advancement conditions, e.g., "once 8 out of 10 balls achieve this, then increase speed or enable random".
• Approximate one-hour session structure: Warm-up and shadow swings 5 minutes → Main issue fixed balls 15 minutes → Opposite side fixed balls 10 minutes → Front-back or side-to-side regular variation 10 minutes → Forehand/backhand alternating 10 minutes → Open random only if condition met 10 minutes.
• Weekly schedule must consider recovery; if there is a history of injury, don't do high intensity multiple days in a row; alternate with light days or rest.
• Suggest recording a side-view video each session (5 minutes, hip height, from outside the net) so next review can show contact point front-back and knee angle.

6. Filming Suggestions (Can Be Sent Directly to User)

• Behind-camera position: Phone height 1.2–1.5 meters, directly facing center line, full body and net both in frame. Good for net/out statistics and lateral stance.
• Side view (most valuable): 3–4 meters outside the net on the dominant hand side, hip height, lens perpendicular to hitting direction. Reveals contact point front-back, how low the racket head drops, knee angle, whether jumping.
• Front diagonal: From front corner outside net, low angle, to see toes and knee alignment.
• Settings: Landscape, full body in frame, leave space below feet; 4K 60 fps or 1080p 120 fps (30 fps will blur the racket); lock focus and exposure; try to shoot continuous minutes, not many 10-second clips.
• Outdoor: Avoid large areas of trees and sky, include more of the opposite court; otherwise the ball is almost impossible to track.

7. Reply to User

Start with the conclusion and a direct answer to their current concern, then summarize in order: "Causes → Best vs. Worst commonalities → Comparison with last session → Knees and footwork → Next practice plan", without repeating the entire report. Mention sample size and unclear parts. Tone direct and specific, like a coach who has carefully watched the footage.