Study Buddy on beta

A repeatable test plan and dated evidence for transcripts, streamed answers and video citations.

SP-681 · Verified 27–28 October 2026 · Desktop, light mode

Dark mode

Status as of 27–28 October 2026

Verdict

Core short-video flow verified. Full release acceptance remains open.

Environment

beta.skillpixel.vn, isolated QA course, public selling disabled.

Release

PR #602 merged 26 October. Active beta frontend commit and backend image tag: f305ddf3, containing merge 67290b88.

Evidence

Real beta UI, real Gemini calls, real Bunny playback. Synthetic lesson and learner. No SQL or mocked provider responses were used.

Scope

Synthetic English video plus a fresh lecture job, isolated quota/provider fixtures, Vietnamese reference audio, noisy audio and a long English lecture. Read each result for its limits.

Why this exists

Instructors need a transcript before learners can ask questions about a video. Study Buddy can now generate that transcript or accept an SRT or VTT file. Learners get answers tied to lesson evidence and can jump to the cited time. This report checks those actions on beta and keeps the remaining release checks visible.

What changed, in one picture

Beta walkthrough

About 92 seconds, silent. The blue dot is the mouse. This is the deployed beta app with synthetic QA data. The synthetic video has a plain navy picture and spoken lesson audio.

1. Open the ready learner panel with instructor-provided transcript attribution.
2. Ask when to review the five new words. Watch provisional text become a validated answer.
3. Click a timestamp and verify that the real Bunny player moves.
4. Turn Study Buddy off, reload, then turn it on again. The transcript is reused.
5. Choose Full course and ask about active recall. This is an extra demonstration, not a full course-retrieval benchmark.

Follow-up execution, 28 October

All nine previously unrun cases were attempted. Pass, Partial and Fail remain separate. API checks used an isolated beta tenant; the original video and screenshots remain from the first QA course.

Reference audio scores

The threshold was set before transcription: normalized word error at most 15%, with nonempty cues inside the video duration. Word error counts substitutions, insertions and deletions against reference words. These scores do not replace an independent human review or prove absolute timing accuracy.

The short Vietnamese videos include a still frame tail after the spoken reference audio. The noisy fixture adds deterministic white noise at about 10 dB signal-to-noise ratio. Sources: Google FLEURS, CC BY 4.0; MIT OpenCourseWare, Ana Bell, Fall 2016, CC BY-NC-SA. The site publishes scores only, with no full reference transcripts or source audio. Download scores.

New execution evidence

• LEC-01: Pass. Fresh beta lecture job completed; origin=lecture_script, segments=7, generation_state=none. Uploaded XLSX/PPTX through presigned R2 URLs; no transcript was manually supplied.
• QUO-01: Pass. Isolated tenant text allowance set to 0. Stream returned HTTP 429, application/json, AI_ALLOWANCE_USED; ledger rows 0 -> 0. No provider row created.
• ERR-01: Partial. Zero-credit toggle created a failed transcription with credit_used_up and a readable message. Restored QA allowance, called Retry, then real Gemini transcription became none, ready=True in 16.1s. This follow-up used API evidence; the failed-state UI was not captured. Readable failure and successful API retry are proven. Playback in the failed-state UI was not independently exercised.
• KEY-01: Pass. Isolated own-key OpenAI tenant reported no_gemini_key. A VTT upload succeeded. Provider ledger rows remained 1. The placeholder key was never used in an OpenAI call.
• RATE-01: Pass. 11 explicit QA requests in 6.7s. Statuses [504, 504, 504, 504, 504, 504, 504, 504, 504, 504, 429]. Invalid QA Gemini credentials prevented successful provider calls; the final request was rate-limited. Ledger grew by 10 rows.
• RACE-01: Pass. Fresh fixture upload raced with a provider ledger operation. Observed pending rows: 1; terminal statuses: ['completed']. Final origin: instructor_upload. All short quality jobs were already terminal; the long quality operation existed in the baseline. The only new transcription row belonged to this fresh fixture. Instructor source survived its completed late result.
• BILL-01: Partial. Inspected beta tenant ledger and signed-in AI Studio usage/spend/billing. Completed transcription and failed own-key attempts are recorded. AI Studio exposed aggregate Skill Pixel usage, but the accessible billing account linked a different project. No beta invoice or per-request provider reconciliation was available. Private financial values are excluded.
• FB-01: Partial. Clicked Helpful on a freshly validated beta answer in a QA browser session tagged is_test/test_run. The button became aria-pressed=true and a screenshot was saved. PostHog SDK/flags loaded successfully, but no matching event appeared in project 417708 after repeated queries. No synthetic analytics event was sent to replace the real UI action. Delivery, identifier payload and absence of conversation text remain unverified.
• QUAL-01: Fail. Fresh beta transcription scored against licensed reference text: 4/5 clips passed the declared 15% normalized word-error threshold and timestamp bounds. Independent human review and absolute timestamp alignment remain unverified.

How to test it again

1. Start with a QA video

   Sign in as an instructor. Open a ready video lesson in an isolated course. Check that the card is off and no transcript exists. Do not use an existing customer lesson.

   

2. Turn Study Buddy on

   Use one toggle action. Expect the pending message, then a ready transcript. This run reached Ready at the 18.07-second polling observation and saved 17 cues in English. Exact job-count and billing reconciliation remain open.

   

   

3. Check uploads and source priority

   Upload a valid SRT, then a valid VTT. Check the language, segment count and instructor origin. Reject invalid timing, a file over 2 MB, the wrong extension and cues past the video end. Confirm each rejection leaves the previous transcript unchanged.

   The test upload reused the machine transcript text. Instructor origin confirms the upload path; it does not mean a human reviewed the wording.

   

4. Ask as an enrolled learner

   Open the lesson in a separate learner session. Ask, “When should I review the five new words?” Confirm that partial text appears before final feedback and source controls.

   

   

5. Check the answer and citation

   The expected intervals are one, three and seven days. Check them against the source cue from 17.8 to 23.1 seconds. Click the timestamp and watch the real player time change.

   

   

6. Check visibility and recovery

   Turn Study Buddy off and reload as the learner. The panel must disappear while the video remains playable. Turn it on again and confirm it reuses the saved transcript. Interrupt a reply, then retry the original question explicitly.

   

7. Check language changes

   Change only the QA course language. The old transcript must stay unchanged and the instructor must see the language warning. Restore the course language after the check.

   

Reading the screen

Why the core result can be trusted

Known answer. The QA cue says to review after one, three and seven days. The observed answer gave those same intervals. Its first citation points to that cue.

Live streaming. The browser showed provisional text at 5.554 seconds and final controls at 6.349 seconds. Measured warm API stream: first text 7.2262s; final 8.405s. No five-second performance target is claimed. Browser sampling and screenshots add overhead. These are single warm observations, not percentile benchmarks.

Real access checks. Student management attempts and unenrolled chat requests were denied. An actual other-tenant hostname was also denied. That request stopped at the entitlement check; other tenant-boundary branches remain outside this smoke test.

Follow-up and refusal. A follow-up explained active recall using lesson cues. A stock-price question was refused because the lesson does not contain that information.

Test plan and execution results

41 cases. P0 is a release gate. P1 covers edge cases and wider acceptance. Expand a case to compare its expected and observed behavior. Not run means no beta proof was collected.

41 cases shown

Known limits and open gates

• Anonymous status code. Beta safely denied the anonymous request with 403. The plan expected 401. This remains a contract mismatch; no authentication bypass was found.
• Fresh lecture-video path. A new job reached DELIVERED and saved seven cues with lecture_script origin.
• Isolated tenant. Quota, missing Gemini key, failed generation, rate limit and upload/generation checks were executed in a separate QA tenant. Its managed key mode and positive allowance were restored. The QA public subdomain returned 521, so scoped API checks used the beta default ingress with an asserted tenant hint.
• Quality and accounting. Independent human quality grading and provider invoice reconciliation remain open. No broad quality or cost-accuracy claim is made.
• Captions and feedback. English captions were attached and fetched successfully. Player menu selection remains open. An actual Helpful click was made; see FB-01 for analytics receipt.
• Recovery. A mid-answer network interruption was attempted, but the retry verification timed out. This case remains Partial. Sidebar navigation did abort the browser stream and left no stale question in the next lesson; provider-side cancellation was not traced.
• English greeting. The beta welcome message shows replacement characters around the lesson title. This is a visible text defect in the captured UI; it did not prevent asking a question.
• Production. This is beta evidence. It is not a production smoke test or a production release approval.

Decisions taken

Rollout checklist

• Done: PR #602 merged; active beta deployment contains the merge.
• Done: real short-video generation, SRT/VTT upload, learner answer, streaming, citation seek and visibility checks.
• Open: resolve or accept the anonymous status-code mismatch.
• Done: fresh lecture-video, provider-mode, quota, rate limit and failure-state API checks. The live upload/generation race also passed.
• Open: reconcile usage against the invoice; obtain independent quality grading and confirm operational alerts.
• Open: migration origin/backfill and rollback rehearsal remain separate release checks. A successful feature request alone does not prove them.
• Open: obtain a release decision before production rollout, then run a production smoke test.