Four problems this week

1Does the model really master a question type?

A four-option question is 25 % luck. To claim a skill we keep asking it on new evidence until the evidence is enough, and ask the same fact in several formats so that format luck cancels out [18].

Inside the examiner

Question: Which of the following sounds can be heard during this clip?
Options: A. running water  B. stirring or whisking  C. scrubbing or wiping  D. two plastic objects knocking together
Reply on exactly two lines:  Answer: <A/B/C/D>   Time: <M:SS>

Three answer formats, one fact

The format is part of the adaptation: changing it on a failed cluster is the examiner's second most used strategy.

Correct / asked in real_1001. Multi-select (exact set) is weakest at every level; 6 of 14 short answers needed the judge.

Adaptive questions, hard threshold

Two real items, with the clip the model saw

Route answers are now four quadrants

Reviewers found "turn left" and "turn around" both defensible for targets behind-left. Answers are now front / back × left / right with a 15° margin from every border.

12 / 33

old route items that survive the margin

re-mine before re-running L4

Human review of the questions

79%

question ok

74%

model answer ok

55%

gold ok

219 judgements, 7 reviewers. Gold is flagged most: near-synonym options and the route ambiguity above.

Next. Run the five-format expansion · re-mine and re-run L4 routes · get L2 / L4 reviewed.

2No dataset has enough annotation

HD-EPIC [1] has narrations, sounds, 3D points and gaze, but no masks over time, 3D instances, fixture surfaces or sub-task events; AEA [4][5] and EPIC-KITCHENS [2][3] have less. So the missing layers come from existing models, around the GT, and every stage is checked against held-out GT and by a human page.

S00 Window selection

camera path + GT 3D objects [1]

S01 Camera & world frame

97.6 % of held-out GT points in box [6]

S03 Position anchors & intervals

90 anchors, per-object state timeline

S04 Segmentation at anchors

SAM 3 · coverage 61 % [7]

S05 Object / state / observation IDs

SigLIP same-object AUROC 0.91 [8]

S06 Reference points + depth

Depth Anything 3 · depth ratio 0.98 [9]

S07 Fixture pseudo-twin

hulls + PGSR mesh · 90 % in own hull [10]

S08 Sub-task events

Gemini · segment F1@0.5 = 0.20 [16]

S09 Verb / object / source / target

Gemini · verb 39 %, link 69 % [16]

S10 Hand–object roles

which hand holds / acts on what

S11 Holi 3D instances

154 OBBs · GT point in a box 82 % [11][13]

S12 Descriptions + spatial QA

137 descriptions · 595 QA kept [11]

One 120 s window of HD-EPIC P08. Models used: SAM 3 [7] · SigLIP [8] · Depth Anything 3 [9] · PGSR [10] · Holi-Spatial [11] · Qwen3-VL / Qwen3-Omni [13][12] · Whisper [14] · CLAP [15] · Gemini 3.5 Flash [16] · Aria MPS [6].

Human check pages

Next. Same pipeline on AEA and EPIC-KITCHENS · fix the two weak stages first: S04 coverage 61 %, S11 label match 12 %.

3API cost and quota

Gemini 3.5 Flash is cheap per token [16] but a free key gives 20 requests a day, and the tiers that lift the caps unlock by cumulative spend and account age [17]. GPT-4o is the obvious alternative examiner and it is both pricier and worse at the job.

Of six keys tried, one paid key runs without throttling; the other paid key hit "credits depleted" on 1 Oct; four free keys 429 after a handful of calls.

GPT as examiner writes worse questions

Next. Keep Gemini as examiner · get the billing account to Tier 2 / 3 · batch tier halves the price.

4Some questions cannot be answered from audio + video

People check things by hand — squeeze fruit, probe a potato, test the tap — and what they learn never reaches the pixels or the audio. A model that answers these from a clip is guessing from common sense; robotics collects touch for exactly this gap [19][20]. 14 such moments from EPIC-KITCHENS [2] are on epic-kitchens-tactile-samples; six below.

62 items, gemini-3.5-flash, 2 Oct. Impact sound could stand in for touch in 3 of 14 clips, partly in 6, not at all in 6.

Next. Give the sound items a gold by listening · decide: fifth track or a NEEDS_TOUCH flag inside the four levels · find the same moments in HD-EPIC.

References

1. T. Perrett et al. HD-EPIC: A Highly-Detailed Egocentric Video Dataset. CVPR 2025. arXiv:2502.04144 arxiv.org/abs/2502.04144
2. D. Damen et al. Rescaling Egocentric Vision: EPIC-KITCHENS-100. IJCV 2022. arXiv:2006.13256 arxiv.org/abs/2006.13256
3. J. Huh et al. EPIC-SOUNDS: A Large-Scale Dataset of Actions That Sound. ICASSP 2023. arXiv:2302.00646 arxiv.org/abs/2302.00646
4. Z. Lv et al. Aria Everyday Activities Dataset. arXiv:2402.13349 arxiv.org/abs/2402.13349
5. M. Chen et al. SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing. arXiv:2506.05414 arxiv.org/abs/2506.05414
6. Meta. Project Aria Machine Perception Services (SLAM, eye gaze). facebookresearch.github.io/projectaria_tools/docs/ARK/mps
7. Meta Superintelligence Labs. SAM 3: Segment Anything with Concepts. arXiv:2511.16719 arxiv.org/abs/2511.16719
8. X. Zhai et al. Sigmoid Loss for Language Image Pre-Training (SigLIP). ICCV 2023. arXiv:2303.15343 arxiv.org/abs/2303.15343
9. H. Lin et al. Depth Anything 3: Recovering the Visual Space from Any Views. arXiv:2511.10647 arxiv.org/abs/2511.10647
10. D. Chen et al. PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction. TVCG 2024. arXiv:2406.06521 arxiv.org/abs/2406.06521
11. Visionary Laboratory. Holi-Spatial: automated spatially-aware multimodal dataset construction from raw video. arXiv:2603.07660 arxiv.org/abs/2603.07660
12. Qwen Team. Qwen3-Omni Technical Report. arXiv:2509.17765 arxiv.org/abs/2509.17765
13. Qwen Team. Qwen3-VL-30B-A3B-Instruct (model card) huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct
14. A. Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356 arxiv.org/abs/2212.04356
15. Y. Wu et al. Large-scale Contrastive Language-Audio Pretraining (CLAP). ICASSP 2023. arXiv:2211.06687 arxiv.org/abs/2211.06687
16. Google. Gemini API models and pricing. ai.google.dev/gemini-api/docs/pricing
17. Google. Gemini API rate limits and usage tiers. ai.google.dev/gemini-api/docs/rate-limits
18. E. Kim et al. LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation. arXiv:2412.10424 arxiv.org/abs/2412.10424
19. F. Yang et al. Touch and Go: Learning from Human-Collected Vision and Touch. NeurIPS 2022 D&B. arXiv:2211.12498 arxiv.org/abs/2211.12498
20. L. Fu et al. A Touch, Vision, and Language Dataset for Multimodal Alignment. ICML 2024. arXiv:2402.13232 arxiv.org/abs/2402.13232

Numbers come from runs logged in experiments/ (1001_exp, 1001_examiner_report, 1002, cost_estimate_0929, gemini_credit_request_0930). Video: HD-EPIC and EPIC-KITCHENS-100, CC BY-NC 4.0.