{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"换候选、换模板、换图像：三个不同干预","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.md","pages":[{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","unit":"已知图文配对怎样变成可用的文字候选接口？","unitId":"clip-pairing-to-candidates","chapter":"III","number":1}],"aliases":[{"oldId":"b-e927cc36ffe27417a68853a4","targetId":"b-e927cc36ffe27417a68853a4","page":"reading-opening","operation":"move","beforeSHA256":"1842952762597f407ee66fa98394aa318670dd06eeca5f7a77bb4f2542b78f5a","afterSHA256":"1842952762597f407ee66fa98394aa318670dd06eeca5f7a77bb4f2542b78f5a","reason":"Same source body in authored semantic page"},{"oldId":"b-82bed45f141fabcf88e7e61e","targetId":"b-82bed45f141fabcf88e7e61e","page":"reading-opening","operation":"move","beforeSHA256":"b9a790420611560584ad9070f7e9e60c9da1afe63b7e0688164e330d859589ab","afterSHA256":"b9a790420611560584ad9070f7e9e60c9da1afe63b7e0688164e330d859589ab","reason":"Same source body in authored semantic page"},{"oldId":"b-06217d9ff70edb86fe821d2d","targetId":"b-06217d9ff70edb86fe821d2d","page":"reading-opening","operation":"move","beforeSHA256":"086e35d8c6f13ba74f478977604e7c9e780a177b02f3aa79c3a25292cdea98f4","afterSHA256":"086e35d8c6f13ba74f478977604e7c9e780a177b02f3aa79c3a25292cdea98f4","reason":"Same source body in authored semantic page"},{"oldId":"b-949ca64233062924471ff44e","targetId":"b-949ca64233062924471ff44e","page":"reading-opening","operation":"move","beforeSHA256":"e92ba23916b55714915a6ebd28f56b3a4572a9b5437ecfffdd9a435b2e49669d","afterSHA256":"e92ba23916b55714915a6ebd28f56b3a4572a9b5437ecfffdd9a435b2e49669d","reason":"Same source body in authored semantic page"},{"oldId":"b-198ca703af25253d00582df6","targetId":"b-198ca703af25253d00582df6","page":"reading-opening","operation":"move","beforeSHA256":"6a6219f9c82279ead960e1642fdd534942d4593d554e02e087ac4e57abb0b4de","afterSHA256":"6a6219f9c82279ead960e1642fdd534942d4593d554e02e087ac4e57abb0b4de","reason":"Same source body in authored semantic page"},{"oldId":"b-89b81cafa5d4db10c2aa342c","targetId":"b-89b81cafa5d4db10c2aa342c","page":"reading-opening","operation":"move","beforeSHA256":"f0fb65af26e6d9ecb0b64bbf069740b4dc2ee1d8c31215d5a62337aac21593c3","afterSHA256":"f0fb65af26e6d9ecb0b64bbf069740b4dc2ee1d8c31215d5a62337aac21593c3","reason":"Same source body in authored semantic page"},{"oldId":"b-47646b5c720b1de460731f8a","targetId":"b-47646b5c720b1de460731f8a","page":"reading-opening","operation":"rewrite","beforeSHA256":"e5fd753029617c0031ad1e914ebb8007957df00f2b3f011d66e069224b0e1bf3","afterSHA256":"5d79bac110e7ff73b5c37bb48e558239e1f65a5bb85c4b15986fbcbd737ad9e5","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-9d49d3c78d786ce730613950","targetId":"b-9d49d3c78d786ce730613950","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-e927cc36ffe27417a68853a4","type":"h1","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-e927cc36ffe27417a68853a4","content":"# 换候选、换模板、换图像：三个不同干预","sha256":"1842952762597f407ee66fa98394aa318670dd06eeca5f7a77bb4f2542b78f5a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-82bed45f141fabcf88e7e61e","prerequisites":[{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-82bed45f141fabcf88e7e61e","type":"p","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-82bed45f141fabcf88e7e61e","content":"这里的 zero-shot 不使用当前下游数据集再训练模型，而用已有编码器比较文字候选。增加候选和更换模板是不同干预：前者可以只变 softmax 分母，后者会改变文本向量。LAB04 的真实猫图四臂比较保留，可直接阅读已有结果，不必为理解接口重跑模型。","sha256":"b9a790420611560584ad9070f7e9e60c9da1afe63b7e0688164e330d859589ab","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-e927cc36ffe27417a68853a4","nextBlock":"b-06217d9ff70edb86fe821d2d","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-06217d9ff70edb86fe821d2d","type":"p","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-06217d9ff70edb86fe821d2d","content":"训练完后，CLIP 可以把类别写成文字候选，例如 `a photo of a cat.`，再比较图像与每条候选的表示。这里的 zero-shot 用法，是不使用当前下游数据集再训练模型，直接用已有的图像和文字编码器比较候选描述。它不保证相关概念从未出现在预训练资料中。改变候选文字，会改变这次比较的答案集合；加入新候选会改变 softmax 归一化分数，因此这些分数不能直接视为稳定校准的真实概率。","sha256":"086e35d8c6f13ba74f478977604e7c9e780a177b02f3aa79c3a25292cdea98f4","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-82bed45f141fabcf88e7e61e","nextBlock":"b-949ca64233062924471ff44e","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-949ca64233062924471ff44e","type":"p","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-949ca64233062924471ff44e","content":"设猫和狗的 logits 是 2 与 1，候选 softmax 给猫 $e^2/(e^2+e)=0.7311$。新增一个 logit 同为 2 的幼猫候选，原猫概率变为 $e^2/(2e^2+e)=0.4223$。同一图像、同一猫文字、同一点积都没有改变，只是分母和选项变了。","sha256":"e92ba23916b55714915a6ebd28f56b3a4572a9b5437ecfffdd9a435b2e49669d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"e^2/(e^2+e)=0.7311","definitionContext":"换候选、换模板、换图像：三个不同干预"},{"latex":"e^2/(2e^2+e)=0.4223","definitionContext":"换候选、换模板、换图像：三个不同干预"}],"previousBlock":"b-06217d9ff70edb86fe821d2d","nextBlock":"b-198ca703af25253d00582df6","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-198ca703af25253d00582df6","type":"p","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-198ca703af25253d00582df6","content":"因此有三个需要隔离的实验。只增候选：已有候选余弦应保持，softmax 可以改变。只改文字模板：文字向量改变，余弦也可能改变。只换图像：图像向量改变，应检查哪些候选关系随视觉内容变化。若三者一起改，就难以解释分数差异。","sha256":"6a6219f9c82279ead960e1642fdd534942d4593d554e02e087ac4e57abb0b4de","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-949ca64233062924471ff44e","nextBlock":"b-89b81cafa5d4db10c2aa342c","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-89b81cafa5d4db10c2aa342c","type":"p","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-89b81cafa5d4db10c2aa342c","content":"LAB04 保留的四臂猫图实验专门做这件事：A 是三类基础候选，B 添加 tiger/fox，C 保持三类但改 cropped 模板，D 从 B 删除 cat。删除正确选项后仍然必有最高项；这不是模型发现照片没有猫，而是回答空间已经被使用者改变。","sha256":"f0fb65af26e6d9ecb0b64bbf069740b4dc2ee1d8c31215d5a62337aac21593c3","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-198ca703af25253d00582df6","nextBlock":"b-47646b5c720b1de460731f8a","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]},{"id":"b-47646b5c720b1de460731f8a","type":"figure","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-47646b5c720b1de460731f8a","content":"图内标签与数值：图像编码器；红色杯子；蓝色杯子；红色球；红色马克杯；同一图像表示；增加近义候选，归一化分数也会改变。；zero-shot：不使用当前下游数据集再训练模型；候选与连线为使用接口示意；不填写未运行的候选分数，也不声称概念从未出现在预训练中。\n\n新增文本候选会改变比较集合；画面没有给出未经运行的新模型分数。 查看对应图解状态\n\n- 图像编码器\n- 红色杯子\n- 蓝色杯子\n- 红色球\n- 红色马克杯\n- 同一图像表示\n- 增加近义候选，归一化分数也会改变。\n- zero-shot：不使用当前下游数据集再训练模型\n- 候选与连线为使用接口示意；不填写未运行的候选分数，也不声称概念从未出现在预训练中。新增文本候选会改变比较集合；画面没有给出未经运行的新模型分数。 [查看对应图解状态](/#/l3-r2-p58/1)[查看完整图解](/#/l3-r2-p58/1)","sha256":"5d79bac110e7ff73b5c37bb48e558239e1f65a5bb85c4b15986fbcbd737ad9e5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"新增文本候选会改变比较集合；画面没有给出未经运行的新模型分数。 查看对应图解状态","labels":["图像编码器","红色杯子","蓝色杯子","红色球","红色马克杯","同一图像表示","增加近义候选，归一化分数也会改变。","zero-shot：不使用当前下游数据集再训练模型","候选与连线为使用接口示意；不填写未运行的候选分数，也不声称概念从未出现在预训练中。"],"description":"新增文本候选会改变比较集合；画面没有给出未经运行的新模型分数。 查看对应图解状态图像编码器红色杯子蓝色杯子红色球红色马克杯同一图像表示增加近义候选，归一化分数也会改变。zero-shot：不使用当前下游数据集再训练模型候选与连线为使用接口示意；不填写未运行的候选分数，也不声称概念从未出现在预训练中。","evidenceIdentity":"新增文本候选会改变比较集合；画面没有给出未经运行的新模型分数。 查看对应图解状态","images":[{"src":"/assets/display/9a66faeae8dd2c354bc56482.webp","alt":""}]},"previousBlock":"b-89b81cafa5d4db10c2aa342c","nextBlock":"b-9d49d3c78d786ce730613950","prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-9d49d3c78d786ce730613950","type":"h2","scope":"clip-candidate-probability","url":"https://codingai-lec03.pages.dev/course/blocks/clip-candidate-probability.html#b-9d49d3c78d786ce730613950","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"reading-opening","title":"换候选、换模板、换图像：三个不同干预","number":1},"sectionTitle":"换候选、换模板、换图像：三个不同干预","currentQuestion":"换候选、换模板、换图像：三个不同干预","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-47646b5c720b1de460731f8a","nextBlock":null,"prerequisites":[{"id":"b-e927cc36ffe27417a68853a4","url":"#b-e927cc36ffe27417a68853a4","role":"section-title"},{"id":"b-82bed45f141fabcf88e7e61e","url":"#b-82bed45f141fabcf88e7e61e","role":"problem-context"}],"relatedBlocks":[{"id":"b-47646b5c720b1de460731f8a","url":"#b-47646b5c720b1de460731f8a","relation":"figure-and-explanation"}]}]}