{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"一张相似度矩阵，连接训练和使用","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html","markdown":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.md","pages":[{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","unit":"已知图文配对怎样变成可用的文字候选接口？","unitId":"clip-pairing-to-candidates","chapter":"III","number":1},{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","unit":"已知图文配对怎样变成可用的文字候选接口？","unitId":"clip-pairing-to-candidates","chapter":"III","number":2}],"aliases":[{"oldId":"clip-matrix-worked","targetId":"clip-pairwise-matrix-content","operation":"split-range-start"},{"oldId":"explanation","targetId":"clip-pairwise-matrix","operation":"split-explanation-range"},{"oldId":"b-402fe27f594137048f725092","targetId":"b-402fe27f594137048f725092","operation":"historical-explanation","sourceScope":"clip-bidirectional-loss"},{"oldId":"b-d7b057faf95ff352987e102c","targetId":"b-d7b057faf95ff352987e102c","operation":"historical-explanation","sourceScope":"document"},{"oldId":"b-3171a2091eaba2db72cb9b7c","targetId":"b-3171a2091eaba2db72cb9b7c","operation":"historical-explanation","sourceScope":"clip-pairwise-matrix"},{"oldId":"b-0509f60cc51d79f0cb203d03","targetId":"b-0509f60cc51d79f0cb203d03","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","afterSHA256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","reason":"Same source body in authored semantic page"},{"oldId":"b-83240fc8d1006516586e2eec","targetId":"b-83240fc8d1006516586e2eec","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"277fd7cf382a94fefa18a7925e57af38573f078d39168ab8c57e77e08247011d","afterSHA256":"277fd7cf382a94fefa18a7925e57af38573f078d39168ab8c57e77e08247011d","reason":"Same source body in authored semantic page"},{"oldId":"b-58c938742970c2183594b6c1","targetId":"b-58c938742970c2183594b6c1","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"ae64809edc18a65a51fb1a003a9bc8a53bb67879ee73cf21d8463bea753cb2c6","afterSHA256":"ae64809edc18a65a51fb1a003a9bc8a53bb67879ee73cf21d8463bea753cb2c6","reason":"Same source body in authored semantic page"},{"oldId":"b-28a5e64a1e04900c6ad3efe8","targetId":"b-28a5e64a1e04900c6ad3efe8","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"17775dd8144b9a7fd31777d1611efe47ffd56052acc4f240b8216acdaa2988d5","afterSHA256":"17775dd8144b9a7fd31777d1611efe47ffd56052acc4f240b8216acdaa2988d5","reason":"Same source body in authored semantic page"},{"oldId":"b-58808538e43ff4afd70cde8e","targetId":"b-58808538e43ff4afd70cde8e","page":"clip-pairwise-matrix","operation":"rewrite","beforeSHA256":"9cc2f7248e03530cb4c524afda9cbf83ab37b696a6a89c902a603b83a15d909e","afterSHA256":"aaba9b9cd2acc11b327bd1b7f7742a343473f61e2c8b58f48e279e34472a9409","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-f34b6ff91ece135ec1926fa0","targetId":"b-f34b6ff91ece135ec1926fa0","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","afterSHA256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","reason":"Same source body in authored semantic page"},{"oldId":"b-3f320fc9b79a3f9e256e53e2","targetId":"b-3f320fc9b79a3f9e256e53e2","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"51cc99ee76b43b536a81dad6c7d4b022c476327552eaac17f6b162371ea53391","afterSHA256":"51cc99ee76b43b536a81dad6c7d4b022c476327552eaac17f6b162371ea53391","reason":"Same source body in authored semantic page"},{"oldId":"b-ca35d0b1b84ca938acb964ac","targetId":"b-ca35d0b1b84ca938acb964ac","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"6c3a21981a8ce3f76bbf1f59e8561a530b19e1f6013e4938562d95ff978d4d51","afterSHA256":"6c3a21981a8ce3f76bbf1f59e8561a530b19e1f6013e4938562d95ff978d4d51","reason":"Same source body in authored semantic page"},{"oldId":"b-eb56c92ee182a4e9a1df6128","targetId":"b-eb56c92ee182a4e9a1df6128","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"87efe740a2b21be719887a28b5b3cfd8c6d8e212a6b15ab44c84bdd96d693ac2","afterSHA256":"87efe740a2b21be719887a28b5b3cfd8c6d8e212a6b15ab44c84bdd96d693ac2","reason":"Same source body in authored semantic page"},{"oldId":"b-05f4bb81c4c6a20a4109158e","targetId":"b-05f4bb81c4c6a20a4109158e","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"857ca913953885703b98782045bf729f79f75c844ce6644ee3883054b71efc79","afterSHA256":"857ca913953885703b98782045bf729f79f75c844ce6644ee3883054b71efc79","reason":"Same source body in authored semantic page"},{"oldId":"b-c800bf0333a830b7610eff55","targetId":"b-c800bf0333a830b7610eff55","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"fe064a2fcb858407600cd6f6c4c997faf3a436b4aea82081df73d284d714ba3f","afterSHA256":"fe064a2fcb858407600cd6f6c4c997faf3a436b4aea82081df73d284d714ba3f","reason":"Same source body in authored semantic page"},{"oldId":"b-9ebe3ed1a4c8a1ade1c6cffe","targetId":"b-9ebe3ed1a4c8a1ade1c6cffe","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"67573bdb249c87f3bc9dfdf450ab3a13bcfa16c550bb73ec5298118acd730f04","afterSHA256":"67573bdb249c87f3bc9dfdf450ab3a13bcfa16c550bb73ec5298118acd730f04","reason":"Same source body in authored semantic page"},{"oldId":"b-f03a8750ffafce82451ec11a","targetId":"b-f03a8750ffafce82451ec11a","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"caa8ea483f5051a9f145a1a981ca2d77ad289e9d05c7b4c5bf618dd2738c48da","afterSHA256":"caa8ea483f5051a9f145a1a981ca2d77ad289e9d05c7b4c5bf618dd2738c48da","reason":"Same source body in authored semantic page"},{"oldId":"b-5c906026e32edb5171d2a3c9","targetId":"b-5c906026e32edb5171d2a3c9","page":"clip-pairwise-matrix","operation":"move","beforeSHA256":"e184e0f8bb11c3a8463f4e106cf928fd98498661eaaedfa98f9404b6e87707c3","afterSHA256":"e184e0f8bb11c3a8463f4e106cf928fd98498661eaaedfa98f9404b6e87707c3","reason":"Same source body in authored semantic page"},{"oldId":"b-04e8944a674e7c5c99b581dd","targetId":"b-04e8944a674e7c5c99b581dd","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"5f5d03b18f32fe9ef45e0e47dc9459338e7af869590cf2a4f35f7c07b37ee552","afterSHA256":"5f5d03b18f32fe9ef45e0e47dc9459338e7af869590cf2a4f35f7c07b37ee552","reason":"Same source body in authored semantic page"},{"oldId":"b-f028cf8cbefd2fd9433dc145","targetId":"b-f028cf8cbefd2fd9433dc145","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"7cb63b04e1a0edf44bbd40f7be4f6471f882de0e7485d96ca394a8b321aeb6cb","afterSHA256":"7cb63b04e1a0edf44bbd40f7be4f6471f882de0e7485d96ca394a8b321aeb6cb","reason":"Same source body in authored semantic page"},{"oldId":"b-891c9f0a16da63982ca628a6","targetId":"b-891c9f0a16da63982ca628a6","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"997240b500a1857825d9d5e874382060b832ca04e5892cf1a198efaf8a4247a5","afterSHA256":"997240b500a1857825d9d5e874382060b832ca04e5892cf1a198efaf8a4247a5","reason":"Same source body in authored semantic page"},{"oldId":"b-1806238c1aa1403471791217","targetId":"b-1806238c1aa1403471791217","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"003f4000ef5478872af45c630b0aee2f6ce26c9522cbfa443a8343611fd3f829","afterSHA256":"003f4000ef5478872af45c630b0aee2f6ce26c9522cbfa443a8343611fd3f829","reason":"Same source body in authored semantic page"},{"oldId":"b-3c5bd63187b5fb33f605416f","targetId":"b-3c5bd63187b5fb33f605416f","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","afterSHA256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","reason":"Same source body in authored semantic page"},{"oldId":"b-39a1d7c7584b8ec2897633e9","targetId":"b-39a1d7c7584b8ec2897633e9","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","afterSHA256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","reason":"Same source body in authored semantic page"},{"oldId":"b-3a6c82b2b59d5f8ccf1e2b00","targetId":"b-3a6c82b2b59d5f8ccf1e2b00","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"0ccb583b2449578582da9ec16ffde2fb6d7ad0c23b5ccc9bb6751309b7bf4ee1","afterSHA256":"0ccb583b2449578582da9ec16ffde2fb6d7ad0c23b5ccc9bb6751309b7bf4ee1","reason":"Same source body in authored semantic page"},{"oldId":"b-61d6558c0f15006f3f702d63","targetId":"b-61d6558c0f15006f3f702d63","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"ab974e2406f01db64e6ece20c6d7ec195f6976f99a8db5ad85894d9236fd945e","afterSHA256":"ab974e2406f01db64e6ece20c6d7ec195f6976f99a8db5ad85894d9236fd945e","reason":"Same source body in authored semantic page"},{"oldId":"b-24b83262d47705b8e790b2f5","targetId":"b-24b83262d47705b8e790b2f5","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","afterSHA256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","reason":"Same source body in authored semantic page"},{"oldId":"b-6777ab3f6f978555d3e321ed","targetId":"b-6777ab3f6f978555d3e321ed","page":"clip-bidirectional-loss","operation":"move","beforeSHA256":"932121d2b767733815b139c5ad8a869da1bdb823cf0f9adeee781a2e53e5045a","afterSHA256":"932121d2b767733815b139c5ad8a869da1bdb823cf0f9adeee781a2e53e5045a","reason":"Same source body in authored semantic page"},{"oldId":"clip-pairwise-matrix-title","targetId":"b-428c1236f73dc74768bbcb40","operation":"retain-source-identity"},{"oldId":"clip-bidirectional-loss-title","targetId":"b-d2b28f664355de7b7eab161a","operation":"retain-source-identity"},{"oldId":"clip-bidirectional-loss-bridge","targetId":"b-73b1943bb124b2a3e249ec2e","operation":"retain-source-identity"}],"blocks":[{"id":"b-428c1236f73dc74768bbcb40","type":"h2","scope":"clip-pairwise-matrix","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-428c1236f73dc74768bbcb40","content":"## 每一行选文字，每一列选图像","sha256":"e2a0b1788fb97514c755f652390a431f7a71a3584534dd9ba2ba112c7959e6a3","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-0509f60cc51d79f0cb203d03","prerequisites":[{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-0509f60cc51d79f0cb203d03","type":"h2","scope":"explanation","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-0509f60cc51d79f0cb203d03","content":"## 解释、算例与证据","sha256":"d71cf15e707c8a3674b4c90a599efee783cb84da5b8c8236a1d01c3add7b4958","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-428c1236f73dc74768bbcb40","nextBlock":"b-83240fc8d1006516586e2eec","prerequisites":[{"id":"b-428c1236f73dc74768bbcb40","url":"#b-428c1236f73dc74768bbcb40","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-83240fc8d1006516586e2eec","type":"h3","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-83240fc8d1006516586e2eec","content":"### 三对图文的双向损失：从单位向量到 0.09567","sha256":"277fd7cf382a94fefa18a7925e57af38573f078d39168ab8c57e77e08247011d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-0509f60cc51d79f0cb203d03","nextBlock":"b-58c938742970c2183594b6c1","prerequisites":[{"id":"b-0509f60cc51d79f0cb203d03","url":"#b-0509f60cc51d79f0cb203d03","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-58c938742970c2183594b6c1","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-58c938742970c2183594b6c1","content":"# P57 · 一张相似度矩阵，连接训练和使用","sha256":"ae64809edc18a65a51fb1a003a9bc8a53bb67879ee73cf21d8463bea753cb2c6","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-83240fc8d1006516586e2eec","nextBlock":"b-28a5e64a1e04900c6ad3efe8","prerequisites":[{"id":"b-83240fc8d1006516586e2eec","url":"#b-83240fc8d1006516586e2eec","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-28a5e64a1e04900c6ad3efe8","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-28a5e64a1e04900c6ad3efe8","content":"这页的数字能从单位向量完全复算：矩阵不是模型实测成绩。先看一行的三个竞争候选，再看一列，最后对两种方向取平均。温度改变 logits 和概率，不改变原始余弦本身。真实 CLIP 的结果需读取相应模型输出，不能沿用这里的教学数值。","sha256":"17775dd8144b9a7fd31777d1611efe47ffd56052acc4f240b8216acdaa2988d5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-58c938742970c2183594b6c1","nextBlock":"b-58808538e43ff4afd70cde8e","prerequisites":[{"id":"b-58c938742970c2183594b6c1","url":"#b-58c938742970c2183594b6c1","role":"section-title"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-58808538e43ff4afd70cde8e","type":"figure","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-58808538e43ff4afd70cde8e","content":"图内标签与数值：单位向量点积 Sᵢⱼ；教学构造；I1；T1；红杯；I2；T2；蓝杯；I3；T3；人物；0.8；0.3；0.0；0.0；0.8；0.3；0.3；0.0；0.8；文字找图像 ↓；τ = 0.2；第一列 logits：4, 0, 1.5；p(正确配对) ≈ 0.909；−log p ≈ 0.0957；行目标 + 列目标；在批次中辨认配对；对角线来自配对顺序；数值是由明确单位向量构造的教学矩阵，不是 CLIP 实测分数。\n\n![I₁ 红杯](/assets/display/9a66faeae8dd2c354bc56482.webp)I₁ · 红杯 / T₁\n\n![I₂ 蓝杯](/assets/display/6b88c20342b6fc400ef87d57.webp)I₂ · 蓝杯 / T₂\n\n![I₃ 人物](/assets/display/512610a019c560ff0384e86a.webp)I₃ · 人物 / T₃\n\n图像 I₁、I₂、I₃ 为红杯、蓝杯、人物；文字 T₁、T₂、T₃ 按相同配对顺序排列。\n\n单位向量点积 Sᵢⱼ（教学构造）\n\n| 图像／文字 | T₁ | T₂ | T₃ |\n| --- | --- | --- | --- |\n| I₁ | .8 | .3 | 0 |\n| I₂ | 0 | .8 | .3 |\n| I₃ | .3 | 0 | .8 |\n\n逐行比较文字候选，逐列比较图像候选；对角线由已知配对确定。\n三幅图与三段文字组成3乘3单位向量点积矩阵，先突出首行，再突出首列，并显示温度与正确项概率。 [查看对应图解状态](/#/l3-r2-p57/1)[查看完整图解](/course/figures/l3-r2-p57-1.html)","sha256":"aaba9b9cd2acc11b327bd1b7f7742a343473f61e2c8b58f48e279e34472a9409","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"I₁ · 红杯 / T₁","labels":["单位向量点积 Sᵢⱼ；教学构造","I1","T1","红杯","I2","T2","蓝杯","I3","T3","人物","0.8","0.3","0.0","0.0","0.8","0.3","0.3","0.0","0.8","文字找图像 ↓","τ = 0.2","第一列 logits：4, 0, 1.5","p(正确配对) ≈ 0.909","−log p ≈ 0.0957","行目标 + 列目标","在批次中辨认配对","对角线来自配对顺序；数值是由明确单位向量构造的教学矩阵，不是 CLIP 实测分数。"],"description":"I₁ · 红杯 / T₁I₂ · 蓝杯 / T₂I₃ · 人物 / T₃图像 I₁、I₂、I₃ 为红杯、蓝杯、人物；文字 T₁、T₂、T₃ 按相同配对顺序排列。单位向量点积 Sᵢⱼ（教学构造）图像／文字T₁T₂T₃I₁.8.30I₂0.8.3I₃.30.8逐行比较文字候选，逐列比较图像候选；对角线由已知配对确定。","evidenceIdentity":"I₁ · 红杯 / T₁","images":[{"src":"/assets/display/9a66faeae8dd2c354bc56482.webp","alt":""},{"src":"/assets/display/6b88c20342b6fc400ef87d57.webp","alt":""},{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":""},{"src":"/assets/display/9a66faeae8dd2c354bc56482.webp","alt":"I₁ 红杯"},{"src":"/assets/display/6b88c20342b6fc400ef87d57.webp","alt":"I₂ 蓝杯"},{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":"I₃ 人物"}]},"previousBlock":"b-28a5e64a1e04900c6ad3efe8","nextBlock":"b-f34b6ff91ece135ec1926fa0","prerequisites":[{"id":"b-58c938742970c2183594b6c1","url":"#b-58c938742970c2183594b6c1","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-f34b6ff91ece135ec1926fa0","type":"h2","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-f34b6ff91ece135ec1926fa0","content":"## 课堂讲述","sha256":"786ce10a6b32a492259a3dccd0a54e61cd41d28a34382fb7b5386c5f03b7ad69","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-58808538e43ff4afd70cde8e","nextBlock":"b-3f320fc9b79a3f9e256e53e2","prerequisites":[{"id":"b-58c938742970c2183594b6c1","url":"#b-58c938742970c2183594b6c1","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-3f320fc9b79a3f9e256e53e2","type":"p","scope":"narrative","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-3f320fc9b79a3f9e256e53e2","content":"假设这一批有三对图文。每幅图都和三段文字比较，每段文字也和三幅图比较，得到这张矩阵。排好顺序以后，对角线是数据中提供的配对。\n训练不只是让对角线数值绝对变大，而是让正确对应相对于其他候选更容易被选出。原始 CLIP 用双向的对比目标，从图找文字，也从文字找图。\n训练以后留下两个编码器。我们可以把新的图片和新的文字送进去得到相似度，这个相似度又能够服务于分类候选或者检索。到这里，训练目标、得到的表示和使用接口终于在同一张图上对齐，而不是三段互不相关的术语。","sha256":"51cc99ee76b43b536a81dad6c7d4b022c476327552eaac17f6b162371ea53391","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f34b6ff91ece135ec1926fa0","nextBlock":"b-ca35d0b1b84ca938acb964ac","prerequisites":[{"id":"b-f34b6ff91ece135ec1926fa0","url":"#b-f34b6ff91ece135ec1926fa0","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-ca35d0b1b84ca938acb964ac","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-ca35d0b1b84ca938acb964ac","content":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。","sha256":"6c3a21981a8ce3f76bbf1f59e8561a530b19e1f6013e4938562d95ff978d4d51","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"e_1,e_2,e_3","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"},{"latex":"(.8,0,.3,\\sqrt{.27})","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"},{"latex":"(.3,.8,0,\\sqrt{.27})","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"},{"latex":"(0,.3,.8,\\sqrt{.27})","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"},{"latex":".64+.09+.27=1","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"}],"previousBlock":"b-3f320fc9b79a3f9e256e53e2","nextBlock":"b-eb56c92ee182a4e9a1df6128","prerequisites":[{"id":"b-f34b6ff91ece135ec1926fa0","url":"#b-f34b6ff91ece135ec1926fa0","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"},{"id":"b-eb56c92ee182a4e9a1df6128","url":"#b-eb56c92ee182a4e9a1df6128","relation":"same-reasoning"}]},{"id":"b-eb56c92ee182a4e9a1df6128","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-eb56c92ee182a4e9a1df6128","content":"$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$","sha256":"87efe740a2b21be719887a28b5b3cfd8c6d8e212a6b15ab44c84bdd96d693ac2","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.","definitionContext":"P57 的矩阵是可实现的教学算例，不是真实 CLIP 输出。图像向量取四维空间前三个单位基 $e_1,e_2,e_3$；文字向量为 $(.8,0,.3,\\sqrt{.27})$、$(.3,.8,0,\\sqrt{.27})$、$(0,.3,.8,\\sqrt{.27})$。每个文字向量的模平方是 $.64+.09+.27=1$，所以点积确实是余弦。\n$S=\\begin{bmatrix}.8&.3&0\\\\0&.8&.3\\\\.3&0&.8\\end{bmatrix}.$"}],"previousBlock":"b-ca35d0b1b84ca938acb964ac","nextBlock":"b-9346a6c12c0cce5367bc15d8","prerequisites":[{"id":"b-f34b6ff91ece135ec1926fa0","url":"#b-f34b6ff91ece135ec1926fa0","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"},{"id":"b-ca35d0b1b84ca938acb964ac","url":"#b-ca35d0b1b84ca938acb964ac","relation":"same-reasoning"}]},{"id":"b-9346a6c12c0cce5367bc15d8","type":"h4","scope":"clip-pairwise-matrix","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-9346a6c12c0cce5367bc15d8","content":"#### 图示条件与公式约定","sha256":"ef7f915c1ae63ed67bd23fecf050249273ba0efa8965d2c8d1c1c6b2d3178122","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-eb56c92ee182a4e9a1df6128","nextBlock":"b-05f4bb81c4c6a20a4109158e","prerequisites":[{"id":"b-f34b6ff91ece135ec1926fa0","url":"#b-f34b6ff91ece135ec1926fa0","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-05f4bb81c4c6a20a4109158e","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-05f4bb81c4c6a20a4109158e","content":"精确构造：图像单位向量为四维空间前三个标准基 e1,e2,e3；文字单位向量 t1=(.8,0,.3,sqrt(.27)), t2=(.3,.8,0,sqrt(.27)), t3=(0,.3,.8,sqrt(.27))。故 S=[[.8,.3,0],[0,.8,.3],[.3,0,.8]]，均为真实可实现的单位向量点积。温度 .2 时第一行 logits=(4,1.5,0)，第一列=(4,0,1.5)；正确配对概率 .9087599243，负对数 .0956743294。所有行列同样的正确概率，双向平均目标也为 .0956743294。这是教学矩阵，不是 CLIP 输出。","sha256":"857ca913953885703b98782045bf729f79f75c844ce6644ee3883054b71efc79","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-9346a6c12c0cce5367bc15d8","nextBlock":"b-1212fd27cb18111e8c062864","prerequisites":[{"id":"b-9346a6c12c0cce5367bc15d8","url":"#b-9346a6c12c0cce5367bc15d8","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-1212fd27cb18111e8c062864","type":"h4","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-1212fd27cb18111e8c062864","content":"#### 这条线索的展开与后续阅读","sha256":"48b8229f7559b36ed4d5628416e3991e5c73293fbce43374db67386c8736f3bc","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-05f4bb81c4c6a20a4109158e","nextBlock":"b-c800bf0333a830b7610eff55","prerequisites":[{"id":"b-9346a6c12c0cce5367bc15d8","url":"#b-9346a6c12c0cce5367bc15d8","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-c800bf0333a830b7610eff55","type":"ul","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-c800bf0333a830b7610eff55","content":"- [一个小矩阵里，同时做两个方向的匹配](/supplement.html?return=l3-r2-p57/0#/l3-clip-train/0) 本页说明\n- [CLIP 从图文对走到候选接口｜核心改动怎样工作](/supplement.html?return=l3-r2-p57/0#/l3-clip-full-chain-mechanism/0) 本页说明\n- [CLIP 从图文对走到候选接口｜一起算一个完整例子](/supplement.html?return=l3-r2-p57/0#/l3-clip-full-chain-example/0) 本页说明","sha256":"fe064a2fcb858407600cd6f6c4c997faf3a436b4aea82081df73d284d714ba3f","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1212fd27cb18111e8c062864","nextBlock":"b-9ebe3ed1a4c8a1ade1c6cffe","prerequisites":[{"id":"b-1212fd27cb18111e8c062864","url":"#b-1212fd27cb18111e8c062864","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-9ebe3ed1a4c8a1ade1c6cffe","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-9ebe3ed1a4c8a1ade1c6cffe","content":"- [一个小矩阵里，同时做两个方向的匹配](/supplement.html?return=l3-r2-p57/0#/l3-clip-train/0) 本页说明","sha256":"67573bdb249c87f3bc9dfdf450ab3a13bcfa16c550bb73ec5298118acd730f04","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c800bf0333a830b7610eff55","nextBlock":"b-f03a8750ffafce82451ec11a","prerequisites":[{"id":"b-1212fd27cb18111e8c062864","url":"#b-1212fd27cb18111e8c062864","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-f03a8750ffafce82451ec11a","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-f03a8750ffafce82451ec11a","content":"- [CLIP 从图文对走到候选接口｜核心改动怎样工作](/supplement.html?return=l3-r2-p57/0#/l3-clip-full-chain-mechanism/0) 本页说明","sha256":"caa8ea483f5051a9f145a1a981ca2d77ad289e9d05c7b4c5bf618dd2738c48da","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-9ebe3ed1a4c8a1ade1c6cffe","nextBlock":"b-5c906026e32edb5171d2a3c9","prerequisites":[{"id":"b-1212fd27cb18111e8c062864","url":"#b-1212fd27cb18111e8c062864","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-5c906026e32edb5171d2a3c9","type":"li","scope":"retained-reading","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-5c906026e32edb5171d2a3c9","content":"- [CLIP 从图文对走到候选接口｜一起算一个完整例子](/supplement.html?return=l3-r2-p57/0#/l3-clip-full-chain-example/0) 本页说明","sha256":"e184e0f8bb11c3a8463f4e106cf928fd98498661eaaedfa98f9404b6e87707c3","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-pairwise-matrix","title":"每一行选文字，每一列选图像","number":1},"sectionTitle":"每一行选文字，每一列选图像","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-f03a8750ffafce82451ec11a","nextBlock":null,"prerequisites":[{"id":"b-1212fd27cb18111e8c062864","url":"#b-1212fd27cb18111e8c062864","role":"section-title"},{"id":"b-28a5e64a1e04900c6ad3efe8","url":"#b-28a5e64a1e04900c6ad3efe8","role":"problem-context"}],"relatedBlocks":[{"id":"b-58808538e43ff4afd70cde8e","url":"#b-58808538e43ff4afd70cde8e","relation":"figure-and-explanation"}]},{"id":"b-d2b28f664355de7b7eab161a","type":"h2","scope":"clip-bidirectional-loss","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-d2b28f664355de7b7eab161a","content":"## 由一行和一列算出双向训练目标","sha256":"68dd206cc3269f0042575db3ea75e90b3eaaea02500544c65e2dbf1601dadf3c","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-73b1943bb124b2a3e249ec2e","prerequisites":[{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-73b1943bb124b2a3e249ec2e","type":"p","scope":"clip-bidirectional-loss","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-73b1943bb124b2a3e249ec2e","content":"上一页的矩阵给出候选关系；现在沿第一行和第一列计算正确配对的概率。","sha256":"c314c9bd40899a5579028c20fc56107fc3f32a28d562a2a66a0c0da7e66ad170","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d2b28f664355de7b7eab161a","nextBlock":"b-e16e097faf7af6d5f655a124","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"}],"relatedBlocks":[]},{"id":"b-e16e097faf7af6d5f655a124","type":"p","scope":"clip-loss-symbols","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-e16e097faf7af6d5f655a124","content":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。","sha256":"da150caaea50d6f73aefa7d50b1a5ac82aa91b20fabe24fcf8b7f3354dbfac7e","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-73b1943bb124b2a3e249ec2e","nextBlock":"b-04e8944a674e7c5c99b581dd","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-04e8944a674e7c5c99b581dd","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-04e8944a674e7c5c99b581dd","content":"行表示图找文字，列表示文字找图。当前排序让正确配对位于对角线。取温度 $\\tau=.2$，第一行 logits 为 $(4,1.5,0)$，正确配对概率为","sha256":"5f5d03b18f32fe9ef45e0e47dc9459338e7af869590cf2a4f35f7c07b37ee552","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"\\tau=.2","definitionContext":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。"},{"latex":"(4,1.5,0)","definitionContext":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。"}],"previousBlock":"b-e16e097faf7af6d5f655a124","nextBlock":"b-f028cf8cbefd2fd9433dc145","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[{"id":"b-f028cf8cbefd2fd9433dc145","url":"#b-f028cf8cbefd2fd9433dc145","relation":"same-reasoning"},{"id":"b-891c9f0a16da63982ca628a6","url":"#b-891c9f0a16da63982ca628a6","relation":"same-reasoning"},{"id":"b-1806238c1aa1403471791217","url":"#b-1806238c1aa1403471791217","relation":"same-reasoning"}]},{"id":"b-f028cf8cbefd2fd9433dc145","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-f028cf8cbefd2fd9433dc145","content":"$p_{11}=\\frac{e^4}{e^4+e^{1.5}+1}=0.9087599243,\\qquad -\\log p_{11}=0.0956743294.$","sha256":"7cb63b04e1a0edf44bbd40f7be4f6471f882de0e7485d96ca394a8b321aeb6cb","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"p_{11}=\\frac{e^4}{e^4+e^{1.5}+1}=0.9087599243,\\qquad -\\log p_{11}=0.0956743294.","definitionContext":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。"}],"previousBlock":"b-04e8944a674e7c5c99b581dd","nextBlock":"b-891c9f0a16da63982ca628a6","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[{"id":"b-04e8944a674e7c5c99b581dd","url":"#b-04e8944a674e7c5c99b581dd","relation":"same-reasoning"},{"id":"b-891c9f0a16da63982ca628a6","url":"#b-891c9f0a16da63982ca628a6","relation":"same-reasoning"},{"id":"b-1806238c1aa1403471791217","url":"#b-1806238c1aa1403471791217","relation":"same-reasoning"}]},{"id":"b-891c9f0a16da63982ca628a6","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-891c9f0a16da63982ca628a6","content":"第一列只是将两个负项换序，概率不变；其他行列同理。双向平均损失为 $L=(L_{I\\to T}+L_{T\\to I})/2=0.0956743294$。一般情况下每行每列并不同分，必须分别计算，不能从一个最好看的单元格推总损失。","sha256":"997240b500a1857825d9d5e874382060b832ca04e5892cf1a198efaf8a4247a5","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"L=(L_{I\\to T}+L_{T\\to I})/2=0.0956743294","definitionContext":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。"}],"previousBlock":"b-f028cf8cbefd2fd9433dc145","nextBlock":"b-1806238c1aa1403471791217","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[{"id":"b-04e8944a674e7c5c99b581dd","url":"#b-04e8944a674e7c5c99b581dd","relation":"same-reasoning"},{"id":"b-f028cf8cbefd2fd9433dc145","url":"#b-f028cf8cbefd2fd9433dc145","relation":"same-reasoning"},{"id":"b-1806238c1aa1403471791217","url":"#b-1806238c1aa1403471791217","relation":"same-reasoning"}]},{"id":"b-1806238c1aa1403471791217","type":"p","scope":"clip-matrix-worked","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-1806238c1aa1403471791217","content":"优化关注正确配对相对于其他候选的概率，不是要求每个正样本点积都变成任意大的数。单位向量点积本身受限于 $[-1,1]$，温度缩放改变 softmax 的区分程度。更换批次会更换竞争候选；两段都适合狗图的文字也可能在配对表里互为负项，因此学习目标与语义真值并不完全重合。","sha256":"003f4000ef5478872af45c630b0aee2f6ce26c9522cbfa443a8343611fd3f829","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":"clip-matrix-worked","formulas":[{"latex":"[-1,1]","definitionContext":"图像找文字：第一行 S₁· = (0.8, 0.3, 0)；文字找图像：第一列 S·₁ = (0.8, 0, 0.3)。S 是归一化向量点积，τ 是温度；logits = S / τ，p 表示本批候选内的相对分配，L 是负对数损失。"}],"previousBlock":"b-891c9f0a16da63982ca628a6","nextBlock":"b-3c5bd63187b5fb33f605416f","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[{"id":"b-04e8944a674e7c5c99b581dd","url":"#b-04e8944a674e7c5c99b581dd","relation":"same-reasoning"},{"id":"b-f028cf8cbefd2fd9433dc145","url":"#b-f028cf8cbefd2fd9433dc145","relation":"same-reasoning"},{"id":"b-891c9f0a16da63982ca628a6","url":"#b-891c9f0a16da63982ca628a6","relation":"same-reasoning"}]},{"id":"b-3c5bd63187b5fb33f605416f","type":"h3","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-3c5bd63187b5fb33f605416f","content":"### 沿着图解追问","sha256":"f16be0424762a44344338696f06ef266c4571d15367ae2cbbd28fac98ac7e9cb","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1806238c1aa1403471791217","nextBlock":"b-39a1d7c7584b8ec2897633e9","prerequisites":[{"id":"b-d2b28f664355de7b7eab161a","url":"#b-d2b28f664355de7b7eab161a","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-39a1d7c7584b8ec2897633e9","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-39a1d7c7584b8ec2897633e9","content":"以下是备课用的模拟师生对话，不是课堂实录。","sha256":"b25685f04cd1b885f3a5999e6a912460c7424021cdc5b2190584242086a82681","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3c5bd63187b5fb33f605416f","nextBlock":"b-3a6c82b2b59d5f8ccf1e2b00","prerequisites":[{"id":"b-3c5bd63187b5fb33f605416f","url":"#b-3c5bd63187b5fb33f605416f","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-3a6c82b2b59d5f8ccf1e2b00","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-3a6c82b2b59d5f8ccf1e2b00","content":"**学生：**温度调小就更自信，为什么不直接调到零？","sha256":"0ccb583b2449578582da9ec16ffde2fb6d7ad0c23b5ccc9bb6751309b7bf4ee1","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-39a1d7c7584b8ec2897633e9","nextBlock":"b-61d6558c0f15006f3f702d63","prerequisites":[{"id":"b-3c5bd63187b5fb33f605416f","url":"#b-3c5bd63187b5fb33f605416f","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-61d6558c0f15006f3f702d63","type":"p","scope":"dialogue","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-61d6558c0f15006f3f702d63","content":"**教师：**它会把当前最大项放大，但当前最大项也可能错；训练梯度和优化也随之变化。把概率变尖不等于学到更正确的对应。","sha256":"ab974e2406f01db64e6ece20c6d7ec195f6976f99a8db5ad85894d9236fd945e","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-3a6c82b2b59d5f8ccf1e2b00","nextBlock":"b-24b83262d47705b8e790b2f5","prerequisites":[{"id":"b-3c5bd63187b5fb33f605416f","url":"#b-3c5bd63187b5fb33f605416f","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-24b83262d47705b8e790b2f5","type":"h2","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-24b83262d47705b8e790b2f5","content":"## 沿这个问题继续","sha256":"cf44fcc62782fb4f6bdc42c16c5187d57556a03cb7ce6065e7bd9c36d8e7d733","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-61d6558c0f15006f3f702d63","nextBlock":"b-6777ab3f6f978555d3e321ed","prerequisites":[{"id":"b-3c5bd63187b5fb33f605416f","url":"#b-3c5bd63187b5fb33f605416f","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-6777ab3f6f978555d3e321ed","type":"p","scope":"practice","url":"https://codingai-lec03.pages.dev/course/pages/l3-r2-p57.html#b-6777ab3f6f978555d3e321ed","content":"核对三个文字向量的长度，再算第一行与第一列的正确配对概率及负对数。检查为什么本例各行各列的平均损失相同，再到 P58 改变候选集合。","sha256":"932121d2b767733815b139c5ad8a869da1bdb823cf0f9adeee781a2e53e5045a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"clip-pairing-to-candidates","title":"已知图文配对怎样变成可用的文字候选接口？"},"page":{"id":"clip-bidirectional-loss","title":"由一行和一列算出双向训练目标","number":2},"sectionTitle":"由一行和一列算出双向训练目标","currentQuestion":"从一对输入到双向损失，再到真实候选干预，哪些量由训练决定、哪些由使用者决定？","sourceSlides":["l3-r2-p57"],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-24b83262d47705b8e790b2f5","nextBlock":null,"prerequisites":[{"id":"b-24b83262d47705b8e790b2f5","url":"#b-24b83262d47705b8e790b2f5","role":"section-title"},{"id":"b-73b1943bb124b2a3e249ec2e","url":"#b-73b1943bb124b2a3e249ec2e","role":"problem-context"}],"relatedBlocks":[]}]}