{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html","markdown":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.md","pages":[{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","unit":"语言怎样选择当前需要的视觉信息？","unitId":"language-local-and-question","chapter":"III","number":1}],"aliases":[{"oldId":"b-a6b8615fa7bde84bc3d33c7e","targetId":"b-a6b8615fa7bde84bc3d33c7e","page":"reading-opening","operation":"move","beforeSHA256":"4d2b86967d673c8a0192862944467c48e0eadfedf077e1b6a7bee68e9a51953b","afterSHA256":"4d2b86967d673c8a0192862944467c48e0eadfedf077e1b6a7bee68e9a51953b","reason":"Same source body in authored semantic page"},{"oldId":"b-2c52970711d30930bcf4fe86","targetId":"b-2c52970711d30930bcf4fe86","page":"reading-opening","operation":"move","beforeSHA256":"24e1538a82758ae32abd530853b95a844b61c3dba2d9c3c045a7eabdfd4bc917","afterSHA256":"24e1538a82758ae32abd530853b95a844b61c3dba2d9c3c045a7eabdfd4bc917","reason":"Same source body in authored semantic page"},{"oldId":"b-098cdfc686bee4d5f5fdf5e9","targetId":"b-098cdfc686bee4d5f5fdf5e9","page":"reading-opening","operation":"move","beforeSHA256":"6b66eb7026936a6211fd1af69e04f8e1bda430ac6e49121d19342d95c3196ff2","afterSHA256":"6b66eb7026936a6211fd1af69e04f8e1bda430ac6e49121d19342d95c3196ff2","reason":"Same source body in authored semantic page"},{"oldId":"b-6be161d27f1cc376638c7446","targetId":"b-6be161d27f1cc376638c7446","page":"reading-opening","operation":"move","beforeSHA256":"140b7f8b1fcac0e0fb869e4d00ebf68b40c5c5c9b147974f227e50a1bd3825d6","afterSHA256":"140b7f8b1fcac0e0fb869e4d00ebf68b40c5c5c9b147974f227e50a1bd3825d6","reason":"Same source body in authored semantic page"},{"oldId":"b-bc312405cd75afde3023aab3","targetId":"b-bc312405cd75afde3023aab3","page":"reading-opening","operation":"move","beforeSHA256":"934be7ba01b3582844964e06b9e11c3888072baf4180f5e80ab0499fd7d72781","afterSHA256":"934be7ba01b3582844964e06b9e11c3888072baf4180f5e80ab0499fd7d72781","reason":"Same source body in authored semantic page"},{"oldId":"b-e9506fa4619d9e85027e4e5f","targetId":"b-e9506fa4619d9e85027e4e5f","page":"reading-opening","operation":"move","beforeSHA256":"81c56d5a04c99fa7229181969b48cab83009bf34a15ef233d494188dab00cf8a","afterSHA256":"81c56d5a04c99fa7229181969b48cab83009bf34a15ef233d494188dab00cf8a","reason":"Same source body in authored semantic page"},{"oldId":"b-683ffaed60afa62a6031ede3","targetId":"b-683ffaed60afa62a6031ede3","page":"reading-opening","operation":"rewrite","beforeSHA256":"f04dfeef2663e182e5fe7911457b8e48fbd8c9a648545ae3b0106956c5b3e299","afterSHA256":"1f748f92ef4a939c7bd6242fe9218fc0b47f32598759ccc6d75a6aafb20b0fc4","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-aa56337b9de28f7e52c44b32","targetId":"b-aa56337b9de28f7e52c44b32","page":"reading-opening","operation":"move","beforeSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","afterSHA256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","reason":"Same source body in authored semantic page"}],"blocks":[{"id":"b-a6b8615fa7bde84bc3d33c7e","type":"h1","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-a6b8615fa7bde84bc3d33c7e","content":"# ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sha256":"4d2b86967d673c8a0192862944467c48e0eadfedf077e1b6a7bee68e9a51953b","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-2c52970711d30930bcf4fe86","prerequisites":[{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-2c52970711d30930bcf4fe86","type":"p","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-2c52970711d30930bcf4fe86","content":"ViLBERT 让前面两部分的积累变得具体：检测器提供区域表示，语言预训练提供文字表示，新的预训练目标学习二者怎样共同服务任务。读本页不需要展开 attention；重点是指出可复用组件、尚未学会的对应以及跨任务迁移的研究证据。","sha256":"24e1538a82758ae32abd530853b95a844b61c3dba2d9c3c045a7eabdfd4bc917","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-a6b8615fa7bde84bc3d33c7e","nextBlock":"b-098cdfc686bee4d5f5fdf5e9","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-098cdfc686bee4d5f5fdf5e9","type":"p","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-098cdfc686bee4d5f5fdf5e9","content":"第二部分的检测器不只留下“这里有杯子”的类别输出，也能留下每个候选区域的特征。语言预训练同样留下文字表示。ViLBERT 将两侧的已有能力接入跨模态预训练，再适配问答、指代表达、常识推理与图文检索等任务。","sha256":"6b66eb7026936a6211fd1af69e04f8e1bda430ac6e49121d19342d95c3196ff2","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-2c52970711d30930bcf4fe86","nextBlock":"b-6be161d27f1cc376638c7446","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-6be161d27f1cc376638c7446","type":"p","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-6be161d27f1cc376638c7446","content":"为什么不直接拼起来就结束？设区域向量来自检测器、词向量来自文字模型。两个向量长度可以相同，但相同坐标号并不自动代表同一种语义。跨模态目标要学习哪些局部与哪些文字对应、哪些信息可以共同预测。维度匹配只让计算合法，训练才使这条关系有任务含义。","sha256":"140b7f8b1fcac0e0fb869e4d00ebf68b40c5c5c9b147974f227e50a1bd3825d6","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-098cdfc686bee4d5f5fdf5e9","nextBlock":"b-bc312405cd75afde3023aab3","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-bc312405cd75afde3023aab3","type":"p","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-bc312405cd75afde3023aab3","content":"原论文通过代理预训练任务学习联合表示，再对四类视觉语言任务做少量结构增补与任务适配，报告了相对当时任务专用模型的改善。它支持的研究方向是：区域—语言对应可以成为预训练并迁移的能力，未必每个下游任务都从头建立全部对应关系。这里保留这一正面进展，不展开内部注意力实现。","sha256":"934be7ba01b3582844964e06b9e11c3888072baf4180f5e80ab0499fd7d72781","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-6be161d27f1cc376638c7446","nextBlock":"b-e9506fa4619d9e85027e4e5f","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-e9506fa4619d9e85027e4e5f","type":"p","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-e9506fa4619d9e85027e4e5f","content":"阅读图时注意“可复用”有两个尺度：检测器区域特征作为输入被复用，跨模态预训练后的表示又被多个任务复用。二者都不是说下游无需目标数据或评价。若关心小字、细小对象或未被候选器提出的区域，首先还要检查最前面的视觉输入是否已经丢失所需证据。","sha256":"81c56d5a04c99fa7229181969b48cab83009bf34a15ef233d494188dab00cf8a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-bc312405cd75afde3023aab3","nextBlock":"b-683ffaed60afa62a6031ede3","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]},{"id":"b-683ffaed60afa62a6031ede3","type":"figure","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-683ffaed60afa62a6031ede3","content":"图内标签与数值：区域视觉输入；检测器留下的区域特征；语言输入；“红衣人拿着球”；预训练的文字表示；ViLBERT · 2019；跨模态目标学习对应；图文匹配；遮挡内容预测；适配问答与检索等任务；检测器区域特征与预训练语言表示是实际组件来源；此页仅呈现模块接口。\n\n两类预训练组件汇入新任务；连线表示可复用接口，并不表示无需跨模态学习。 查看对应图解状态\n\n- 区域视觉输入\n- 检测器留下的区域特征\n- 语言输入\n- “红衣人拿着球”\n- 预训练的文字表示\n- ViLBERT · 2019\n- 跨模态目标学习对应\n- 图文匹配\n- 遮挡内容预测\n- 适配问答与检索等任务\n- 检测器区域特征与预训练语言表示是实际组件来源；此页仅呈现模块接口。两类预训练组件汇入新任务；连线表示可复用接口，并不表示无需跨模态学习。 [查看对应图解状态](/#/l3-r2-p55/0)[查看完整图解](/#/l3-r2-p55/0)","sha256":"1f748f92ef4a939c7bd6242fe9218fc0b47f32598759ccc6d75a6aafb20b0fc4","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"figure":{"caption":"两类预训练组件汇入新任务；连线表示可复用接口，并不表示无需跨模态学习。 查看对应图解状态","labels":["区域视觉输入","检测器留下的区域特征","语言输入","“红衣人拿着球”","预训练的文字表示","ViLBERT · 2019","跨模态目标学习对应","图文匹配","遮挡内容预测","适配问答与检索等任务","检测器区域特征与预训练语言表示是实际组件来源；此页仅呈现模块接口。"],"description":"两类预训练组件汇入新任务；连线表示可复用接口，并不表示无需跨模态学习。 查看对应图解状态区域视觉输入检测器留下的区域特征语言输入“红衣人拿着球”预训练的文字表示ViLBERT · 2019跨模态目标学习对应图文匹配遮挡内容预测适配问答与检索等任务检测器区域特征与预训练语言表示是实际组件来源；此页仅呈现模块接口。","evidenceIdentity":"两类预训练组件汇入新任务；连线表示可复用接口，并不表示无需跨模态学习。 查看对应图解状态","images":[{"src":"/assets/display/512610a019c560ff0384e86a.webp","alt":""}]},"previousBlock":"b-e9506fa4619d9e85027e4e5f","nextBlock":"b-aa56337b9de28f7e52c44b32","prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-aa56337b9de28f7e52c44b32","type":"h2","scope":"vilbert-components-and-transfer","url":"https://codingai-lec03.pages.dev/course/blocks/vilbert-components-and-transfer.html#b-aa56337b9de28f7e52c44b32","content":"## 继续阅读与实验","sha256":"2b9ccef7e1220e2250cceff1abd8bdd736b42f8da5a223f26b029f44b962e807","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-local-and-question","title":"语言怎样选择当前需要的视觉信息？"},"page":{"id":"reading-opening","title":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","number":1},"sectionTitle":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","currentQuestion":"ViLBERT：把已有区域和文字表示变成可迁移的联合起点","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-683ffaed60afa62a6031ede3","nextBlock":null,"prerequisites":[{"id":"b-a6b8615fa7bde84bc3d33c7e","url":"#b-a6b8615fa7bde84bc3d33c7e","role":"section-title"},{"id":"b-2c52970711d30930bcf4fe86","url":"#b-2c52970711d30930bcf4fe86","role":"problem-context"}],"relatedBlocks":[{"id":"b-683ffaed60afa62a6031ede3","url":"#b-683ffaed60afa62a6031ede3","relation":"figure-and-explanation"}]}]}