{"version":2,"relationshipURLBase":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html","relationshipResolution":"Resolve relative relationship URLs against relationshipURLBase; complete records and absolute URLs are in blocks.","title":"CLS 09 · 图像条件输出：描述与问答","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html","markdown":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.md","pages":[{"id":"reading-opening","title":"CLS 09 · 图像条件输出：描述与问答","unit":"语言加入后，任务究竟改变了什么？","unitId":"language-task-interfaces","chapter":"III","number":1},{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","unit":"语言加入后，任务究竟改变了什么？","unitId":"language-task-interfaces","chapter":"III","number":2},{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","unit":"语言加入后，任务究竟改变了什么？","unitId":"language-task-interfaces","chapter":"III","number":3},{"id":"reading-block-0bf24433979946a493db9110cc38c355","title":"看清图像条件，再打开解码器","unit":"语言加入后，任务究竟改变了什么？","unitId":"language-task-interfaces","chapter":"III","number":4}],"aliases":[{"oldId":"b-545a812e16c11629dc634df7","targetId":"b-545a812e16c11629dc634df7","page":"reading-opening","operation":"move","beforeSHA256":"0869a21a5e70ead026f87cc0ba7340e1480c2d98d71496705a4271b274042716","afterSHA256":"0869a21a5e70ead026f87cc0ba7340e1480c2d98d71496705a4271b274042716","reason":"Same source body in authored semantic page"},{"oldId":"b-992a4502ade6b86be1976c28","targetId":"b-992a4502ade6b86be1976c28","page":"reading-opening","operation":"move","beforeSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","afterSHA256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","reason":"Same source body in authored semantic page"},{"oldId":"b-52e742adfa875c2ba8f687f2","targetId":"b-52e742adfa875c2ba8f687f2","page":"reading-opening","operation":"move","beforeSHA256":"a192bf58b03c510999643a276b8d2261f6c9389ea09aebd6ad3419fe70e2a7b9","afterSHA256":"a192bf58b03c510999643a276b8d2261f6c9389ea09aebd6ad3419fe70e2a7b9","reason":"Same source body in authored semantic page"},{"oldId":"b-a1fcf789d45a61ddcfab143a","targetId":"b-a1fcf789d45a61ddcfab143a","page":"reading-opening","operation":"move","beforeSHA256":"f22152fd55dcdb0090b5653d37145694fd19388d31e10226549464fe8b483578","afterSHA256":"f22152fd55dcdb0090b5653d37145694fd19388d31e10226549464fe8b483578","reason":"Same source body in authored semantic page"},{"oldId":"b-dbcc0044a38f921fbb8b62f6","targetId":"b-dbcc0044a38f921fbb8b62f6","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"a57513e59eb07960710e5ece7951cc57af4a94c446630eb5fc6d70b460982042","afterSHA256":"a57513e59eb07960710e5ece7951cc57af4a94c446630eb5fc6d70b460982042","reason":"Same source body in authored semantic page"},{"oldId":"b-90eed75bfbdf593dd35ef9f5","targetId":"b-90eed75bfbdf593dd35ef9f5","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"6b3101dc74d613fd527b60ff21d1ac65db81cca6fc49d581090c9a90654d2057","afterSHA256":"6b3101dc74d613fd527b60ff21d1ac65db81cca6fc49d581090c9a90654d2057","reason":"Same source body in authored semantic page"},{"oldId":"b-cacda570991051b81cd224ff","targetId":"b-cacda570991051b81cd224ff","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"98588b7069ba7f92211a8e6cd5c8be5f8249ccf2425e964a426643a339f7a008","afterSHA256":"98588b7069ba7f92211a8e6cd5c8be5f8249ccf2425e964a426643a339f7a008","reason":"Same source body in authored semantic page"},{"oldId":"b-d1239837c351f441ab18cf47","targetId":"b-d1239837c351f441ab18cf47","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"46d053d1cc569605d484255e6af2c3466f7d40f4fe2b28cec955d4de4a9dd9d8","afterSHA256":"46d053d1cc569605d484255e6af2c3466f7d40f4fe2b28cec955d4de4a9dd9d8","reason":"Same source body in authored semantic page"},{"oldId":"b-aa5f96f678a8c7034c8511f7","targetId":"b-aa5f96f678a8c7034c8511f7","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"5d04e11d8c2e016f93a6fd003c5aafbcfa45431a430ae351461bb69f38e77a4d","afterSHA256":"5d04e11d8c2e016f93a6fd003c5aafbcfa45431a430ae351461bb69f38e77a4d","reason":"Same source body in authored semantic page"},{"oldId":"b-54ea7203d3bd45ea5bf220f1","targetId":"b-54ea7203d3bd45ea5bf220f1","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"621909530fe228f941698b8aa1acd5e19c591377be6a124d71ff93cc33cf1036","afterSHA256":"621909530fe228f941698b8aa1acd5e19c591377be6a124d71ff93cc33cf1036","reason":"Same source body in authored semantic page"},{"oldId":"b-b543e5441fa6c99ff40df691","targetId":"b-b543e5441fa6c99ff40df691","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"d3f6a0e9d3a04e431b7a9c6faca644a0357974e570ae3eea66067da85832db98","afterSHA256":"d3f6a0e9d3a04e431b7a9c6faca644a0357974e570ae3eea66067da85832db98","reason":"Same source body in authored semantic page"},{"oldId":"b-8d687fd72a5924c1cb2fcc08","targetId":"b-8d687fd72a5924c1cb2fcc08","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"132ec47bbd37bd8b6573626070e54b3184a5c5a21294a10bd9ab4c419181acc7","afterSHA256":"132ec47bbd37bd8b6573626070e54b3184a5c5a21294a10bd9ab4c419181acc7","reason":"Same source body in authored semantic page"},{"oldId":"b-9e2e2e7cfdf34a7c5c701710","targetId":"b-9e2e2e7cfdf34a7c5c701710","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"9e28174c8843e3c00738239016bd81d74ee8e3b5857fa1ff02121cc770476f86","afterSHA256":"9e28174c8843e3c00738239016bd81d74ee8e3b5857fa1ff02121cc770476f86","reason":"Same source body in authored semantic page"},{"oldId":"b-54e3a6a10af5faefa398df12","targetId":"b-54e3a6a10af5faefa398df12","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"18fc193179c81a0c886864470e3271551a6d8a3537b9fc9d935e5490e732aadb","afterSHA256":"18fc193179c81a0c886864470e3271551a6d8a3537b9fc9d935e5490e732aadb","reason":"Same source body in authored semantic page"},{"oldId":"b-0601bbb027c5b61284e112fd","targetId":"b-0601bbb027c5b61284e112fd","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"70da78e6d74f17d276ba01f6377f5f483608d6e486db66f7ffab4af473c8c68d","afterSHA256":"70da78e6d74f17d276ba01f6377f5f483608d6e486db66f7ffab4af473c8c68d","reason":"Same source body in authored semantic page"},{"oldId":"b-6d7fc7084235a9a89a221e1a","targetId":"b-6d7fc7084235a9a89a221e1a","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"6bca41597071d63a25f16e927be8d8382b5da7c125bd9f583e5399e0520cc939","afterSHA256":"6bca41597071d63a25f16e927be8d8382b5da7c125bd9f583e5399e0520cc939","reason":"Same source body in authored semantic page"},{"oldId":"b-97812c661aa5e556f3edfe6d","targetId":"b-97812c661aa5e556f3edfe6d","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"rewrite","beforeSHA256":"a77ee9b5843b0305eb36f4a9cad8be08c2b7708ae228d53535a9946b02f8b488","afterSHA256":"76596d43d047ab4ae6858c71f16377d45d925bd1b3cb8a695d4f1606a5d9cbc1","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-c4d227f6b3a22d76e878857f","targetId":"b-c4d227f6b3a22d76e878857f","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"e85602c65ca5ff78574d827dfabe48bbd83b4b0913f2f7b07f928129a1cd5593","afterSHA256":"5fad46273be0df138b5d8cd83825e637a26e4d2054efd42ac82536606c85cd42","reason":"Same source body in authored semantic page"},{"oldId":"b-247cdaac3f84d64511c0d7b0","targetId":"b-247cdaac3f84d64511c0d7b0","page":"reading-block-dd1046d546e24b26aecee5957553a6e2","operation":"move","beforeSHA256":"54a440266f441a2ca24062791492c56d30a26070e099032b39ed5da18c6879ea","afterSHA256":"54a440266f441a2ca24062791492c56d30a26070e099032b39ed5da18c6879ea","reason":"Same source body in authored semantic page"},{"oldId":"b-9a075058f8bd5c4238ce17f9","targetId":"b-9a075058f8bd5c4238ce17f9","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"7cd1087c34e727cf26f3c9f759ef1904c52a96c60228e8922bcb000cfa7dc8d8","afterSHA256":"7cd1087c34e727cf26f3c9f759ef1904c52a96c60228e8922bcb000cfa7dc8d8","reason":"Same source body in authored semantic page"},{"oldId":"b-68c356ce73e4c6c12ce5f828","targetId":"b-68c356ce73e4c6c12ce5f828","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"efb06658a2c54105750282521a6f6c76787166ee696d375661365ac389ae0fa7","afterSHA256":"efb06658a2c54105750282521a6f6c76787166ee696d375661365ac389ae0fa7","reason":"Same source body in authored semantic page"},{"oldId":"b-df3d2acfc44787cd8aff78bb","targetId":"b-df3d2acfc44787cd8aff78bb","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"831626528161a3c074e9863fbc320f83c4427099d962c088e152ac90ac1b36db","afterSHA256":"831626528161a3c074e9863fbc320f83c4427099d962c088e152ac90ac1b36db","reason":"Same source body in authored semantic page"},{"oldId":"b-25f9d762ba22c68c6e472610","targetId":"b-25f9d762ba22c68c6e472610","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"7b74cc3e2f3205469e80f80e2972062d26a1c35de2e0f355d3b31ebd4564d1ad","afterSHA256":"7b74cc3e2f3205469e80f80e2972062d26a1c35de2e0f355d3b31ebd4564d1ad","reason":"Same source body in authored semantic page"},{"oldId":"b-8bc5187c6d6e053e54568278","targetId":"b-8bc5187c6d6e053e54568278","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"rewrite","beforeSHA256":"b2f1ae3ddd998d2fd4d04045f73e8ac1a983afde542afa752a36e3866cda4682","afterSHA256":"c029a7f961a8f3f4fc6629058322f9734e26c4f0f26ae8f34ba8cb501a091ec6","reason":"Same source identity with figure semantics or reading context enriched"},{"oldId":"b-c50022dae8d35de55cf5d292","targetId":"b-c50022dae8d35de55cf5d292","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"dd655aba933eeca3efa133773efe24b93715ba055621e0986f10caa85cd381e9","afterSHA256":"08d0aa022e98bb75079211b073bb26e8245e159a9c4dcb17e3839b2d504f1d23","reason":"Same source body in authored semantic page"},{"oldId":"b-afa27b1bd817e6033b7b07de","targetId":"b-afa27b1bd817e6033b7b07de","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"707e7bc48c07443cf12149644f705a6c7c56301b0b49bf4524b81dbab3c92e29","afterSHA256":"707e7bc48c07443cf12149644f705a6c7c56301b0b49bf4524b81dbab3c92e29","reason":"Same source body in authored semantic page"},{"oldId":"b-115dd44545e326043a75a14a","targetId":"b-115dd44545e326043a75a14a","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"3db3903d93095f36602506f7c9a92536b7083ddc36b16b3535a3e4f55a1cdff6","afterSHA256":"3db3903d93095f36602506f7c9a92536b7083ddc36b16b3535a3e4f55a1cdff6","reason":"Same source body in authored semantic page"},{"oldId":"b-3db8998813fc2191bcecb9ba","targetId":"b-3db8998813fc2191bcecb9ba","page":"reading-block-1b53d32e9e684591be91385c30d46982","operation":"move","beforeSHA256":"3df86520780b10397874c31809ad84a890727b5848b6a2551cd77e3f4ffd45bd","afterSHA256":"3df86520780b10397874c31809ad84a890727b5848b6a2551cd77e3f4ffd45bd","reason":"Same source body in authored semantic page"},{"oldId":"b-1e9944a8f8b2071d6b668132","targetId":"b-1e9944a8f8b2071d6b668132","page":"reading-block-0bf24433979946a493db9110cc38c355","operation":"move","beforeSHA256":"998e4aa89e86386109534916c3365369ad33129026c1bad72b87fe9e4604612a","afterSHA256":"998e4aa89e86386109534916c3365369ad33129026c1bad72b87fe9e4604612a","reason":"Same source body in authored semantic page"},{"oldId":"b-44a3a02b2dac7e60ca1f5b82","targetId":"b-44a3a02b2dac7e60ca1f5b82","page":"reading-block-0bf24433979946a493db9110cc38c355","operation":"move","beforeSHA256":"e6f220aac2e44a39989ddf391bb915ed36cca0070722c1f760af3134ffc4f3d1","afterSHA256":"e6f220aac2e44a39989ddf391bb915ed36cca0070722c1f760af3134ffc4f3d1","reason":"Same source body in authored semantic page"},{"oldId":"b-2e1644b7a273314c1f3d037f","targetId":"b-2e1644b7a273314c1f3d037f","page":"reading-block-0bf24433979946a493db9110cc38c355","operation":"move","beforeSHA256":"bebca56938c02b0a6d803f51787a60adaa7a3695479f8eee343d1949bc01e733","afterSHA256":"bebca56938c02b0a6d803f51787a60adaa7a3695479f8eee343d1949bc01e733","reason":"Same source body in authored semantic page"},{"oldId":"block-dd1046d546e24b26aecee5957553a6e2--图像前缀目标三个位置分开看","targetId":"b-aa5f96f678a8c7034c8511f7","operation":"retain-source-identity"},{"oldId":"block-dd1046d546e24b26aecee5957553a6e2--训练与推理的分界发生在前缀","targetId":"b-8d687fd72a5924c1cb2fcc08","operation":"retain-source-identity"},{"oldId":"block-dd1046d546e24b26aecee5957553a6e2--已有实验确实完成了图像条件生成","targetId":"b-0601bbb027c5b61284e112fd","operation":"retain-source-identity"},{"oldId":"block-1b53d32e9e684591be91385c30d46982--vqa问题控制读什么图像控制答什么","targetId":"b-9a075058f8bd5c4238ce17f9","operation":"retain-source-identity"},{"oldId":"block-0bf24433979946a493db9110cc38c355--看清图像条件再打开解码器","targetId":"b-1e9944a8f8b2071d6b668132","operation":"retain-source-identity"}],"blocks":[{"id":"b-545a812e16c11629dc634df7","type":"h1","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-545a812e16c11629dc634df7","content":"# CLS 09 · 图像条件输出：描述与问答","sha256":"0869a21a5e70ead026f87cc0ba7340e1480c2d98d71496705a4271b274042716","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-opening","title":"CLS 09 · 图像条件输出：描述与问答","number":1},"sectionTitle":"CLS 09 · 图像条件输出：描述与问答","currentQuestion":"CLS 09 · 图像条件输出：描述与问答","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-992a4502ade6b86be1976c28","prerequisites":[{"id":"b-992a4502ade6b86be1976c28","url":"#b-992a4502ade6b86be1976c28","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-992a4502ade6b86be1976c28","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-992a4502ade6b86be1976c28","content":"按问题继续","sha256":"e91f24e4e4ca40e2099a0d81d30db6568f4bf2074b499eeee6e3959617bd97ac","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-opening","title":"CLS 09 · 图像条件输出：描述与问答","number":1},"sectionTitle":"CLS 09 · 图像条件输出：描述与问答","currentQuestion":"CLS 09 · 图像条件输出：描述与问答","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-545a812e16c11629dc634df7","nextBlock":"b-52e742adfa875c2ba8f687f2","prerequisites":[{"id":"b-545a812e16c11629dc634df7","url":"#b-545a812e16c11629dc634df7","role":"section-title"}],"relatedBlocks":[]},{"id":"b-52e742adfa875c2ba8f687f2","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-52e742adfa875c2ba8f687f2","content":"图像条件、teacher forcing、真实生成、greedy/beam与VQA控制","sha256":"a192bf58b03c510999643a276b8d2261f6c9389ea09aebd6ad3419fe70e2a7b9","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-opening","title":"CLS 09 · 图像条件输出：描述与问答","number":1},"sectionTitle":"CLS 09 · 图像条件输出：描述与问答","currentQuestion":"CLS 09 · 图像条件输出：描述与问答","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-992a4502ade6b86be1976c28","nextBlock":"b-a1fcf789d45a61ddcfab143a","prerequisites":[{"id":"b-545a812e16c11629dc634df7","url":"#b-545a812e16c11629dc634df7","role":"section-title"},{"id":"b-992a4502ade6b86be1976c28","url":"#b-992a4502ade6b86be1976c28","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-a1fcf789d45a61ddcfab143a","type":"p","scope":"document","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-a1fcf789d45a61ddcfab143a","content":"可先阅读已有结果，再决定是否运行。重算时请核对各节记录的数据、参数、环境与日期。","sha256":"f22152fd55dcdb0090b5653d37145694fd19388d31e10226549464fe8b483578","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-opening","title":"CLS 09 · 图像条件输出：描述与问答","number":1},"sectionTitle":"CLS 09 · 图像条件输出：描述与问答","currentQuestion":"CLS 09 · 图像条件输出：描述与问答","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-52e742adfa875c2ba8f687f2","nextBlock":null,"prerequisites":[{"id":"b-545a812e16c11629dc634df7","url":"#b-545a812e16c11629dc634df7","role":"section-title"},{"id":"b-992a4502ade6b86be1976c28","url":"#b-992a4502ade6b86be1976c28","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-dbcc0044a38f921fbb8b62f6","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-dbcc0044a38f921fbb8b62f6","content":"**学习路径：** [图像怎样约束句子与回答](/course/study/caption-vqa.html)","sha256":"a57513e59eb07960710e5ece7951cc57af4a94c446630eb5fc6d70b460982042","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-90eed75bfbdf593dd35ef9f5","prerequisites":[],"relatedBlocks":[]},{"id":"b-90eed75bfbdf593dd35ef9f5","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-90eed75bfbdf593dd35ef9f5","content":"从[P51 看图描述](/course/pages/l3-r2-p51.html)与[P54 视觉问答](/course/pages/l3-r2-p54.html)进入：同一视觉编码器怎样服务“生成一句话”和“针对问题回答”两种输出？","sha256":"6b3101dc74d613fd527b60ff21d1ac65db81cca6fc49d581090c9a90654d2057","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-dbcc0044a38f921fbb8b62f6","nextBlock":"b-cacda570991051b81cd224ff","prerequisites":[{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-cacda570991051b81cd224ff","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-cacda570991051b81cd224ff","content":"先读图像、文字前缀和下一词目标，再看VQA的同图换问/同问换图。按兴趣选择[LAB 13](https://deepnote.com/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/3da47f3ee3554ee0a36e0bb6a2fdc61f)或[LAB 14](https://deepnote.com/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/11d9f74871544a77b42024a9fe06b81a)；已有结果能先读，运行时各册独立从头执行。","sha256":"98588b7069ba7f92211a8e6cd5c8be5f8249ccf2425e964a426643a339f7a008","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-90eed75bfbdf593dd35ef9f5","nextBlock":"b-d1239837c351f441ab18cf47","prerequisites":[{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-d1239837c351f441ab18cf47","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-d1239837c351f441ab18cf47","content":"读完应能解释：为什么训练损失不能代替自由生成评价，为什么问题先验不能代替视觉证据。GRU/RNN 内部与 beam 搜索已移入[第五讲](https://deepnote.com/project/841ec8e7-e352-4840-b628-d3742b03d6d0/notebook/b1c8e2fd764441e8820b808fc40ec3cb?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=b1c8e2fd764441e8820b808fc40ec3cb&utm_term=get_notebook)，不是理解视觉条件的前置要求。","sha256":"46d053d1cc569605d484255e6af2c3466f7d40f4fe2b28cec955d4de4a9dd9d8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-cacda570991051b81cd224ff","nextBlock":"b-aa5f96f678a8c7034c8511f7","prerequisites":[{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-aa5f96f678a8c7034c8511f7","type":"h2","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-aa5f96f678a8c7034c8511f7","content":"## 图像、前缀、目标，三个位置分开看","sha256":"5d04e11d8c2e016f93a6fd003c5aafbcfa45431a430ae351461bb69f38e77a4d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-d1239837c351f441ab18cf47","nextBlock":"b-54ea7203d3bd45ea5bf220f1","prerequisites":[{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-54ea7203d3bd45ea5bf220f1","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-54ea7203d3bd45ea5bf220f1","content":"令图像为 $I$、句子为 $y_1,\\ldots,y_T$，模型学习\n$p(y_1,\\ldots,y_T\\mid I)=\\prod_t p(y_t\\mid I,y_{<t}).$\n视觉编码器把像素变成特征；文字预测器接收视觉特征和允许看到的文字；目标是下一词。课堂从“红衣人”后预测“拿着”，再从“红衣人拿着”后预测“球”。图上的目标词是人工参考，不是实测生成。","sha256":"621909530fe228f941698b8aa1acd5e19c591377be6a124d71ff93cc33cf1036","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"I","definitionContext":"图像、前缀、目标，三个位置分开看"},{"latex":"y_1,\\ldots,y_T","definitionContext":"图像、前缀、目标，三个位置分开看"},{"latex":"p(y_1,\\ldots,y_T\\mid I)=\\prod_t p(y_t\\mid I,y_{<t}).","definitionContext":"图像、前缀、目标，三个位置分开看"}],"previousBlock":"b-aa5f96f678a8c7034c8511f7","nextBlock":"b-b543e5441fa6c99ff40df691","prerequisites":[{"id":"b-aa5f96f678a8c7034c8511f7","url":"#b-aa5f96f678a8c7034c8511f7","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-b543e5441fa6c99ff40df691","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-b543e5441fa6c99ff40df691","content":"一图可以有“两人在球场”与“红衣人拿球”等不同真实描述。模型学习条件下的表达分布，任务再决定必须包含哪些内容。[完整信息流与多参考解释](/course/blocks/caption-conditioned-token.html)。","sha256":"d3f6a0e9d3a04e431b7a9c6faca644a0357974e570ae3eea66067da85832db98","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-54ea7203d3bd45ea5bf220f1","nextBlock":"b-8d687fd72a5924c1cb2fcc08","prerequisites":[{"id":"b-aa5f96f678a8c7034c8511f7","url":"#b-aa5f96f678a8c7034c8511f7","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-8d687fd72a5924c1cb2fcc08","type":"h2","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-8d687fd72a5924c1cb2fcc08","content":"## 训练与推理的分界发生在前缀","sha256":"132ec47bbd37bd8b6573626070e54b3184a5c5a21294a10bd9ab4c419181acc7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-b543e5441fa6c99ff40df691","nextBlock":"b-9e2e2e7cfdf34a7c5c701710","prerequisites":[{"id":"b-aa5f96f678a8c7034c8511f7","url":"#b-aa5f96f678a8c7034c8511f7","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-9e2e2e7cfdf34a7c5c701710","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-9e2e2e7cfdf34a7c5c701710","content":"参考 `[BOS,a,red,square,EOS]` 的输入是 `[BOS,a,red,square]`，目标是 `[a,red,square,EOS]`。PAD 仅为批量补齐，不计损失；EOS 是有效停止目标。损失按有效词求平均，不能把补齐格数也算入分母。","sha256":"9e28174c8843e3c00738239016bd81d74ee8e3b5857fa1ff02121cc770476f86","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-8d687fd72a5924c1cb2fcc08","nextBlock":"b-54e3a6a10af5faefa398df12","prerequisites":[{"id":"b-8d687fd72a5924c1cb2fcc08","url":"#b-8d687fd72a5924c1cb2fcc08","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-54e3a6a10af5faefa398df12","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-54e3a6a10af5faefa398df12","content":"训练时看到真实前缀叫 teacher forcing；推理必须接续自己的输出。先误说 blue 后，就要在 blue 条件下继续，不能改回参考 red。保留集 teacher-forced 交叉熵因此不能代替自由生成评价。[逐位置表与损失公式](/course/blocks/caption-shift-and-stop.html)。","sha256":"18fc193179c81a0c886864470e3271551a6d8a3537b9fc9d935e5490e732aadb","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-9e2e2e7cfdf34a7c5c701710","nextBlock":"b-0601bbb027c5b61284e112fd","prerequisites":[{"id":"b-8d687fd72a5924c1cb2fcc08","url":"#b-8d687fd72a5924c1cb2fcc08","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-0601bbb027c5b61284e112fd","type":"h2","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-0601bbb027c5b61284e112fd","content":"## 已有实验确实完成了图像条件生成","sha256":"70da78e6d74f17d276ba01f6377f5f483608d6e486db66f7ffab4af473c8c68d","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-54e3a6a10af5faefa398df12","nextBlock":"b-6d7fc7084235a9a89a221e1a","prerequisites":[{"id":"b-8d687fd72a5924c1cb2fcc08","url":"#b-8d687fd72a5924c1cb2fcc08","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-6d7fc7084235a9a89a221e1a","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-6d7fc7084235a9a89a221e1a","content":"[LAB 13](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/3da47f3ee3554ee0a36e0bb6a2fdc61f?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=3da47f3ee3554ee0a36e0bb6a2fdc61f&utm_term=get_notebook) 保存2026-09-22本地 CPU 基准：720张训练图、180张独立渲染保留图，18种语义组合两侧都出现。CNN产生48维视觉向量，文字模块采用小型GRU；本讲把它作为明确输入输出模块，内部留L05。推理不读取真实颜色/位置标签。","sha256":"6bca41597071d63a25f16e927be8d8382b5da7c125bd9f583e5399e0520cc939","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-0601bbb027c5b61284e112fd","nextBlock":"b-97812c661aa5e556f3edfe6d","prerequisites":[{"id":"b-0601bbb027c5b61284e112fd","url":"#b-0601bbb027c5b61284e112fd","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-97812c661aa5e556f3edfe6d","type":"table","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-97812c661aa5e556f3edfe6d","content":"| 保留条件 | 整句精确匹配 |\n| --- | --- |\n| 正确图像，greedy / beam=3 | 100% / 100% |\n| 同一模型，像素置零 | 5.56% |\n| 同一模型，循环错配图像 | 0% |","sha256":"76596d43d047ab4ae6858c71f16377d45d925bd1b3cb8a695d4f1606a5d9cbc1","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"table":{"headers":["保留条件","整句精确匹配"],"rows":[["保留条件","整句精确匹配"],["正确图像，greedy / beam=3","100% / 100%"],["同一模型，像素置零","5.56%"],["同一模型，循环错配图像","0%"]]},"previousBlock":"b-6d7fc7084235a9a89a221e1a","nextBlock":"b-c4d227f6b3a22d76e878857f","prerequisites":[{"id":"b-0601bbb027c5b61284e112fd","url":"#b-0601bbb027c5b61284e112fd","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-c4d227f6b3a22d76e878857f","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-c4d227f6b3a22d76e878857f","content":"![同一前缀、不同图像的真实生成](/assets/display/9ebe76d49a449b9940b6470f.webp)","sha256":"5fad46273be0df138b5d8cd83825e637a26e4d2054efd42ac82536606c85cd42","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-97812c661aa5e556f3edfe6d","nextBlock":"b-247cdaac3f84d64511c0d7b0","prerequisites":[{"id":"b-0601bbb027c5b61284e112fd","url":"#b-0601bbb027c5b61284e112fd","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-247cdaac3f84d64511c0d7b0","type":"p","scope":"block-dd1046d546e24b26aecee5957553a6e2","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-247cdaac3f84d64511c0d7b0","content":"固定 `BOS a`，红方块左侧图生成 `a red square on left`，蓝圆右侧图生成 `a blue circle on right`。这支持当前有限任务使用正确视觉条件；不是自然照片或未见组合泛化，也没有显示 beam 优于 greedy。本轮整理未重跑训练。[数据、协议与证据细节](/course/blocks/caption-visual-controls.html)。","sha256":"54a440266f441a2ca24062791492c56d30a26070e099032b39ed5da18c6879ea","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-dd1046d546e24b26aecee5957553a6e2","title":"图像、前缀、目标，三个位置分开看","number":2},"sectionTitle":"图像、前缀、目标，三个位置分开看","currentQuestion":"图像、前缀、目标，三个位置分开看","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c4d227f6b3a22d76e878857f","nextBlock":null,"prerequisites":[{"id":"b-0601bbb027c5b61284e112fd","url":"#b-0601bbb027c5b61284e112fd","role":"section-title"},{"id":"b-dbcc0044a38f921fbb8b62f6","url":"#b-dbcc0044a38f921fbb8b62f6","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-9a075058f8bd5c4238ce17f9","type":"h2","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-9a075058f8bd5c4238ce17f9","content":"## VQA：问题控制读什么，图像控制答什么","sha256":"7cd1087c34e727cf26f3c9f759ef1904c52a96c60228e8922bcb000cfa7dc8d8","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-68c356ce73e4c6c12ce5f828","prerequisites":[{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-68c356ce73e4c6c12ce5f828","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-68c356ce73e4c6c12ce5f828","content":"VQA建模 $p(a\\mid I,q)$。同图问数量、颜色、关系，需要不同信息；同一问题换相关图像，答案也应改变。输出可为有限答案分类，也可逐词生成，二者的答案范围和评价协议不同。","sha256":"efb06658a2c54105750282521a6f6c76787166ee696d375661365ac389ae0fa7","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[{"latex":"p(a\\mid I,q)","definitionContext":"VQA：问题控制读什么，图像控制答什么"}],"previousBlock":"b-9a075058f8bd5c4238ce17f9","nextBlock":"b-df3d2acfc44787cd8aff78bb","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"}],"relatedBlocks":[]},{"id":"b-df3d2acfc44787cd8aff78bb","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-df3d2acfc44787cd8aff78bb","content":"图A左红右蓝：问左色答红、问右色答蓝。换成图B左蓝右红，保持左色问题则应答蓝。若“视觉”分支只读取场景字典的 `left_color`，它是已知属性 oracle，不是从像素感知。[任务条件与VQA研究背景](/course/blocks/vqa-question-selects-information.html)。","sha256":"831626528161a3c074e9863fbc320f83c4427099d962c088e152ac90ac1b36db","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-68c356ce73e4c6c12ce5f828","nextBlock":"b-25f9d762ba22c68c6e472610","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-25f9d762ba22c68c6e472610","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-25f9d762ba22c68c6e472610","content":"[LAB 14](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/11d9f74871544a77b42024a9fe06b81a?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=11d9f74871544a77b42024a9fe06b81a&utm_term=get_notebook) 已有真正读取像素和问题ID的CNN问答，以及单独训练的question-only。2026-09-22本地保留180张图、每图三问共540图问对，先按图划分再扩展问题，避免同图跨切分。","sha256":"7b74cc3e2f3205469e80f80e2972062d26a1c35de2e0f355d3b31ebd4564d1ad","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-df3d2acfc44787cd8aff78bb","nextBlock":"b-8bc5187c6d6e053e54568278","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-8bc5187c6d6e053e54568278","type":"table","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-8bc5187c6d6e053e54568278","content":"| 保存条件 | 数量 | 左色 | 右形 | 全部 |\n| --- | --- | --- | --- | --- |\n| 图像＋问题 | 100% | 100% | 100% | 100% |\n| question-only | 50% | 33.33% | 50% | 44.44% |\n| 同模型，像素置零 | 50% | 33.33% | 50% | 44.44% |\n| 同模型，循环错配图像 | 83.33% | 50% | 0% | 44.44% |","sha256":"c029a7f961a8f3f4fc6629058322f9734e26c4f0f26ae8f34ba8cb501a091ec6","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"table":{"headers":["保存条件","数量","左色","右形","全部"],"rows":[["保存条件","数量","左色","右形","全部"],["图像＋问题","100%","100%","100%","100%"],["question-only","50%","33.33%","50%","44.44%"],["同模型，像素置零","50%","33.33%","50%","44.44%"],["同模型，循环错配图像","83.33%","50%","0%","44.44%"]]},"previousBlock":"b-25f9d762ba22c68c6e472610","nextBlock":"b-c50022dae8d35de55cf5d292","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-c50022dae8d35de55cf5d292","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-c50022dae8d35de55cf5d292","content":"![同图换问、同问换图](/assets/display/2f7b2168f5b21ee2095709fa.webp)","sha256":"08d0aa022e98bb75079211b073bb26e8245e159a9c4dcb17e3839b2d504f1d23","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-8bc5187c6d6e053e54568278","nextBlock":"b-afa27b1bd817e6033b7b07de","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-afa27b1bd817e6033b7b07de","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-afa27b1bd817e6033b7b07de","content":"循环错配不是独立重抽，排列常保留数量却改变形状；总分相同掩盖不同错误。正确图像下的成功和干预后的下降，支持这个有限任务使用了视觉信息。问题是三个固定模板ID，不主张开放语言泛化。[逐问题指标的完整解释](/course/blocks/vqa-pixel-evidence.html)。","sha256":"707e7bc48c07443cf12149644f705a6c7c56301b0b49bf4524b81dbab3c92e29","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-c50022dae8d35de55cf5d292","nextBlock":"b-115dd44545e326043a75a14a","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-115dd44545e326043a75a14a","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-115dd44545e326043a75a14a","content":"**备课模拟对话。** 学生：“无图也有44%，图像是不是不重要？”教师：“二选一、三选一、二选一的常量率平均就是4/9；同任务视觉模型100%且错配后下降，才说明新增条件有作用。”","sha256":"3db3903d93095f36602506f7c9a92536b7083ddc36b16b3535a3e4f55a1cdff6","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-afa27b1bd817e6033b7b07de","nextBlock":"b-3db8998813fc2191bcecb9ba","prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-3db8998813fc2191bcecb9ba","type":"p","scope":"block-1b53d32e9e684591be91385c30d46982","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-3db8998813fc2191bcecb9ba","content":"本轮没有重跑这些实验。Caption产生新句子，匹配返回候选分数，VQA还要根据问题选择信息。理解区别后，按需要进入CLS 10 的匹配或 CLS 11 的区域指代，不必把所有册重新读一遍。","sha256":"3df86520780b10397874c31809ad84a890727b5848b6a2551cd77e3f4ffd45bd","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-1b53d32e9e684591be91385c30d46982","title":"VQA：问题控制读什么，图像控制答什么","number":3},"sectionTitle":"VQA：问题控制读什么，图像控制答什么","currentQuestion":"VQA：问题控制读什么，图像控制答什么","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-115dd44545e326043a75a14a","nextBlock":null,"prerequisites":[{"id":"b-9a075058f8bd5c4238ce17f9","url":"#b-9a075058f8bd5c4238ce17f9","role":"section-title"},{"id":"b-68c356ce73e4c6c12ce5f828","url":"#b-68c356ce73e4c6c12ce5f828","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-1e9944a8f8b2071d6b668132","type":"h2","scope":"block-0bf24433979946a493db9110cc38c355","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-1e9944a8f8b2071d6b668132","content":"## 看清图像条件，再打开解码器","sha256":"998e4aa89e86386109534916c3365369ad33129026c1bad72b87fe9e4604612a","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-0bf24433979946a493db9110cc38c355","title":"看清图像条件，再打开解码器","number":4},"sectionTitle":"看清图像条件，再打开解码器","currentQuestion":"看清图像条件，再打开解码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":null,"nextBlock":"b-44a3a02b2dac7e60ca1f5b82","prerequisites":[{"id":"b-44a3a02b2dac7e60ca1f5b82","url":"#b-44a3a02b2dac7e60ca1f5b82","role":"problem-context"}],"relatedBlocks":[]},{"id":"b-44a3a02b2dac7e60ca1f5b82","type":"p","scope":"block-0bf24433979946a493db9110cc38c355","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-44a3a02b2dac7e60ca1f5b82","content":"本讲先检查图像是否真正影响逐词预测与生成结果：[LAB 13](https://deepnote.com/project/38755e4e-23f6-4f5f-8c68-8a2e4114b1c1/notebook/3da47f3ee3554ee0a36e0bb6a2fdc61f?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=3da47f3ee3554ee0a36e0bb6a2fdc61f&utm_term=get_notebook)保留训练、实际接续输出、遮图与错图消融。较高模型概率和更符合图片是两个评价问题，不能由搜索方式直接互相保证。","sha256":"e6f220aac2e44a39989ddf391bb915ed36cca0070722c1f760af3134ffc4f3d1","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-0bf24433979946a493db9110cc38c355","title":"看清图像条件，再打开解码器","number":4},"sectionTitle":"看清图像条件，再打开解码器","currentQuestion":"看清图像条件，再打开解码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-1e9944a8f8b2071d6b668132","nextBlock":"b-2e1644b7a273314c1f3d037f","prerequisites":[{"id":"b-1e9944a8f8b2071d6b668132","url":"#b-1e9944a8f8b2071d6b668132","role":"section-title"}],"relatedBlocks":[]},{"id":"b-2e1644b7a273314c1f3d037f","type":"p","scope":"block-0bf24433979946a493db9110cc38c355","url":"https://codingai-lec03.pages.dev/course/notebooks/c9f328a00ef44ecc90804e47276a7811.html#b-2e1644b7a273314c1f3d037f","content":"循环状态方程、整句概率分解、规范化概率树及 greedy/beam 比较，已整理到[第五讲：序列模型与图像条件解码](https://deepnote.com/project/841ec8e7-e352-4840-b628-d3742b03d6d0/notebook/b1c8e2fd764441e8820b808fc40ec3cb?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=b1c8e2fd764441e8820b808fc40ec3cb&utm_term=get_notebook)。需要解释 LAB 13 的内部实现时再沿此入口展开。","sha256":"bebca56938c02b0a6d803f51787a60adaa7a3695479f8eee343d1949bc01e733","chapter":{"id":"III","title":"视觉与语言"},"unit":{"id":"language-task-interfaces","title":"语言加入后，任务究竟改变了什么？"},"page":{"id":"reading-block-0bf24433979946a493db9110cc38c355","title":"看清图像条件，再打开解码器","number":4},"sectionTitle":"看清图像条件，再打开解码器","currentQuestion":"看清图像条件，再打开解码器","sourceSlides":[],"knowledgeBlock":null,"formulas":[],"previousBlock":"b-44a3a02b2dac7e60ca1f5b82","nextBlock":null,"prerequisites":[{"id":"b-1e9944a8f8b2071d6b668132","url":"#b-1e9944a8f8b2071d6b668132","role":"section-title"},{"id":"b-44a3a02b2dac7e60ca1f5b82","url":"#b-44a3a02b2dac7e60ca1f5b82","role":"problem-context"}],"relatedBlocks":[]}]}