Lecture 03 学习讲义
正文
工具
按问题查找
Lecture 032. 先按图像切分,避免同一图的不同问题泄漏 · 实现代码

2. 先按图像切分,避免同一图的不同问题泄漏 · 实现代码

独立阅读

按问题继续

RGB+问题输入、question-only对照、同图换问与同问换图

展开完整实现代码
def draw_scene(count,left_color,right_shape,rng):
    img=Image.new('RGB',(40,32),(244,242,236));d=ImageDraw.Draw(img)
    slots=[10] if count==1 else [10,30]
    for j,cx in enumerate(slots):
        cy=16+int(rng.integers(-3,4));r=int(rng.integers(4,7));cx+=int(rng.integers(-1,2))
        color=left_color if j==0 else str(rng.choice(list(COLORS)))
        shape=right_shape if j==len(slots)-1 else str(rng.choice(['square','circle']))
        box=(cx-r,cy-r,cx+r,cy+r)
        if shape=='square':d.rectangle(box,fill=COLORS[color])
        else:d.ellipse(box,fill=COLORS[color])
    arr=np.array(img,dtype=np.float32)/255
    return np.clip(arr+rng.normal(0,.014,arr.shape),0,1).astype(np.float32)

def make_vqa(n,seed,biased):
    rng=np.random.default_rng(seed);images=[];answers=[];meta=[]
    # Training has a 75% red prior. Test covers all 12 condition combinations equally.
    combos=[(c,col,sh) for c in [1,2] for col in COLORS for sh in ['square','circle']]
    for i in range(n):
        if biased:
            count=int(rng.choice([1,2]));color=str(rng.choice(list(COLORS),p=[.75,.125,.125]));shape=str(rng.choice(['square','circle']))
        else:count,color,shape=combos[i%len(combos)]
        images.append(draw_scene(count,color,shape,rng))
        answers.append([A['one' if count==1 else 'two'],A[color],A[shape]])
        meta.append({'count':count,'left_color':color,'right_shape':shape})
    return torch.tensor(np.stack(images)).permute(0,3,1,2),torch.tensor(answers),meta
Xtr,Ytr,train_meta=make_vqa(720,14031,True)
Xte,Yte,test_meta=make_vqa(180,14032,False)
train_hashes={hashlib.sha256(x.numpy().tobytes()).hexdigest() for x in Xtr}
assert not train_hashes.intersection(hashlib.sha256(x.numpy().tobytes()).hexdigest() for x in Xte)
# Metadata remains only in this data/label construction and plot captions.
# model.forward receives tensors (pixels, question_id), never these dictionaries.
np.savez_compressed(OUT/'lab14-data.npz',train_images=Xtr.numpy(),train_answers=Ytr.numpy(),test_images=Xte.numpy(),test_answers=Yte.numpy())
print('train/test image counts:',len(Xtr),len(Xte),'pairs:',Ytr.numel(),Yte.numel())
print('train left-color prior:',{c:sum(m['left_color']==c for m in train_meta)/len(train_meta) for c in COLORS})
print('test left-color prior:',{c:sum(m['left_color']==c for m in test_meta)/len(test_meta) for c in COLORS})