All-Day Multi-Scenes Lifelong Vision-And-Language Navigation With Tucker-Adaption

In order to learn the $t$-th navigation scenario task (with the $s$-th scene and the $e$-th environment) $T_{t} = \{S_{s},E_{e}\}$, we propose a new type of finetuning method named Tucker-Adaption (TuKA) to finetune the pretrained StreamVLN agent $\mathcal{F}_{\theta_{0}}$, in a higher-dimensional space, on the task-specific data $S_{t}=\{\mathcal{O}_{t},\mathcal{I}_{t}\}$, and then obtain an updated model $\mathcal{F}_{\theta'_{t}}$, where $\theta'_{t}=\theta_{0}+\Delta\theta_{t}$, $\Delta\theta_{t} = \{\Delta W^{l}_{t}\}^{L}_{l=1}$, and $\Delta W^{l}_{t}\in\mathbb{R}^{a_{l} \times b_{l}}$ is the updated weight in $l$-th layer of a total of $L$ transformer layers for task $T_{t}$. In our TuKA, we follow the Tensor Tucker Decomposition Approach (TAKE) to decouple the higher-order tensor knowledge for better multi-hierarchical navigation knowledge learning. Specifically, a four-order tensor $\mathcal{X} \in \mathbb{R}^{a\times b\times M \times N}$ can be decomposed into: $$\mathcal{X} = \mathcal{G} \times_{1} U^{1} \times_{2} U^{2} \times_{3} U^{3} \times_{4} U^{4}, \tag{1}$$ where $\times_{n}, n=1,2,3,4$ denotes the $n$-th modal product of the tensor and matrix. $\color{orange}{\mathcal{G}}\in \mathbb{R}^{r_{1}\times r_{2}\times r_{3}\times r_{4}}$ is core tensor, which contains interaction information between all patterns, and is used to learn the $\color{orange}{\text{navigation-shared knowledge}}$. The factor matrix $\color{orange}{U^{1}} \in \mathbb{R}^{a \times r_{1}}$ represents the transformation pattern of the feature from $r_{1}$ dimension to $a$, which can be regarded as a decoder; $\color{orange}{U^{2}} \in \mathbb{R}^{b \times r_{2}}$ represents the transformation pattern of the feature from $b$ dimension to $r_{2}$, which can be regarded as a encoder. The factor matrix $\color{purple}{U^{3}} \in \mathbb{R}^{M \times r_{3}}$ represents the $M$ group of scene experts, with each scene expert $\color{purple}{U^{3}[i,:]}$ is used to learn the $i$-th $\color{purple}{\text{specific scene knowledge}}$; $\color{cyan}{U^{4}} \in \mathbb{R}^{N \times r_{4}}$ represents $N$ group of environment experts, with each expert $\color{cyan}{U^{4}[j,:]}$ is used to learn the $j$-th $\color{cyan}{\text{specific environment knowledge}}$. Thus, for the $t$-th scenario with $s$-th scene and $e$-th environment adaptation, we extract the task-specific matrices $U^{3}[s,:]$ and $U^{4}[e,:]$ from the high-order tensor $\mathcal{X}$ to constitute weight $W_{t}$: $$\Delta W_{t} = U^{1} \cdot(\mathcal{G} \times_{3} U^{3}[s,:] \times_{4} U^{4}[e,:]) \cdot (U^{2})^{T}. \tag{2}$$