Recently, reinforcement learning and deep neural networks have opened up new ways to think about representing a 3D scene. Two key elements that relate to the ideas discussed here are that they do not represent the scene using 3D coordinates and the representation is task-dependent. A good example is Zhu et al . They use two networks as 'siamese' layers that process to current visual stimulus (i.e. s in our description) and the current task (related to t in our description).
- Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017, May). Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Robotics and Automation (ICRA), 2017 IEEE International Conference on (pp. 3357-3364). IEEE.