If you're wearing XanderGlasses, you do not need to be looking at the speaker for the glasses to caption.
Microphones are embedded within the arms of the glasses and they listen for speech, then project captions in real time. So you do not need to look directly at the person speaking for the microphone to caption. If the person is speaking loudly and clearly enough for the microphone to hear, their speech will be captioned for you to see.