Sensory-motor system identification of active perception in ecologically valid environments

The brain is a dynamical system mapping sensory inputs to motor actions. This relationship has been widely characterised by reductionist controlled experiments. Here we present work moving out of the lab ``into the wild'' to capture, rather than constrain, sensory inputs and motor outputs, by recording 90\% of sensory inputs using head mounted eye-tracking, scene camera and microphone as well as recording 95\% of skeletal motor outputs by motion tracking 51 degrees of freedom in the body and a total of 40 degrees of freedom from the hands. We can thus begin to systematically characterise the perception-action loop through system identification. This enables use to evaluate classical relationships in ecologically valid settings and behaviours including 3 daily scenarios: breakfast in the kitchen, evening chores and activities and in-door ambulation . This level of data richness (97 DOF, 60Hz), coupled with the extensive recordings of natural perceptual and behavioural data (total $>$ 30 hrs, 10 subjects) enables us to answer general questions of how lab tasks and protocols will produce systematically different results from those found in daily life.