Affective Computing Projects
eiman.kanjo@ntu.ac.uk
+44 (0)115 848 4820
The Construction Upskill App has been developed for the construction sector. It provides easy and instant access to training and CPD for construction workers. The current training available consists of four short modules co-designed with the sector to provide the construction sector with insights into collaboration and innovation and how each can be effective in driving productivity in the workplace.
MEMO Project inscription: Make your mark for biodiversity allows you to draw your own virtual inscription. In doing so, you will help to explore the profound threats posed to modern biodiversity by the actions of humanity, identify emerging solutions and innovations, and provide hope for a future that is safe, joyful and fulfilling.
We're on the crest of a breaking wave of a mass extinction event, unlike anything that has been seen for 65 million years. Eden Portland will be an underground cathedral where the ancient art of stone carving and cutting edge technology will combine to breathe life into stories of biodiversity, extinction and evolution.
Above ground, the natural beauty of the Jurassic Coast stretches as far as the eye can see but, out of the sun, the artificially-lit tunnels of Bowers Mine resemble a deserted underground city, criss-crossed with roads and flanked with towering slabs of limestone. The tunnels are cut from the same Portland stone whose embedded fossils inspired the first theories of extinction in the 17th century. The stone, formed 145 million years ago, has been used by stonemasons since at least Roman times and can be found on thousands of buildings across the UK and around the world, from the Tower of London and St Paul's Cathedral in London, to the United Nations Headquarters in New York.
Tag in the Park & Tag 4 Active Lives
The ubiquity of mobile devices make them ideal platforms for exercise games (exergames) to promote physical activities. Advances in Internet of Things (IoT) technologies including Bluetooth Low Energy (BLE) beacons can be utilised for proximity detection to promote physical activities and the use of Artificial Intelligence (AI) in the form of object recognition can accelerate engagement with location-based pervasive games.
We are currently implementing a casual exergame, Tag With Me, in the form of a treasure hunt that provides the approximate location of nearby points of interest in real-time within the vicinity of Bluetooth beacons. The system exploits the signal strength of the BLE beacons to measure proximity which makes it suitable for outdoor and indoor functioning where GPS signals are not accessible. Once the player walks towards a point of interest one of the interactive challenges is activated such as the AI and camera challenges where the player must scan the nearby object with their smartphone camera, the tag challenge where the player must search for a tag and tap it or an education quiz.
Multimodel Sensor Fusion Approach to study the impact of Environment Pollutants on Mental Wellbeing
The short and long term exposure to environmental urban factors (such as air pollution, gases, particulates and noise) can significantly impact an individual’s wellbeing and mental health. The World Health Organisation (WHO) found that 91% of people are living in places where the air quality guidelines are not met and the use of non-clean fuels and household emissions in the atmosphere are causing over 4.2 million deaths each year. In addition, those living in some locations in the UK have a higher risk of developing serious health conditions such as higher heart rate, asthma and cardio-cerebrovascular disease where a lifetime of exposure to high-levels of pollution can result in reduced life expectancy.
Repeated and continuous human exposure to the environment and highconcentrated air pollutants have been found to increase the risk of developing serious conditions such as respiratory and cardiovascular diseases or even death. Research recently has began focusing towards how the environment can impact physical health but it also is necessary to explore how the environment can impact mental wellbeing. Pollution within the urban environment is a continual problem contributing to rising health and mental wellbeing challenges. The ability to monitor air pollutants, physiology and mental wellbeing will enable the relationship between repeated environment exposures and mental wellbeing to be established.
The the term ’DigitalExposome’ as a framework to quantify an individual’s exposure to the environment by utilising a range of technological, mobile-sensing and digital devices. The concept aims to measure multiple environmental factors using mobile technologies and then quantify them in real-life settings. Combining multiple data collection methods helps to support DigitalExposome and gain a better understanding into how exposures to the environment can impact mental wellbeing.
Voronoi visualisations have given an indication of how changes within the environment can have an impact on mental wellbeing. Typically, it was found that where air pollution such as PM1, 2.5, 10 and Noise was increasing, participants labelled their wellbeing as very negative. This demonstrates consistent results with previous studies in this area. This form of spatial analysis, greatly helps in understanding the degree to which a place is similar to other nearby places.
Statistical analysis including PCA, Multi variant Linear Regression, Voronoi and data spatial visualisations were implemented to explore the variation in data and the factor importance. We found that physiological (on-body) sensor data is directly correlated to pollution (PM in particular) within the environment. In addition, DBNs have helped successfully classify five states of wellbeing with up to 80.8% accuracy using the fused physiological and pollution data.
Tangible Fidgeting Interfaces for Mental Wellbeing Recognition using Deep Learning applied to Physiological Sensor Data
The momentary assessment of an individual's affective state is critical to the monitoring of mental wellbeing and the ability to instantly apply interventions. This research introduced the concept of tangible fidgeting interfaces for affective recognition from design and development through to evaluation. Tangible interfaces expand upon the affordance of familiar physical objects as the ability to touch and fidget may help to tap into individuals' psychological need to feel occupied and engaged. Embedding digital technologies within interfaces capitalises on motor and perceptual capabilities and allows for the direct manipulation of data, offering people the potential for new modes of interaction when experiencing mental wellbeing challenges.
Tangible interfaces present an ideal opportunity to digitally enable physical fidgeting interactions along with physiological sensor monitoring to unobtrusively and comfortable measure non-visable changes in affective state. This opportunity initiated the investigation of factors that would bring about the designing of more effective intelligent solutions using participatory design techniques to engage people in designing solutions relevant to themselves.
Adopting an artificial intelligence approach using physiological signals created the possibility to quantify affect with high levels of accuracy. However, labelling is an indispensable stage of data pre-processing that is required before classification and can be extremely challenging with multi-model sensor data. LabelSens introduced new techniques for labelling at the point of collection using five custom built tangible labelling interfaces.
When classifying labelled physiological sensor data, individual differences between people limit the generalisability of models. To address this challenge, a transfer learning approach has been developed that personalises affective models using few labelled samples. This approach to personalise models and improve cross-domain performance is completed on-device, automating the traditionally manual process, saving time and labour. Furthermore, monitoring trajectories over long periods of time inherits some critical limitations in relation to the size of the training dataset. This shortcoming may hinder the development of reliable and accurate machine learning models. A second framework has been developed to overcome the limitation of small training datasets using an image-encoding transfer learning approach.
This research offered the first attempt at the development of tangible interfaces using artificial intelligence towards building a real-world continuous affect recognition system in addition to offering real-time feedback to perform as interventions. This exploration of affective interfaces has many potential applications to help improve quality of life for the wider population.
The COVID-19 Pandemic brought about closure to work spaces, public venues and academic environments. These works investigated a solution that enables a safe exit strategy out of restrictions. Through the use of an IoT based Bluetooth Low Energy (BLE) device, individuals were reminded of the social distancing measures, facilitated through visual and haptic feedback. The device also enabled individuals to monitor close contacts, recorded on a database. The technology developed provides the additional benefit of no further equipment for contract tracing being provided. It also provides real-time alerts for larger gatherings, something that alternative systems of the time lacked in.
Children can find it challenging to communicate their emotions especially when experiencing mental health challenges. Technological solutions may help children communicate digitally and receive support from one another as advances in networking and sensors enable the real-time transmission of physical interactions. In this work, we pursue the design of multiple tangible user interfaces designed for children containing multiple sensors and feedback actuators. Bluetooth is used to provide communication between Tangible Toys (TangToys) enabling peer to peer support groups to be developed and allowing feedback to be issued whenever other children are nearby. TangToys can provide a non-intrusive means for children to communicate their wellbeing through play.
The world is changing when it comes down to building infrastructure. There is a need more now, than ever before for people to monitor their building data as we look into ways in which organisations can be greener, with a potential change in legislation being on the horizon. Not only this, but although general life is beginning to get back underway after the COVID-19 episode, globally, we are still feeling the effects that came along with the pandemic. This made huge changes to the way that people work and how organisations operate – with a lot taking a work from home approach or a half and half, resulting in a reduction in the space needed for organisations to operate efficiently.
The implementation of the dashboard and sensor technology would also be a route to allow an organisation to analyse data and make changes off the back of it, for example cutting costs.
As the world is moving smarter and technology continues to play a huge part, the most obvious thing is to have all your data in one place rather than over multiple platforms.
With the above cases in mind, Scenariio ltd alongside NTU smart sensing team, have been looking at ways to best pull, store and visualise data in order to have clarity and make decisions more easily. For this to be the case, the visualisation need to be in a format in which makes sense to people, it needs to be informative, accurate and meaningful.
The implementation of the dashboard will allow vast kinds of data to be visualised - Temperature, Occupancy, Energy, Illuminance, Humidity, CO2 levels, CCTV integration and the list goes on. The sensors implemented can be tailored by the needs and wants of an organisation. The dashboard will be bespoke for every business it is integrated into, as everyone’s needs are different. But using the software and technology we have created, this is an easy task.
There are many uses for the dashboard, whether that being:
Many of the sensors that are used come with the option of being light-based integrated. This then also gives the option of a user to customise their own choice of lighting – levels and colours. This also allows the data to command the lighting to do something. If the sensors sensed no motion in a space for a given period, a push command could be sent to turn the lighting off and visa-versa.
We are also working on alert commands which will be sent to mobile devices if pre-specified criteria is met. Visualisations are great, for the periods you are looking at them. But it’s when you aren’t that things could be missed. An example being motion sensed when no one is in the building, or a temperature going over a certain threshold that may suggest a fire. We have these areas covered.
In order to showcase this innovation, there are some showcase sites set up to give a real feel as to what Scenariio have to offer. The main showcase is located at Scenariio’s shared home with Buster Market – The Chocolate Factory, in Derby. Further to this there is a small demonstrator located in the SWIFt lab, Engineering building at NTU’s Clifton campus.