Continuous Multimodal Multi-view Dataset of Human Fall (CMDFALL)

About the Project

The project is a collaboration between Computer Vision Department (COMVIS) of MICA International Research Institute (MICA) and Posts & Telecommunications Institute of Technology (PTIT). The objective of this project is to build a Continuous Multimodal Multi-view Dataset of human fall actions and to develop algorithms for human fall recognition. The dataset consists of both normal daily activities and simulated falls.

Description

Safety is a key concern for families with elderly people. Fall is the major risk for elderly people living at home alone, which might cause serious health consequence if medical treatment is not afforded in-time. Many techniques were proposed such to detect human fall to immediately alert other family members/medical staffs for in-time first aid. Two most potential types of sensing devices are cameras and accelerometers. Many computer vision/ accelerometer signal processing algorithms have been proposed for human fall recognition but there is a lack of a common, large-enough dataset of human fall actions for research community to evaluate their algorithms. This project aims at building a Continuous Multimodal Multi-view Dataset of human fall actions acquired from Microsoft Kinect cameras and wireless accelerometers simultaneously. The CMDFALL dataset contains 20 actions performed by 50 persons (30 males and 20 females in the range 21-40 years of age) with age and gender distribution as follows:

Figure 1: Age Distribution of 50 Subjects

Figure 2: Gender Distribution of 50 Subjects

Each person performed all 20 actions in 7.5 minutes, yielding about 375 minutes recording time in total. Falls were done in all different directions (front/back/left/right). A mattress was laid on the floor to protect the subjects during the simulated falls.

Figure 3: Data from seven Kinect cameras and two accelerometers

Hardware

In this section we describe the layout of the multimodal data collection system used for the collection of CMDFALL. Each action was simultaneously captured by two different sensing systems: seven Microsoft Kinect cameras and two wireless accelerometers.

The action list is divided into two scenes. Scene 1 is a room with no bed as shown in Figure 4. Scene 2 is a room with a bed as shown in Figure 5.

Figure 4: Scene 1 Room with no bed

Figure 5: Scene 2 Room with a bed

Kinect System

There are seven Kinect cameras (numbered from Kinect1 to Kinect7). The locations of devices are described as follows:

Six Kinect cameras (from Kinect1 to Kinect6) are installed at height 1.8m.
One Kinect camera (Kinect7) is installed on the ceiling at height 3.0m.

Each Kinect camera captured a color image with a resolution of 640x480 pixels and a 16-bit depth image, both with an acquisition rate of 20 Hz. The laptop connected to Kinect6 was set as a controller for time synchronization for all seven Kinect cameras.

Accelerometers

Two accelerometers are used for each subject during recording, one on the left wrist and one on the left hip of the subject to capture the physical activities. The sensor used in this project is 3-dimensional wireless accelerometer (WAX3) developed by researchers at Open Lab, Newcastle University. Originally, WAX3 was developed for embedding inside kitchen utensils and appliances. WAX3 is a wireless, small +/-16g 3-axis accelerometer with the dimension of 23 x 32.5 x 7.6 mm, and weighted just 7 grams. WAX is MEMS accelerometer that operates with the IEEE 802.15.4 low power radio protocol. It has the sampling rate of up to 2ks.sec-1 and supports Open Sound Control (OSC) message, binary, and American Standard Code for Information Interchange (ASCII) format. WAX consists of two components: WAX receiver (connecting to the computer via a USB port) and WAX transmitter (wireless communication to the receiver with the range of up to 25 m). WAX is also equipped with a re-chargeable Li-Polymer battery providing up to 8 hours of continuously signal transmission mode and up to 56 days in hibernate mode.

Figure 6: WAX3 Accelerometer

Action Checklist

For each subject, we collected the movements as shown in the following table.

Scene 1: Room with no bed

ID	Action	Recording Length (sec)	Annotation Start Point	Annotation End Point
1	Forward fall	30	Subject stands straight before forward fall	Subject lies motionless on the floor
2	Back fall	30	Subject stands straight before back fall	Subject lies motionless on the floor
3	Left fall	30	Subject stands straight before left fall	Subject lies motionless on the floor
4	Right fall	30	Subject stands straight before right fall	Subject lies motionless on the floor
7	Left fall while sitting on a chair	20	Subject sits on the chair before left fall	Subject lies motionless on the floor
8	Right fall while sitting on a chair	20	Subject sits on the chair before right fall	Subject lies motionless on the floor
9	Walk in all direction	20	Subject starts walking	Subject stops walking
10	Run slowly	20	Subject starts running	Subject stops running
11	Stagger	20	Subject starts staggering	Subject stops staggering
12	Crawl	10	Subject starts crawling	Subject stops crawling
13	Move the chair	20	Subject starts moving the chair	Subject stops moving the chair
14	Move hands and knees	20	Subject starts moving hands and knees	Subject stops moving hands and knees
15	Crouch down to pick up things by left hand	20	Subject stands straight before crouching down to pick up things by left hand	Subject's head is at the highest position after picking up things
16	Crouch down to pick up things by right hand	20	Subject stands straight before crouching down to pick up things by right hand	Subject's head is at the highest position after picking up things
17	Jump in place	20	Subject starts jumping	Subject stops jumping
18	Sit down on a chair then stand up	20	Subject stands straight before sitting on the chair	Subject stands straight after standing up

Scene 2: Room with a bed

ID	Action	Recording Length (sec)	Annotation Start Point	Annotation End Point
5	Left fall while lying on a bed	30	Subject lies on the bed before left fall	Subject lies motionless on the floor
6	Right fall while lying on a bed	30	Subject lies on the bed before right fall	Subject lies motionless on the floor
19	Sit down on a bed then stand up	20	Subject stands straight before sitting down on the bed	Subject stands straight after standing up
20	Lie on a bed then sit up	20	Subject lies on the bed before sitting up	After sitting up

Action Classification

Twenty actions are recorded among which eight actions are falls of different styles and twelve actions are daily and fall-like activities. For the task of fall detection, we categorize the actions into two groups: fall and non-fall. In the group of falls, we have three types of fall, each has different styles and directions. In the group of non-fall activities, we categorize in term of movement of the whole body or only component and direction of movement.

Group	Activity Group	ID
Fall	While walking While lying on the bed While sitting on the chair	1,2,3,4 5,6 7,8
Non-fall	Horizontal movement of the whole body Hand and leg movement Horizontal movement of the whole body	9,10,11,12,13 14,15,16 17,18,19,20

Collected Data and File Formats

In the table below, we provide information on the type and the size of data collected per subject by different sensors in the acquisition system.

System	Data Format	Collected Data	File Name	Approximate Size per Subject
Microsoft Kinect	AVI BIN ASCII	RGB color data 16-bit depth map UNIX time stamps	color.avi depth.bin timestamp.txt	7 GB
Accelerometers	ASCII	Acceleration in XYZ axes UNIX time stamps	wrist-worn accelerometer.txt hip-worn accelerometer.txt	2 MB

Demo Video

Download Instructions

Our CMDFALL dataset is available and free for research purpose. If you want to use our dataset, please contact thanh-hai.tran@mica.edu.vn.