The care-providing robotic system FRIEND (Functional Robot arm with user-frIENdly interface for Disabled people) is a semi-autonomous robot designed to support disabled and elderly people in their daily life activities, like preparing and serving a meal, or reintegration in professional life. FRIEND make it possible for such people, e.g. patients which are paraplegic, have muscle diseases or serious paralysis, e.g. due to strokes, to perform special tasks in daily life self-determined and without help from other people like therapists or nursing staff.
The robot FRIEND is the third generation of such robots developed at the Institute of Automation (IAT) of University of Bremen within different research projects.[1][2][3][4] Within the last project AMaRob, an interdisciplinary consortium, consisting of technicians, designers as well as therapists and further representatives of various interest groups, influences the development of FRIEND. Besides covering the various technical aspects, also design aspects were included as well as requirements from daily practice given by therapists, in order to develop a care-providing robot that is suitable for daily life activities. The AMaRob project was founded by the German Federal Ministry of Education and Research ("BMBF - Bundesministerium für Bildung und Forschung") within the "Leitinnovation Servicerobotik".
Contents |
FRIEND is built from reliable industrial components. It is based on the wheelchair platform Nemo Vertical, which is an electrical wheelchair from Meyra. This basic platform has been equipped with various additional components, which are described in the following.
Within the AMaRob (AMaRob web page ) project three scenarios were developed that support disabled and elderly people in their Activities of Daily Life (ADL) as well as in professional life.
This scenario enables the user to prepare and eat a meal. A special meal-tray has been designed which can be gripped by the manipulator. First the meal-tray is fetched from a refrigerator which is equipped with an automatic door opener. Then the manipulator puts the meal-tray in a remote controlled microwave oven to cook the meal. After cooking the meal-tray is fetched from the microwave oven and placed on the tray of the wheelchair. After that FRIEND supports the user to eat the meal. With a special designed spoon the manipulator can take a bit meal and feed the user. After the eating procedure the meal-tray is put away. The handles are designed in such a way that they can be robustly recognized by the vision system and easily grasped by manipulator's gripper.
The first professional scenario is situated at a library service desk. Professional scenarios are very important for disabled people from the viewpoint of re-integration into daily life activities. With FRIEND the user can manage tasks like lending and return of books.
The second professional scenario takes place in a rehabilitation workshop. The realized scenario is representative for many quality control tasks in industry and consists of checking the functionality of keypads for public telephone boxes. A keypad has to be taken from a keypad magazine by the gripper and put into a test adapter to check the correct working of pad's electronics. After that each keypad has to be inspected visually by the user to detect cracks or similar damages. Based on the results the keypad have to be sorted.
The realization of semi-autonomy is based on the central idea to include human tasks into task execution. A good explanation of this principle is given in the following quotation:
With respect to a semi-autonomous service robotic system this means that the user's cognitive capabilities are taken into account, whenever a robust and reliable technical solution is unavailable. It is obvious that the acceptance of such a system will be low in general. But for people that are dependent on a personnel assistant, like the disabled or elderly, this approach offers the opportunity to decrease this dependency and therefore to increase their quality of life. A service robot has to be able to pursue a certain mission goal as commanded from its user but also needs to react flexibly to dynamic changes within the workspace. To meet these requirements, hybrid multi-layer control architectures have been successfully applied.[5][6] These architectures usually consist of three layers:
The software framework MASSiVE (Multi-layer Architecture for Semi-autonomous Service robots with Verified task Execution) [7][8] is a special kind of hybrid multi-layer control architecture which is tailored to the requirements of semi-autonomous and distributed systems, like the care-providing robot FRIEND, acting in environments with distributed smart components. These intelligent wheelchair mounted manipulator systems allow to benefit from the inclusion of the user's cognitive capabilities into task execution and consequently lower the system complexity compared to a fully autonomous system. The semi-autonomous control requires a sophisticated integration of a human-machine-interface (HMI) which is able to couple input devices according to the user's impairment,[9] for example a haptic suit, eye-mouse, speech-recognition, chin joystick or a brain-computer interface (BCI).[10][11] The resulting MASSiVE control architecture with special emphasis on the HMI component is depicted in Fig. 2. Here, the deliberator has been moved to the sequencer component, and the HMI has direct access to control the actuators in the reactive layer during user interactions (e.g., to move the camera, until the desired object to be manipulated is in field of view).
Besides the focus on semi-autonomous system control, the MASSiVE framework includes a second main paradigm, namely the pre-structuring of task knowledge. This task planner input is specified offline in a scenario and model driven approach with the help of so-called process-structures on two levels of abstraction, the abstract level and the elementary level.[12] After specification and before being used for task execution, the task knowledge is verified offline, to guarantee a robust runtime behavior. This development process model provides a structured guidance and enforce consistency throughout the whole process, so that uniform implementations and maintainability are achieved. Furthermore it guides through development and test of system core functionality (skills). The whole paradigm is depicted in Fig. 3.
The tasks is selected and started by the user via the HMI, depicted in Fig. 4, on a high level of abstraction, e.g. "cook meal". After initial monitoring to define the current state of the system and the environment, the tasks execution is performed and a list of elementary operations are created which can be executed autonomously by the system. These elementary operations consists of, e.g. image processing algorithm to recognize objects in the environment or manipulative algorithms to calculate a special trajectory to grasp an object.
Besides these layers, a world model is included in the control architecture that contains the current system's perspective on the world according to the task to be executed. Due to the hybrid architecture a separation of world-model data into two categories is mandatory: The deliberator operates with symbolic object representations (e.g. "C" for the representation of a cup), while the reactive layer deals with the sensor percepts taken from these objects, so-called sub-symbolic information. Examples for sub-symbolic information are the color, size, shape, location or weight of an object.
The main problem with service robotic systems such as the care-providing robot FRIEND is that they have to operate in dynamic surroundings where the state of the environment is unpredictable and changes stochastically, hence two main problems have been encountered when developing image processing systems for service robotics: unstructured environment and variable illumination conditions. They have to cope with a large amount of visual information and for the implementation of the vision system a high degree of complexity is necessary. A second major problem in robot vision is the wide spectrum of illumination conditions that appear during the on-line operation of the machine vision system, since colors are one important attribute in object recognition. The human visual system has the ability to compute color descriptors that stay constant even in variable illumination conditions, which is not the case for machine vision systems. A key requirement in this field is the reliable recognition of objects in the robot's camera image, extraction of object features from the images and, based on the extracted features, subsequent correct object localization in a complex 3D environment so that these information can be used for reliable object grasping and manipulation.
In order to cope with the above described problems in the care-providing robot FRIEND the special vision system ROVIS (RObust machine VIsion for Service robotics) [13][14] was developed. The structure of ROVIS is depicted in Fig. 5. There are two main ROVIS components: hardware and object recognition and reconstruction chain. The connection between ROVIS and the overall robot control system is represented by the World Model where ROVIS stores the processed visual information. At the initialization phase of ROVIS the extrinsic camera parameters needed for 3D object reconstruction and the transformation between stereo camera and manipulator which is necessary for vision based object manipulation are calculated by the Camera Calibration module. The object recognition and reconstruction chain consists of robust algorithms for object recognition and 3D reconstruction for the purpose of reliable manipulation motion planning and grasping in unstructured environments and variable illumination conditions. Therefore an accuracy of 5mm for the estimated 3D pose is necessary which enforces very good and precise algorithms. In ROVIS, robustness must be understood as the capability to the system to adapt to varying operational conditions and is realized through the inclusion feedback mechanisms at the image processing level and also between different hardware and software components of the vision system.[15] A core part of the system is the automatic, closed-loop calculation of an image Region of Interest (ROI) on which vision methods are applied. By using a ROI the performance of object recognition and 3D reconstruction can be improved since the scene complexity is reduced.
Within ROVIS there are several methods used for object recognition, e.g. robust region based color segmentation [16] and robust edge detection. The first one is for objects with uniform color and without texture (e.g. bottle, glass, handles) and the second one for objects with textures (e.g. books). In order to recognize objects in FRIEND's environment robustly special feature are extracted which are used to identify objects and to determine their pose.[17] For big objects like refrigerator or microwave an improved SIFT (Scale Invariant Feature Transform) algorithm is used, which was developed at the Institute of Automation at the University of Bremen [18]
Primary task of the motion planning is to find a collision free trajectory. Nevertheless, in service robotics, other aspects must be additionally taken into account like safety and smoothness of the trajectories. In order to suffice all these requirements for FRIEND a sensor-based motion planning for the manipulator was developed. The procedure is based on Cartesian space information,[19] and therefore computationally efficient as well as with a high precision. The object grasping frames are calculated on-line which increases the flexibility of the system during execution. All implemented algorithms are suitable for real-time applications in service robotics but also for industrial usage.
The motion planning is performed based on a Mapped Virtual Reality (MVR), displayed in Fig. 6. This 3D-model of the environment is built from the information within the world model which was perceived by the sensors. Before any motion of the robot arm is performed the trajectory from the current to the goal configuration is calculated within this 3D-model. During motion on-line collision checking is done, since obstacles can move during motion. Obstacles are in this case all objects of the environment, which are not included in the current motion, also in some tasks the user in the wheelchair. It is important that there is at any time no danger for the user.
At the wrist of the manipulator a force-torque-sensor is installed. The information from this sensor is used to detect collisions or for fine-tuning during manipulative operations, e.g. when the robot arm should put a gripped object in a small opening. This ensures robustness during execution.
In general the trajectories calculated from the motion planning algorithms are robot like, i.e. clipped and jerky. In order to enhance this and make the movement of the manipulator more pleasant to the user the trajectories are smoothed and the quality is enhanced.[20] Therefore the used robot arm is helpful. Since six degrees of freedom are sufficient to define a 3D pose in the environment (three degrees for position and three degrees for orientation) and FRIEND's manipulator has seven degrees of freedom, one degree of freedom can be chosen optional, i.e. the manipulator can turn its elbow joint by 360 degrees without changing gripper's pose. This can be used to find the best trajectory from a start to a goal configuration among an infeasible set of possible configurations. This seventh degree of freedom is also used to solve and avoid dead-locks during the motion process and to keep a minimal distance to obstacles.