Abstract – This paper provides a vision based HCI method for a user to command a cleaning robot to move to a specific location in home environment. Six hand poses are detected from a video sequence taken from a camera on the cleaning robot. AdaBoost based hand-pose detectors are trained with a reduced Haar-like feature set to make the detectors robust to the influence of the complex background. The first three stages of the cascade in the six detectors are used as pose estimation to reduce the computational complexity. The cleaning area is determined from the detected pose. The performances of the proposed detectors are validated with a set of test images with cluttered background. The cleaning area control is simulated with real-world video sequences. The proposed method can effectively control a cleaning robot without the need for a user to wear or employ any input devices.