publications

2020

1. Action-conditioned Benchmarking of Robotic Video Prediction Models: a Comparative Study Serra Nunes, M., Dehban, A., Moreno, P., and Santos-Victor, J. In IEEE International Conference on Robotics and Automation (ICRA) 2020 [Abs] [Git] [bib]

   A defining characteristic of intelligent systems is the ability to make action decisions based on the anticipated outcomes. Video prediction systems have been demonstrated as a solution for predicting how the future will unfold visually, and thus, many models have been proposed that are capable of predicting future frames based on a history of observed frames (and sometimes robot actions). However, a comprehensive method for determining the fitness of different video prediction models at guiding the selection of actions is yet to be developed. Current metrics assess video prediction models based on human perception of frame quality. In contrast, we argue that if these systems are to be used to guide action, necessarily, the actions the robot performs should be encoded in the predicted frames. In this paper, we are proposing a new metric to compare different video prediction models based on this argument. More specifically, we propose an action inference system and quantitatively rank different models based on how well we can infer the robot actions from the predicted frames. Our extensive experiments show that models with high perceptual scores can perform poorly in the proposed action inference tests and thus, may not be suitable options to be used in robot planning systems.

2019

1. The impact of domain randomization on object detection: a case study on parametric shapes and synthetic textures Dehban, A., Borrego, J., Figueiredo, R., Moreno, P., Bernardino, A., and Santos-Victor, J. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2019 [Abs] [Git] [PDF] [bib]

   Recent advances in deep learning–based object detection techniques have revolutionized their applicability in several fields. However, since these methods rely on unwieldy and large amounts of data, a common practice is to download models pre-trained on standard datasets and fine-tune them for specific application domains with a small set of domain–relevant images. In this work, we show that using synthetic datasets that are not necessarily photo-realistic can be a better alternative to simply fine-tune pre-trained networks. Specifically, our results show an impressive 25% improvement in the mAP metric over a fine-tuning baseline when only about 200 labelled images are available to train. Finally, an ablation study of our results is presented to delineate the individual contribution of different components in the randomization pipeline.

2. Robotic Interactive Physics Parameters Estimator (RIPPE) Dehban, A., Cardoso, C., Vicente, P., Bernardino, A., and Santos-Victor, J. In IEEE International Conference on Developmental and Learning and on Epigenetic Robotics (ICDL-EPIROB) 2019 [Abs] [Git] [PDF] [bib]

   Recent advances in deep learning-based object detection techniques have revolutionized their applicability in several fields. However, since these methods rely on unwieldy and large amounts of data, a common practice is to download models pre-trained on standard datasets and fine-tune them for specific application domains with a small set of domain relevant images. In this work, we show that using synthetic datasets that are not necessarily photo-realistic can be a better alternative to simply fine-tune pre-trained networks. Specifically, our results show an impressive 25% improvement in the mAP metric over a fine-tuning baseline when only about 200 labelled images are available to train. Finally, an ablation study of our results is presented to delineate the individual contribution of different components in the randomization pipeline.

3. A robust and efficient framework for fast cylinder detection Figueiredo, R., Dehban, A., Bernardino, A., Santos-Victor, J., and Araújo, H. IEEE Robotics and Automation Systems 2019 [Abs] [Git] [PDF] [bib]

   In this work, a complete solution is provided for detecting and identifying cylindrical shapes, which are commonly found in household and industrial environments, using consumer-grade RGB-D cameras. Most standard approaches to detect and identify cylinders are not robust to outliers (e.g. points on other objects in the scene), which limits their applicability in realistic scenes. In addition, these methods fail to benefit from environmental constraints, e.g. the fact that cylinders often lie or stand on flat surfaces. To tackle the aforementioned limitations, we introduce three main novelties: (i) a point cloud soft voting scheme with curvature information that reduces the influence of outliers and noise, (ii) a selective sampling of the orientation space that favorsorientations known a priori, and (iii) a deep-learning based classifier to filter out objects with non-cylindrical appearance in the 2D images, thus further improving robustness to outliers. A set of experiments with synthetically generated data are used to assess the robustness of our fitting method to different levels of outliers and noise. The results demonstrate that incorporating the principal curvature direction within the orientation voting process allows for large improvements on cylinders parameters estimation. Furthermore, we demonstrate that combining the 2D deep-learning cylinder classifier with the 3D orientation voting scheme allows for large speed-up and accuracy improvements on cylinder identification. The qualitative and quantitative results with real data acquired from a consumer RGB-D camera, confirm the advantages of the proposed framework.

2018

1. Automatic generation of object shapes with desired functionalities Andries, M., Dehban, A., and Santos-Victor, A. CoRR 2018 [Abs] [arXiv] [bib]

   3D objects (artefacts) are made to fulfill functions. Designing an object often starts with defining a list of functionalities that it should provide, also known as functional requirements. Today, the design of 3D object models is still a slow and largely artisanal activity, with few Computer-Aided Design (CAD) tools existing to aid the exploration of the design solution space. To accelerate the design process, we introduce an algorithm for generating object shapes with desired functionalities. Following the concept of form follows function, we assume that existing object shapes were rationally chosen to provide desired functionalities. First, we use an artificial neural network to learn a function-to-form mapping by analysing a dataset of objects labeled with their functionalities. Then, we combine forms providing one or more desired functions, generating an object shape that is expected to provide all of them. Finally, we verify in simulation whether the generated object possesses the desired functionalities, by defining and executing functionality tests on it.

2. A generic visual perception domain randomisation framework for Gazebo Borrego, J., Figueiredo, R., Dehban, A., Moreno, P., Bernardino, A., and Santos-Victor, J. In IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC) 2018 [Abs] [Git] [PDF] [bib]

   The impressive results of applying deep neural networks in tasks such as object recognition, language translation, and solving digital games are largely attributed to the availability of massive amounts of high quality labelled data. However, despite numerous promising steps in incorporating these techniques in robotic tasks, the cost of gathering data with a real robot has halted the proliferation of deep learning in robotics. In this work, a plugin for the Gazebo simulator is presented, which allows rapid generation of synthetic data. By introducing variations in simulator-specific or irrelevant aspects of a task, one can train a model which exhibits some degree of robustness against those aspects, and ultimately narrow the reality gap between simulated and real-world data. To show a use-case of the developed software, we build a new dataset for detection and localisation of three object classes: box, cylinder and sphere. Our results in the object detection and localisation task demonstrate that with small datasets generated only in simulation, one can achieve comparable performance to that achieved when training on real-world images.

2017

1. A Deep Probabilistic Framework for Heterogeneous Self-Supervised Learning of Affordances Dehban, A., Jamone, L., Kampff, A., and Santos-Victor, J. In IEEE RAS International Conference on Humanoid Robots (Humanoids) 2017 [Abs] [PDF] [bib]

   The perception of affordances provides an actioncentered parametric representation of the environment. By perceiving an object’s visual features in terms of what actions they afford, novel behavior opportunities can be inferred about previously unseen objects. In this paper, a flexible deep probabilistic framework is proposed which allows an explorative agent to learn tool-object affordances in continuous space. To this end, we use a deep variational auto-encoder with heterogeneous probabilistic distributions to infer the most probable action that achieves a desired effect or to predict a parametric probability distribution over action consequences i.e. effects. Our experiments show the generalization of the method to unseen objects and tools and we have analyzed the influence of different design choices. Our framework goes beyond other proposals by incorporating various probability distributions tailored for each individual modality and by eliminating the need for any pre-processing of the data.

2. Shape-Based Attention for Identification and Localization of Cylindrical Objects Dehban, A., Figueiredo, R., Bernardino, A., Santos-Victor, J., and Araújo, H. In IEEE International Conference on Developmental and Learning and on Epigenetic Robotics (ICDL-EPIROB) 2017 [Abs] [PDF] [bib]

   In this paper, we propose a novel framework for detecting and identifying cylindrical shapes, commonly found in daily contexts, using multi-modal visual information provided by RBG-D cameras. The current state-of-the-art methods for cylinder detection are based on RANSAC and Hough transforms which estimate cylinder parameters using 3D point cloud information. However, the presence of distracting non-cylindrical shapes leads to time-consuming parametric fitting of wrong detections, compromising the efficiency of the whole method. We tackle the aforementioned problem by introducing a biologically plausible framework, incorporating a pre-attentive mechanism which learns in a supervised and data-efficient manner to selectively discard irrelevant shapes before further processing. A set of experiments with real data are conducted to assess the advantages of our framework. The results demonstrate that combining bottom-up 3D segmentation with top-down shape-based attention allows for large speedup and accuracy improvements on cylinder identification. The qualitative and quantitative results with real data acquired from a consumer RGB-D camera, confirm the advantages of the proposed framework.

3. Learning at the ends: From hand to tool affordances in humanoid robots Saponaro, G., Vicente, P., Dehban, A., Jamone, L., Bernardino, A., and Santos-Victor, J. In IEEE International Conference on Developmental and Learning and on Epigenetic Robotics (ICDL-EPIROB) 2017 [Abs] [PDF] [bib]

   One of the open challenges in designing robots that operate successfully in the unpredictable human environment is how to make them able to predict what actions they can perform on objects, and what their effects will be, i.e., the ability to perceive object affordances. Since modeling all the possible world interactions is unfeasible, learning from experience is required, posing the challenge of collecting a large amount of experiences (i.e., training data). Typically, a manipulative robot operates on external objects by using its own hands (or similar end-effectors), but in some cases the use of tools may be desirable; nevertheless, it is reasonable to assume that while a robot can collect many sensorimotor experiences using its own hands, this cannot happen for all possible human-made tools. Therefore, in this paper we investigate the developmental transition from hand to tool affordances: what sensorimotor skills that a robot has acquired with its bare hands can be employed for tool use? By employing a visual and motor imagination mechanism to represent different hand postures compactly, we propose a probabilistic model to learn hand affordances, and we show how this model can generalize to estimate the affordances of previously unseen tools, ultimately supporting planning, decision-making and tool selection tasks in humanoid robots. We present experimental results with the iCub humanoid robot, and we publicly release the collected sensorimotor data in the form of a hand posture affordances dataset.

4. Efficient Resource Allocation for Sparse Multiple Object Tracking. Figueiredo, R., Avelino, J., Dehban, A., Bernardino, A., Lima, P. U, and Araújo, H. In VISIGRAPP (6: VISAPP) 2017 [Abs] [PDF] [bib]

   In this work we address the multiple person tracking problem with resource constraints, which plays a fundamental role in the deployment of efficient mobile robots for real-time applications involved in Human Robot Interaction. We pose the multiple target tracking as a selective attention problem in which the perceptual agent tries to optimize the overall expected tracking accuracy. More specifically, we propose a resource constrained Partially Observable Markov Decision Process (POMDP) formulation that allows for real-time on-line planning. Using a transition model, we predict the true state from the current belief for a finite-horizon, and take actions to maximize future expected belief-dependent rewards. These rewards are based on the anticipated observation qualities, which are provided by an observation model that accounts for detection errors due to the discrete nature of a state-of-the-art pedestrian detector. Finally, a Monte Carlo Tree Search method is employed to solve the planning problem in real-time. The experiments show that directing the attentional focci to relevant image sub-regions allows for large detection speed-ups and improvements on tracking precision.

2016

1. Denoising auto-encoders for learning of objects and tools affordances in continuous space Dehban, A., Jamone, L., Kampff, A., and Santos-Victor, J. In IEEE International Conference on Robotics and Automation (ICRA) 2016 [Abs] [Git] [PDF] [bib]

   The concept of affordances facilitates the encoding of relations between actions and effects in an environment centered around the agent. Such an interpretation has important impacts on several cognitive capabilities and manifestations of intelligence, such as prediction and planning. In this paper, a new framework based on denoising Auto-encoders (dA) is proposed which allows an agent to explore its environment and actively learn the affordances of objects and tools by observing the consequences of acting on them. The dA serves as a unified framework to fuse multi-modal data and retrieve an entire missing modality or a feature within a modality given information about other modalities. This work has two major contributions. First, since training the dA is done in continuous space, there will be no need to discretize the dataset and higher accuracies in inference can be achieved with respect to approaches in which data discretization is required (e.g. Bayesian networks). Second, by fixing the structure of the dA, knowledge can be added incrementally making the architecture particularly useful in online learning scenarios. Evaluation scores of real and simulated robotic experiments show improvements over previous approaches while the new model can be applied in a wider range of domains.

2. A Moderately Large Size Dataset to Learn Visual Affordances of Objects and Tools Using iCub Humanoid Robot Dehban, A., Jamone, L., Kampff, A., and Santos-Victor, J. In ECCV Workshop on Action and Anticipation for Visual Learning 2016 [Abs] [HTML] [PDF] [bib]

   Affordances are defined as action opportunities that an environment offers to an agent: relations between visually perceived properties of an object, the possible actions afforded, and the effects of such actions. This notion can be generalized to the concept of tools: ie the visual appearance of a tool suggests what actions it affords to do on an object. Inspired by the amount of trials human babies need to learn these kinds of relations, we have gathered a relatively big dataset using iCub humanoid robot. The robot performs different actions using easy-to-rebuild tools on various objects selected from the YCB objects set and observes the results of the actions. The dataset can facilitate research in the areas of sensorimotor learning, active perception and cognitive developmental robotics.

3. A cognitive based driver’s steering behavior modeling Dehban, A., Sajedin, A., and Menhaj, MB. In International Conference on Control, Instrumentation, and Automation (ICCIA) 2016 [Abs] [PDF] [bib]

   This paper introduces an integrated computational model of driver steering behavior in turns. This model uses a cognitive architecture to explain how the driver can acquire information in his/her visual field and also how the driver manipulates its environment. The cognitive architecture incorporates an Artificial Neural Network (ANN) as the basis of the underlying control mechanism. The proposed model can be considered as a hybrid connectionist-symbolic cognitive architecture. In order to demonstrate the abilities of this model a simulation environment is developed through which driver behaviors data is collected and used for the purpose of training, validation and verification check of the model. The simulation results easily approve that the model’s behavior is almost perfectly aligned with that of the human driver.

4. A novel approach for optimization in dynamic environments based on modified artificial fish swarm algorithm Yazdani, D., Sepas-Moghaddam, A., Dehban, A., and Horta, N. International Journal of Computational Intelligence and Applications 2016 [Abs] [HTML] [PDF] [bib]

   Swarm intelligence algorithms are amongst the most efficient approaches toward solving optimization problems. Up to now, most of swarm intelligence approaches have been proposed for optimization in static environments. However, numerous real-world problems are dynamic which could not be solved using static approaches. In this paper, a novel approach based on artificial fish swarm algorithm (AFSA) has been proposed for optimization in dynamic environments in which changes in the problem space occur in discrete intervals. The proposed algorithm can quickly find the peaks in the problem space and track them after an environment change. In this algorithm, artificial fish swarms are responsible for finding and tracking peaks and several behaviors and mechanisms are employed to cope with the dynamic environment. Extensive experiments show that the proposed algorithm significantly outperforms previous algorithms in most of tested dynamic environments modeled by moving peaks benchmark.

2015

1. Robotic tool use and problem solving based on probabilistic planning and learned affordances Antunes, A., Saponaro, G., Dehban, A., Jamone, L., Ventura, R., Bernardino, A., and Santos-Victor, J. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. Workshop Learn. Object Affordances Fundamental Step to Allow Prediction Plan. Tool use 2015 [Abs] [bib]

   The ability of predicting the consequences of actions is a distinctive sign of human intelligence. Predictions are used to efficiently plan actions sequences that allow problem solving. This ability is acquired incrementally by humans during development through sensorimotor experience: ie interacting with objects while perceiving salient events. An appealing concept to represent this sensorimotor knowledge has been proposed in psychology under the name of object affordances: action possibilities that an object offers to an agent. Affordances are learned by the agent (animal or robot) and exploited for action planning. Clearly, endowing artificial agents with such cognitive capabilities is a fundamental challenge both in artificial intelligence and robotics. We propose a framework in which an embodied agent (ie in our case, the humanoid robot iCub) explores the environment and learns object affordances as relationships between actions, object visual properties and observed effects. The learned affordances enable a number of cognitive skills: eg i) predicting the effects of an action over an object, or ii) selecting the best action to obtain a desired effect. By exploring object-object interactions the robot can develop the concept of tool (ie a hand-held object that allows to obtain a desired effect on another object), and eventually use the acquired knowledge to plan sequences of actions to attain a desired goal (ie problem solving).

2. Neuro-ACT Cognitive Architecture Applications in Modeling Driver’s Steering Behavior in Turns Dehban, A., Menhaj, MB., and Sajedin, A AUT Journal of Modeling and Simulation 2015 [Abs] [bib]

   Cognitive Architectures (CAs) are the core of artificial cognitive systems. A CA is supposed to specify the human brain at a level of abstraction suitable for explaining how it achieves the functions of the mind. Over the years a number of distinct CAs have been proposed by different authors and their limitations and potentials were investigated. These CAs are usually classified as symbolic and sub-symbolic architectures. In this work, a novel hybrid architecture is proposed that encompasses a symbolic part (ie ACT-R) to explain the controlled aspects of behavior and a sub-symbolic part (ie Artificial Neural Networks) to describe automated skills. In order to demonstrate the capabilities of the proposed model, an experiment was conducted in which, a rather complex real life task was carried out by the model and its result were compared with those of human participants. Simulation results have shown promising capabilities of the new architecture in modeling complex human behavior.