Preliminary investigations on accelerometer-based activity recognition in construction have shown that it has good potential to be utilized for recognizing categories of work in a construction trade. Selecting the accelerometer locations is an important consideration in activity recognition studies, but currently it is decided primarily on the basis of comfort requirements. This article proposes a methodology for selecting the location of accelerometers using video annotations and decision trees. A video annotation tool is used to track the movement of body segments and decision tree algorithm helps to prioritize the relevant body segments for classifying activities. A two-phase experimental study was conducted to assess the methodology. In the first phase, video annotation studies were carried out on four bricklayers, and based on decision tree analysis three locations: right lower arm, left lower arm, and waist were selected. In the second phase, an activity recognition study was conducted on another group of bricklayers with accelerometers attached at the selected locations. The results of study show that the location of accelerometer has a significant influence on accuracy and the proposed methodology is effective in selecting accelerometer locations. In the current study only bricklaying activity was considered, however, the methodology is generic and has the potential to be applied to objectively evaluate accelerator placement location for a wide range of structured activities. © 2013 Computer-Aided Civil and Infrastructure Engineering.