This brief studies the problem of developing an area-time efficient VLSI architecture for a novel self-feedback Convolutional Neural Network (CNN). Self-feedback CNNs offer the promise of high-precision object detection amidst occlusions. However, the size of a typical network required for practical applications presents a challenge for embedded system development. We first present the structure of the self-feedback CNN. We then present an efficient systolic array architecture for the self-feedback CNN with low on-chip memory requirement. The self-feedback CNN has been tested on the KITTI benchmark dataset and it achieves high accuracy for detecting occluded cyclists and pedestrians. FPGA implementation of the proposed architecture on Xilinx Virtex7 XC7VX485T achieves roughly 1.14 Tera Operations per second (TOP/s) at 386 MHz with 9× reduction in on-chip memory requirement compared to recent CNN architectures. © 2004-2012 IEEE.