We propose an efficient high-performance scaling algorithm based on the oriented polynomial image model. We develop a simple classification scheme that classifies the region around a pixel as an oriented or nonoriented block. Based on this classification, a nonlinear oriented interpolation is performed to obtain high quality video scaling. In addition, we also propose a generalization that can perform scaling for arbitrary scaling factors. Based on this algorithm, we develop an efficient architecture for image scaling. Specifically, we consider an architecture for scaling a Quarter Common Intermediate Format (QCIF) image to 4CIF format. We show the feasibility of the architecture by describing the various computation units in a hardware description language (Verilog) and synthesizing the design into a netlist of gates. The synthesis results show that an application specific integrated circuit (ASIC) design which meets the throughput, requirements can be built with a reasonable silicon area.