Recent developments on the complexity of the non-commutative determinant and permanent (Arvind and Srinivasan, 2010; Chien et al., 2011; Bläser, 2013; Gentry, 2014) have settled the complexity of non-commutative determinant with respect to the structure of the underlying algebra. Continuing the research further, we look to obtain more insights on hard instances of non-commutative permanent. We show that any Algebraic Branching Program (ABP) computing the Cayley permanent of a collection of disjoint directed two-cycles with distinct variables as edge labels requires exponential size on an average, where the average is taken over all possible permutations of variables. For graphs where every connected component contains at most six vertices, we show that evaluating the Cayley permanent over any algebra containing 2 × 2 matrices is #P complete. Further, we obtain efficient algorithms for computing the Cayley permanent/ determinant on graphs with bounded component size, when vertices within each component are not far apart from each other in the Cayley ordering. This gives a tight upper and lower bounds for size of ABPs computing the permanent of disjointtwo-cycles. Finally, we exhibit more families of non-commutative polynomial evaluation problems that are complete for #P. Our results demonstrate that apart from the structure of underlying algebras, relative ordering of the variables plays a crucial role in determining the complexity of non-commutative polynomials. © 2019 Elsevier B.V.