This paper considers a matching model with heterogenous jobs (unskilled and skilled) and workers (low and high-educated) which allows for on-the-job search by mismatched workers. The latter are high-educated workers who transitorily accept unskilled jobs and continue to search for skilled jobs. Our findings show that on-the-job search introduces an additional source of between and within-group wage inequality. Furthermore, the higher quit rate of mismatched workers exerts a negative externality on unskilled jobs and weakens the labour market position of low-educated workers. This last feature changes the effects of skill-biased technological change and it alters the response of the labour market to shifts in the skill distribution.