We have studied numerically the stability and defect nucleation in epitaxial layers on a substrate with lattice mismatch. Stress relaxation and energy barriers for misfit dislocation nucleation are estimated using modern methods for saddle point search based on a combination of activation with local repulsive potential and the Nudged Elastic Band method. Stress relaxation processes correspond to different transition paths from coherent to incoherent states of the epitaxial layer.Using a two-dimensional atomistic model with Lennard-Jones interacting potential, we and different equilibrium critical thickness and activation energy behavior for dislocation nucleation of epitaxial films under tensile and compressive strain. For tensile strain, the energy barrier decreases with thickness while it reaches a constant value for compressive strain.