Accepted for publication in IEEE Journal of Quantum Electronics.We use a nonlinear dynamics approach to study the deterministic behavior of ultra-pure microwave generators referred to as optoelectronic oscillators. In conventional studies, the standard nonlinear effects are very strongly rejected because they generate harmonics of the microwave frequency that are definitely out of the selective oscillator bandwidth. However, an important point we emphasize in this paper is that the nonlinearity still affects the slowly varying dynamics of microwave envelope, thereby inducing dynamical instabilities within the oscillator bandwidth. Starting from a full integro-differential model, we use the multiple timescales method to build a delay-differential equation for the slowly-varying complex envelope of the microwave. Then, the corresponding stationary solutions are derived, and the stability of their amplitude and phase is investigated in detail as a function of the feedback gain. We evidence essential bifurcation phenomena, and in particular we demonstrate that the generated microwave may turn unstable if the gain is increased beyond a precise critical value. This nonlinear dynamics approach therefore demonstrates that the ultra-pure microwave's amplitude does not monotonously increase with the gain. The theoretical study is confirmed by numerical simulations and experimental measurements.Y.K.C. acknowledges a research fellowship from the Région de Franche-Comté in France. Financial support from MEC (Spain) and FEDER under projects TEC2006-10009 (PhoDeCC) and FIS2007-60327 (FISICOS) is acknowledged.Peer reviewed