Modeling the impact of the order flow on asset prices is of primary importance to understand the behavior of financial markets. Part I of this paper reported the remarkable improvements in the description of the price dynamics which can be obtained when one incorporates the impact of past returns on the future order flow. However, impact models presented in Part I consider the order flow as an exogenous process, only characterized by its two-point correlations. This assumption seriously limits the forecasting ability of the model. Here we attempt to model directly the stream of discrete events with a so-called Mixture Transition Distribution (MTD) framework, introduced originally by Raftery (1985). We distinguish between price-changing and non price-changing events and combine them with the order sign in order to reduce the order flow dynamics to the dynamics of a four-state discrete random variable. The MTD represents a parsimonious approximation of a full high-order Markov chain. The new approach captures with adequate realism the conditional correlation functions between signed events for both small and large tick stocks and signature plots. From a methodological viewpoint, we discuss a novel and flexible way to calibrate a large class of MTD models with a very large number of parameters. In spite of this large number of parameters, an out-of-sample analysis confirms that the model does not overfit the data.