In industrial settings, robots are typically employed to accurately track a reference force to exert on the surrounding environment to complete interaction tasks. Interaction controllers are typically used to achieve this goal. Still, they either require manual tuning, which demands a significant amount of time, or exact modeling of the environment the robot will interact with, thus possibly failing during the actual application. A significant advancement in this area would be a high-performance force controller that does not need operator calibration and is quick to be deployed in any scenario. With this aim, this paper proposes an Actor-Critic Model Predictive Force Controller (ACMPFC), which outputs the optimal setpoint to follow in order to guarantee force tracking, computed by continuously trained neural networks. This strategy is an extension of a reinforcement learning-based one, born in the context of human-robot collaboration, suitably adapted to robot-environment interaction.