The rapid advancement of artificial intelligence (AI) techniques has unlocked new avenues to transform various fields, including operations research (OR). In recent years, we've delved into integrating AI within the OR process, particularly emphasizing the use of natural language processing (NLP) to automatically generate formulations for OR problems, streamlining the modeling process and reducing manual effort. We constructed two benchmark datasets with varying complexity, where each instance contains an OR problem described in natural language alongside its corresponding ground truth mathematical model. We evaluated both open-source and commercial large language models (LLMs) on these datasets and calculated their accuracy using an innovative automated evaluation method. Through our development and assessment of various prompting approaches based on intermediate planning and multi-turn model construction, we achieved high accuracy for simpler OR problems and commendable accuracy for the more complex benchmark dataset. Our findings highlights a promising future for automating OR problem formulations from their natural language description, which seemed impossible until now. While we have made significant strides, we acknowledge that challenges remain and ongoing research is essential.