Courses

Introductory lectures:

  • Introduction to Queue theory and its applications [PDF].
  • What is Operations Research? History, Subjects and Applications [PDF].
  • How Randomness Rules Our Lives? An Intro to Probability Theory [PDF].

Advanced lectures:

  • Intro to Stochastic-process limits [PDF].
  • Ways to Prove Weak convergence in functional spaces [PDF].
  • Staffing and scheduling to differentiate service levels [PDF]

Graduate and Undergraduate Courses Taught:

Columbia University

  • SIEO 3600 – Intro to Probability and Statistics (Undergraduate), Spring 2010  

North Carolina State University 

  • ISE/OR 760 – Applied Stochastic Models (Ph.D. core course in IE SA&O and OR), Fall 2011 – present [Syllabus]
    • Description: This is a Ph.D. level course in stochastic models designed to develop a solid understanding of uncertain phenomena and the mathematical tools used to model and analyze random observations in industrial engineering. This course will provide both rigorous mathematical basis (proof based) and useful engineering values (applications). This course will cover 5 major topics: (i) review of probability theory, (ii) discrete-time Markov chain, (iii) Poisson process and its generalizations, (iv) continuous-time Markov chain and (v) renewal counting process. This course has 12 homework sets (each having 8 problems), two midterms and one final exam. The textbooks are “Introduction to probability models”, 10th Ed, Sheldon Ross; and “Stochastic Processes”, 2nd Ed, Sheldon Ross. This course is part of the ISE/OR Ph.D. qualifying exam courses.
    • Textbooks:
      • Ross, S. Introduction to probability models, 11th Ed., Academic Press, 2014.
      • Ross, S. Stochastic processes, 2nd Ed., Wiley, 1995.
  • ISE/OR 790 – Stochastic Models with Applications in Queueing Theory (Ph.D.), Spring 2012 [Syllabus]
    • Description: This is a seminar course on stochastic modeling with applications in queueing theory, as a natural continuation of ISE 760. One goal is to help students learn about various application context. A second goal is to focus on a class of mathematical models and analysis techniques that have proven useful in the application context. As is almost always the case in operations research, these models and analysis techniques have many other applications, so the course can be useful even if you are primarily interested in other applications. There is no exams or textbooks. Lecture notes and research papers will be distributed.
  • ISE 362 – Intro to Stochastic Models (undergraduate), Spring 2013, Fall 2013, Fall 2014 [Syllabus]
    • Description: This course is an introduction to stochastic models with applications in industrial engineering. This course also can serve as part of an introduction to probability theory and its applications, being a sequel to ST 371 – Introduction to Probability Theory. In a first course on probability (such as ST 371) students learn about random variables and their probability distributions. In that first course, attention is usually focused on only one or two random variables. In this subsequent course we will extend the focus to stochastic processes, which are collections of random variables, usually indexed by time (either discrete or continuous). Stochastic modeling appears to be an important and fundamental tool in operations research. For example, we might use stochastic processes to model the evolution of a stock price over time, the damage claims received by an insurance company over time, the work-in-process inventory in a factory over time or the number of calls waiting in telephone call center over time, all of which evolve with considerable uncertainty.
    • Textbook: “Introduction to probability models”, 10th Ed, Sheldon Ross.
  • ISE/OR 761 – Advanced Stochastic Models and Queues (Ph.D.), Spring 2015 – present [Syllabus]
    • Prerequisit: ISE/OR 760
    • Description: This course is a sequel to ISE 760 Applied Stochastic Models, aiming at supplementing ISE 760 by introducing new stochastic processes with an emphasis on queueing theory. Queueing theory is the mathematical theory of congestion as is associated with delays while waiting in a line or queue for service in a system. Examples of such systems include banks, post offices and supermarkets, as well as telecommunication systems involving telephones, computer networks, internet/world wide web, inventory, health care and manufacturing processes. Much of the theory relies heavily on the use of probability theory and stochastic processes (of which queueing theory is viewed as a subfield). There is also significant interplay with other fields such as scheduling theory, inventory theory and insurance risk theory. Queueing theory is a central part of operations research.
  • ISE/OR 762 – Stochastic Simulation techniques (Ph.D.), Spring 2016 – present [Syllabus]
    • Prerequisits: ISE/OR 760, ISE 560
    • Description: Advanced topics in stochastic system simulation, including random variate generation, output estimation for stationary and non-stationary models, performance optimization techniques, variance reduction approaches. Student learn to apply these techniques to the simulation design of complex systems.
    • Textbooks:
      • S. M. Ross, Simulation. 5th Edition, Academic Press, 2014
      • A. M. Law, Simulation Models and Analysis. 5nd Edition, McGraw-Hill
      • P. Glasserman, Monte Carlo Methods in Financial Engineering. Springer. 2003
  • FIM/MA 548 – Monte Carlo Methods for Financial Mathematics (Master), Spring 2018
    • Prerequisits:  MA 421, 341, 405 (ST 501, MA 547 recommended)
    • Description: Monte Carlo (MC) methods for accurate option pricing, hedging and risk management. Modeling using stochastic asset models (e.g. geometric Brownian motion) and parameter estimation. Stochastic models, including use of random number generators, random paths and discretization methods (e.g. Euler-Maruyama method), and variance reduction. Implementation using Matlab. Incorporation of the latest developments regarding MC methods and their uses in Finance.
    • Textbooks:
      • P. Glasserman, Monte Carlo Methods in Financial Engineering, Springer, 2003.
      • S. M. Ross, Simulation. 5th Edition, Academic Press, 2014
  • ISE789/OR791 – Dynamic Stochastic Optimization and Reinforcement Learning (graduate), Spring 2021. [Syllabus] [Project]
    • Prerequisits: ISE/OR 760, 761, 505
    • Description: This is a graduate level class on dynamic decision making and reinforcement learning (RL). RL is one of the three basic machine learning paradigms, alongside supervised learning and unsupervised learning. RL concerns with how an “agent” aims to maximize its cumulative reward by dynamically interacting with the “environment”, where the focus is to balance between the exploration of uncharted territory and the exploitation of current knowledge. The goal of this class is to teach theories and solution methods of RL in the context of industrial engineering, operations research, and operations management. Unlike most extant RL textbooks which often use examples arising from computer science, robotics, psychology and physics, in this class we will discuss how to apply RL methods to solve IE and OR problems including: machine maintenance and replacement, resource allocation, investment, gambling (e.g., blackjack, two slot machine problem), pricing an American option, inventory control, revenue management, queueing control, Baysian sequential analysis, and multi-armed bandits.
    • Textbooks:
      • S. Ross, Introduction to Stochastic Dynamic Programming, Academic Press, 1983.
      • R. Sutton and A. Barto, Reinforcement Learning: An Introduction, A Bradford Book, 2018.
      • C. Szepesvari, Algorithms for Reinforcement Learning, 2010.
      • D. Bertsekas, Reinforcement Learning and Optimal Control, 2019.
      • D. Bertsekas, Dynamic Programming and Optimal Control, 2012.

 

Future classes:

  • ISE 495 – Introduction to Service Engineering (undergraduate), Term: TBD.
    • Description: Service systems currently make up 60-80% of western economies. Important examples are healthcare systems (hospitals), financial services (banks) and telephone and internet services. In many respects, service systems are similar to manufacturing systems, communication systems, and transportation systems, but there are important differences, largely due to the significant role played by humans in providing the service. Fortunately, there is a growing service science that provides skills and tools to manage service systems. In this course, service systems are viewed as stochastic networks. Thus the main theoretical framework is queueing theory, which primarily involves a large class of stochastic models. However, the subject matter is highly multi-disciplinary; hence alternative frameworks will be useful as well, including ones from Statistics, Psychology, and Marketing. The course will provide a framework for modeling service systems and techniques that are useful to design, analyze, and operate service systems. We will use real service system data from banks, hospitals, and call centers to demonstrate the use of the decision support tools.
  • ISE 76x – Applied Stochastic Models II (Ph.D.), Term: TBD.
    • Description: This future course will be a sequel to ISE 760 Applied Stochastic Models, aiming at supplementing ISE 760 by introducing new stochastic processes, such as martingales, Brownian motions, branching processes, regenerative processes and Markov renewal processes. If your research involves using stochastic models; if you want to know more stochastic processes; if liked ISE 760 or if you took ISE 760 and got a good grade, you should consider taking this course. There will be about 6 homework sets (biweekly), a midterm and a take home final exam.
  • ISE589 – Reinforcement Learning in IE and OR (graduate), TBD.
    • Description: This is a master-level variate of my Ph.D. class “ISE789-Dynamic Stochastic Optimization and Reinforcement Learning”. RL is one of the three basic machine learning paradigms, alongside supervised learning and unsupervised learning. RL concerns with how an “agent” aims to maximize its cumulative reward by dynamically interacting with the “environment”, where the focus is to balance between the exploration of uncharted territory and the exploitation of current knowledge. The goal of this class is to teach theories and solution methods of RL in the context of industrial engineering, operations research, and operations management. Unlike most extant RL textbooks which often use examples arising from computer science, robotics, psychology and physics, in this class we will discuss how to apply RL methods to solve IE and OR problems including: machine maintenance and replacement, resource allocation, investment, gambling (e.g., blackjack, two slot machine problem), pricing an American option, inventory control, revenue management, queueing control, Baysian sequential analysis, and multi-armed bandits. Unlike its Ph.D. level analog ISE789 which includes more careful theoretical derivations, this course will focus more on the algorithmic implementation of various RL methods with detailed demonstrations of how to use Python (along with practical packages such as Pytorch and TensorFlow) in RL problems.