5 Papers You Can’t-Miss: Reinforcement Learning | by Ulrik Thyge Pedersen | May, 2023

Reinforcement Learning (RL) is an important subfield in the area of machine learning that deals with agent programs learning actions in an environment to minimize a loss function in order to improve. It has been applied in areas such as robotics, games, finance and healthcare and has seen signification advancements in the last few years. New algorithms and architectures has been developed to tackle the complex challenges.

In this article we will take a deep dive into the most impactful research papers in the area of reinforcement learning. From a discussion on control to learning to play Atari, we will cover the latest advances in reinforcement learning. One of the main challenges is how we make agents learn more efficiently and these articles have made crucial advances to improve performance and discover novel applications to real-world problems.

By the end you will have a glimpse into the exciting world of reinforcement learning and its latest developments. Lets get started!

The first paper we will discuss is “Latent State Marginalization as a Low-cost Approach for Improving Exploration”. The authors propose a new approach to look at and distinguish the capabilities of agents in reinforcement learning. A new latent variable model is presented to be incoorperated into the maximum entropy (MaxEnt) framework. This allows agents to reason about uncertainty and make more informed decisions based on its surroundings

One crucial aspect of reinforcement learning is exploration. Agents must explore the environment in order to make informed decisions, discovers new actions and learn paths to a higher reward. Navigating between random noise and meaningful events is extremely challenging in a complex environment and this is the challenge where the MaxEnt framework is a proposed solution.

By applying the latent variable model into the MaxEnt framework, the paper proposes that agents can improve exploration and achieve better performance in navigation and complex tasks by learning policies. Not only is this more more efficient for the agents, but makes for a computationally efficient, cheap and scalable solution, making it more practical for real-world applications.

Exploration is a fundamental challenge in the field of reinforcement learning. The proposed method of the paper could be a potential solution. Improved efficiency, enhanced robustness and more intuitive learning could lead to RL being applied on novel problems where it was though impractical or impossible.

Interested in more? Check out the full article here!

The next paper on our list is “Dichotomy of Control: Separating What You Can Control from What You Cannot”. The theme of the article is about supervised learning in RL and proposes a new framework that can separate mechanisms within a policy from those outside of the policy. Dichotomy of Control (DOC) is the proposed framework and its uses supervised learning improve agent performance in highly stochastic environments.

Many RL problems are often stochasticity sources that are beyond the agents control, such as external disturbances and uncontrollable variables. These sources can negatively impact the performance of the agent and make it hard to learn an efficient policy. By separating controllable and uncontrollable factors in the environment, the DoC framework addresses this challenge.

The paper shows that better performance can be achieved using the DoC framework when compared to other state-of-the-art methods, such as decision transformer (DT) on a range of tasks. These tasks include robotic manipulation and simple games, such as Atari.

A novel framework, like DoC, addresses an important challenge by forcing the agents to only focus on things they can control and influence. A refocus to whats important is a promising and human-like direction for future research in the field of RL. DoC has the potential to improve performance and robustness of agents in complex, stochastic environments and make more efficient and effective solutions.

Interested in more? Check out the full article here!

The third item on the list of papers is “Emergence of Maps in the Memories of Blind Navigation Agents”. This paper shows that even blind agents, with no ability to identify its environment, does remarkably well in new environments. The article discovered that a agent with limited sensory information had great ability to explore and navigate new environments.

This suggest that reinforcement learning agents don’t necesarily need an explicit map in order to navigate successfully. The paper investigated apparent neural mechanisms that underlie successful navigation and reported interesting insight. They found that blind agents would have emerging collision detection neurons in they memory, make them able to navigate well in new, unseen environments.

This is a big deal for agents navigating novel or noise environments, especially when they have limited sensory information. The agents using the statistical structure of their environment to perform well in new environments, without being told explicit map building. Development of novel mechanics to navigate could inspire new and more biologically plausible reinforcement learning algorithms.

Interested in more? Check out the full article here!

The forth and second last paper we are going to look at is “GFlowNets and Variational Inference”. The paper explores generative flow networks and variational inference and their relationship. It also highlights advantages of GFlowNets in various tasks such as diversity capture and target distributions.

The authors proposes a novel approach using deep flow-based architecture called GFlowNets. These nets aims to solve and improve probabilistic generative modelling. The models are training using maximum likelihood estimation and can model complex, multimodel distributions with high accuracy. The new training procedure is coined Diversity-Promoting Variational Inference (DPVI) and its aims to maximize the sample diversity of the model.

These GFlowNets are highly efficient when it comes to capturing underlying structures of complex distributions. They outperform several state-of-the-art methods on a variety of benchmarks and the DPVI training procedure is also highlu effective at increasing diversity in samples generated.

These advances has endless applications in a variety of fields. Some of the most prominent areas include:

• Machine Learning
• Generative Modelling
• Image Synthesis
• Natural Language Processing

Being able to accurately model complex, multimodal distributions is essential when it comes to applications in the real world and GFlowNets is a promising new tool for this approach. Combined with DPVI training, generative models can be more efficient and promote diversity for generative AI such as image and language diversity.

Interested in more? Check out the full article here!

The last paper we are going to look at is “Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals”. This last paper proposes a framework for leveraging instruction manuals, written by humans, to improve reinforcement learning agents that play Atari games.

Read and Reward is the name of the newly proposed framework that incorporates text-based instructions into the agents behavior, making them able to learning quickly and effectively for novel tasks. By adding instructions, the agents achieved significantly higher scores in several Atari games, including Breakout and Pong.

To be able to extract instructions and gain knowledge from the text, agents using natural language processing techniques, such as keyword extraction and sentence clustering to identify the most relevant parts of the manuals and add them to the agents observation. This highly increased the agents learning curve, making them able to score highly in Atari games much quicker than agents without any instructions.

The ability to use NLP to give instructions to agents, makes training, learning and performance much more effective and efficient. The approach is not limited to reinforcement learning, but can also be extended to robotics or chatbots to learn human knowledge faster than before expected.

Interested in more? Check out the full article here!

The five papers we have discussed in this article represent some of the most impactful and exciting contributions to the field, from improving exploration and robustness capabilities to disentangling policy control and leveraging human knowledge for improved performance.

The area of reinforcement learning is booming, along with the rest of artificial intelligence. The area is growing fast and becoming more diverse, opening new problems, opportunities and horizons to discover and overcome.

The five papers are some of the most impactful, novel and impactful in the field of reinforcement learning. Improving exploration, robustness, detangling policies and leveraging human knowledge all signify new directions and heights for the area. The articles we discussed highlight the importance of new ideas, novel thinking and innovation and creativity in attacking problems and obstacles in reinforcement learning, with new, fresh approaches.

Robotics, natural language processing and gaming, reinforcement learning starts to touch many areas areas we once thought impossible and out of reach. As the field matures we can expect more exciting developments and breakthroughs in the coming years, stay tuned, reinforcement learning is here to stay!