Control of nonlinear, complex and black-boxed greenhouse system with reinforcement learning

Keywords—System Engineering, Deep Learning, Machine Learning, Reinforcement Learning

Introduction

Frequently, the approaches in nonlinear system control is to model the target system in highly sophisticated level. For example, systems biologist starts with defining the components of the system; the number of components are usually tens but sometimes thousands. The next step is to revealing interaction among all components and to draw interactional network. Continuous scaled systems are usually modeled in ordinary differential equation model [2][3]. When the number of component is too great to handle, researchers apply discrete model such as Boolean networks and write full truth table for individual interactions or establish Boolean equations for each component’s input condition [4]. After modeling the system, they apply complicated control methods [5][6][7][8], which usually have non-polynomial computational cost.

Those routine is time-consuming and not realistic in two points. First, when the model is big enough, a researcher may fail to gather all interactions among the components because current approaches mainly depends on paper search. Second, control methods for nonlinear systems often takes exponential time complexity so applying them on large scaled system is not feasible. Not to mention most of the methods are not even applicable when the model is black-box status.

Thanks to 상상텃밭 Inc.

The idea of our work is to provide a control method for black-boxed complex system whose input and output are only observable variables.

Reinforcement Learning Backgrounds

Reinforcement learning is interactions between the environment and agent over some discrete time steps, where the agent learns how to apply action on environment to maximize future rewards. At each time step t, the agent observes current state $s_{t}$ and decide an action $a_{t}$ according to its policy $\pi$ which, is a function whose domain is $s_{t}$ and has possible actions as codomain. The model decides future action by approximation of action value which is expected future rewards by applying $a_{t}$ to the environment.

Critic-only methods

Critic-only methods, value based reinforcement learning methods in other words, aims to approximate the value or action-value function Q (s, a; w), where $w$ is internal parameter for Q function [9] and is updated with various reinforcement learning algorithms [10]. For example, in one-step Q-learning [11], the learning is done by minimizing the loss function;

$L_{t}\left( w_{t} \right)\mathbb{= \ E}\left( r + \ \gamma\max_{a'}{Q(s',\ a';\ }w_{t - 1}) - Q(s,\ a,\ w_{t}) \right)^{2}$, where $s'$ is the state encountered after state s and $\gamma$ is discount factor. The output value of value-based reinforcement learning agent neural network is Q.

Actor-only methods

Policy gradient methods belongs to actor-only methods. The output of policy-based reinforcement learning agent neural network is the policy function $\pi_{\theta}(s,\ a)$, not Q itself. [12] Policy based reinforcement learning is to find best $\theta$ for $\pi_{\theta}$. To measure the quality of $\pi_{\theta}$, policy based approaches suggest various objective functions. By the policy gradient theorem proposed by Sutton et al., the policy gradient is described as

$\sum_{s}^{}{\sum_{t = 0}^{\infty}{\gamma^{t}Pr\{ s_{t} = s|s_{0},\ \pi\}}}\ \sum_{a}^{}{\frac{\partial\pi(s,\ a)}{\partial\theta}Q^{\pi}(s,a)}$.

Actor-critic policy gradient

Actor-critic methods aims to combine and maximize the strong point of value-based and policy-based models. This approach train 2 models, critic model and actor model, at the same time.

Critic is a temporal difference learning algorithm with a linearly parameterized approximation architecture for action-value function Q. [13] With linear approximation, with the basis function $\varnothing,$ the value function is described as $Q_{w}(s,\ a) = \ \varnothing{(s,\ a)}^{T}w$. [14][15] It updates action-value function parameter w by linear TD. Critic estimates current policy with Q function. Actor updates policy parameter $\theta$, in direction suggested by critic.

Actor-critic algorithms follow an approximate policy gradient:

$\nabla_{\theta}J(\theta)\mathbb{\approx E\lbrack}\ \nabla_{\theta}\log{\pi_{\theta}(s,\ a)Q_{w}(s,\ a)\rbrack\ }$,

$\mathrm{\Delta}\theta = \alpha\nabla_{\theta}\log{\pi_{\theta}(s,\ a)Q_{w}(s,\ a)}$. [14].

Method

Our approach is implementation of actor-critic policy gradient based reinforcement learning agent with artificial neural networks. This deep reinforcement learning algorithm tries to find out how to maximize the future rewards: maximize control performance.

Network Architecture

The agent is combined actor-critic networks. The structural backbone is described on Figure 1 (a). The structure is same as pemami4911’s work on his GitHub repository [16] but parameters are changed to fit for nonlinear, black-boxed and complex system control.

Actor network has 4 layers of fully connected neural networks. The activation functions are ReLu except the final layer; it is softmax function. The first and second layers has 128 neurons while the third layer has 32 neurons. The number of the neurons in final layer is same with the number of possible actions. The actor produces one-hot vector formed output corresponding with an action to be applied to the environment. Input to the actor is observation only. Network parameters are updated with critic output value.

Critic network has slightly unique structure. It is 3-layered FNN model, which has fuse operation on the second layer. Another single layer receives actor’s output to be merged with encoded observation information. The backpropagation of critic network follows both subnetworks at the same time so this network naturally trains how to optimize the actor policy by evaluating actor policy regarding the observation at the same time.

Network parameters are updated to reduce the mean squared error between previously predicted Q value with discounted real rewards.

Training

For each step, the agent receives observation form the environment. Actor decides which action to make regarding the observed state. For each step, the critic updates its internal parameter for Q function and returns gradient for actor. Then the actor is updated with the given gradient. This routine is repeated until the environment returns game over signal; after an episode is done, new episode starts with cleared environment.

Expereiment

Environment

The environment used for experiment is artificial greenhouse simulator. It simulates a greenhouse on very barren environment; weather changes a lot and sunlight is too strong so its inside circumstance becomes detrimental for the plants only in 10 hours. The derivatives of internal variables are proportional to difference between the internal value and external value and the coefficient for the derivative is chosen stochastically between [0.01, 0.2]. Perturbation on the external weather and entrance of human user randomly occurs.

The Environment has 7 observable values; current temperature, current humidity of air, current humidity of ground, current height of the plants, pesticide density, the existence of human and insects, and receives 8 actions; fan on, curtain open, water injection inside, water injection outside, pesticide injection, light on, spraying nutrients and harvest.

Figure 1 (b) shows the complexity of greenhouse system. Each action affects several internal values at the same time so any action can cause undesirable effects on the system. For example, if an agent take ‘curtain open’ action to reduce internal temperature, current humidity of both air and ground is changed with gradient calculated with outside values, and the pesticide density drops too.

Usually, Smart-farm owners apply several PIDs to maintain each value individually. However, it is very inefficient; notwithstanding the initial cost, it consumes lots of energies; it’s common for them to activate the air conditioner, dehumidifier and humidifier at the same time to maintain all the variable at the same time with individual PIDs. The agent of our method learned how to avoid such problem while maintaining the environment appropriately, taking least costs. If the environment is still sustainable after each hour, it returns +1 reward; total reward is hours for which the agent can maintain the environment.

Details on the environment are described on the supplementary information.

Training

The agent only observes current status but never receives information on interactions of internal variables to figure out appropriate control input. The environment is initialized as follows; allowed temperature range as [10, 100], allowed air humidity as [10, 90], allowed ground humidity as [30, 80], initial temperature as 21, initial air humidity as 50, initial ground humidity as 50. The episode is interrupted even a single variable exceeds the range.

To avoid converging in local minimum, actor’s sample action is ignored with probability eps, which is initially 0.95 and decays to 10% per 9 episodes; initially the model train and calculates value and policy for random action. For each step, the action and rewards are stored in a queue and 256 samples are drawn for mini batch.

Computational Environments

The agent was implemented with Tensor Flow on python 3. Numpy library is used for training routine and the implementation of environment. NVIDIA GTX 1080 8gb was used for training.

Results

We compared actor-critic network with conventional PID and Deep Q learning algorithm. Deep reinforcement learning approach showed better performance than conventional PID approach.

Conventional PID

PID controller measures each observable variable, and apply actions which can draw it into normal state. We applied it with 3 different conditions.

Trial A: Apply actions at every hour

Trial B: Apply actions with 3-hour interval

Trial C: Apply actions after 6^th hour.

Results from Trial A and B was frustrating. The environment itself requires average 10 hours to be destroyed without any control input but the result showed that PID accelerated destruction of the system. The trial C started after 5 hours passed, and managed to maintain the environment slightly longer than 10 hours. (Table 1)

It was mainly due to side effects of each control inputs. The major variable for the system is temperature. However, most control input related to temperature drops ground humidity dramatically so the system easily get dry.

Deep Q learning

Deep Q learning suggested by Google DeepMind [1], implemented with 4 layers of fully connected networks failed to achieve desirable performance. As the circumstance is sustainable up to 10 hours without any actions, this model definitely failed to learn how to maintain the greenhouse. It sometimes showed more than 10 scores but most of the trial showed less than 10 scores.

Actor-critic policy gradient

Our model showed outstanding result. After 2,000 episodes, the score increased dramatically and it succeeded to maintain the environment for 200~350 hours which is enough for farmers to grow and harvest lettuce. Regarding that the artificial greenhouse is unrealistically infertile, it can already be applied commercially in real world especially in the climate around the Republic of Korea.

Conclusion

We showed the performance comparison between PID and simple DQN with actor-critic network on controlling so extreme environment even a single action changes several internal values of the system. Moreover, the system is nonlinear and observation is limited so the agent doesn’t know the internal dynamics (black-boxed). It is not surprising that simple DQN, critic-only approach, failed to learn how to control such complex system.

Actor-critic policy gradient method showed surprisingly great performance because the complex rules are naturally trained not only on the action-value function but also on the policy gradient so that the agent could to figure out the appropriate policy and future rewards at the same time. We suggest applying actor-critic policy gradient on complex system control problems.

References

[1] Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013).

[2] Xu, Alexander M., and Paul H. Huang. "Receptor tyrosine kinase coactivation networks in cancer." Cancer research 70.10 (2010): 3857-3860.

[3] Grieco, Luca, et al. "Integrative modelling of the influence of MAPK network on cancer cell fate decision." PLoS Comput Biol 9.10 (2013): e1003286.

[4] Fumia, Herman F., and Marcelo L. Martins. "Boolean network model for cancer pathways: predicting carcinogenesis and targeted therapy outcomes." PloS one 8.7 (2013): e69008.

[5] Mochizuki, Atsushi, et al. "Dynamics and control at feedback vertex sets. II: A faithful monitor to determine the diversity of molecular activities in regulatory networks." Journal of theoretical biology 335 (2013): 130-146.

[6] Liu, Yang-Yu, Jean-Jacques Slotine, and Albert-László Barabási. "Controllability of complex networks." Nature 473.7346 (2011): 167-173.

[7] Wu, Fang-Xiang, et al. "Transittability of complex networks and its applications to regulatory biomolecular networks." Scientific reports 4 (2014): 4819.

[8] Gao, Jianxi, et al. "Target control of complex networks." Nature communications 5 (2014).

[9] Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine learning 8.3-4 (1992): 279-292.

[10] Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International Conference on Machine Learning. 2016.

[11] Watkins, Christopher John Cornish Hellaby. Learning from delayed rewards. Diss. University of Cambridge, 1989.

[12] Sutton, Richard S., et al. "Policy gradient methods for reinforcement learning with function approximation." Advances in neural information processing systems. 2000.

[13] Konda, Vijay R., and John N. Tsitsiklis. "Actor-critic algorithms." Advances in neural information processing systems. 2000.

[14] David Silver, “Lecture 7: Policy Gradient”, UCL Course on RL, 2015, Lecture.

[15] Grondman, Ivo, et al. "A survey of actor-critic reinforcement learning: Standard and natural policy gradients." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42.6 (2012): 1291-1307.

[16] https://github.com/pemami4911/deep-rl/tree/master/ddpg

Supplementary Information

This section covers details of the greenhouse environment.

Hidden Parameters

The environment has both observable and hidden parameters. For example, (a) outside temperature, (b) outside humidity, (c) weather, (d) temperature of the water are not observable to the agent. But they affect the internal state change.

Observable Parameters

The environment Vinylhouse provides observation of the system status. An agent can observe (a) current temperature inside the greenhouse, (b) current humidity of inside air, (c) current humidity of the ground, (d) current height of plants inside, (e) current density of pesticide, (f) existence of a human inside the greenhouse, (g) existence of insect inside the greenhouse.

Gradient

We supposed that the internal parameter of this nonlinear system is changed by simple different equation below;

$\frac{\Delta s}{\Delta t} = \alpha(\varphi - \ s)$,

where s is single parameter such as current humidity and $\varphi$ is comparison partner. $\alpha$ is random float between [0.01, 0.2]; this value is randomly chosen for every step of gradient calculation to increase difficulty of maintenance problem.

The update for internal parameter can be described;

$s_{t + 1} = s_{t} + \ \mathrm{\Delta}s_{t}$, where

$\mathrm{\Delta}s_{t} = \ \frac{\Delta s_{t}}{\Delta t}\ \Delta t$ = $\alpha\left( \varphi - \ s_{t} \right)\Delta t$.

As $\Delta t$ is just 1 for each discrete time step, update rule for internal parameter is simply approximated to $s_{t + 1} \approx s_{t} + \ \alpha\left( \varphi - \ s_{t} \right)$.

Actions

The environment may receive 8 actions. Each action is modeled to realistic ones. Usually, smart farm owners apply several PIDs to maintain each value individually. However, it is very inefficient; notwithstanding the initial cost, it consumes lots of energies; it’s common for them to activate the air conditioner, dehumidifier and humidifier at the same time to maintain all the variable at the same time with individual PIDs. The agent will learn how to avoid such problem while maintaining the environment appropriately, taking least costs.

Fan On

Open the window and activate ventilator. It continues for 10 hours. For each hour, current temperature, humidity of air, humidity of ground, density of pesticide changes. $T_{t}$ is inside temperature at time t and ${H\_ a}_{t}$ is humidity of the air at time t and $P_{t}$ is pesticide density at time t.

$$T_{t + 1} \approx T_{t} + \ \alpha\left( Outside\_ temparature - \ T_{t} \right)$$

$${H\_ a}_{t + 1} \approx {H\_ a}_{t} + \ \alpha\left( Outside\_ humidity - \ {H\_ a}_{t} \right)$$

$${H\_ g}_{t + 1} \approx {H_{g}}_{t} + \ 0.1\ \times \ \alpha\left( {H\_ a}_{t} - \ {H\_ g}_{t} \right)$$

$$P_{t + 1} \approx 0.9\ {\times \ P}_{t}$$

Curtain Open

Open the side curtain of greenhouse. It continues for 30 hours. For each hour, current temperature, humidity of air, humidity of ground, density of pesticide changes. $T_{t}$ is inside temperature at time t and ${H\_ a}_{t}$ is humidity of the air at time t and $P_{t}$ is pesticide density at time t.

$$T_{t + 1} \approx T_{t} + \ 3\ \times \ \alpha\left( Outside\_ temparature - \ T_{t} \right)$$

$${H\_ a}_{t + 1} \approx {H\_ a}_{t} + 3\ \times \ \alpha\left( Outside\_ humidity - \ {H\_ a}_{t} \right)$$

$${H\_ g}_{t + 1} \approx {H\_ g}_{t} + 0.2\ \times \ \alpha\left( {H\_ a}_{t} - \ {H\_ g}_{t} \right)$$

$$P_{t + 1} \approx 0.8\ \times \ P_{t}$$

Water Injection Inside

Switch on the sprinkler inside the greenhouse. It continues for 10 hours. For each hour, current temperature, humidity of air, humidity of ground, density of pesticide changes. $T_{t}$ is inside temperature at time t and ${H\_ a}_{t}$ is humidity of the air at time t and $P_{t}$ is pesticide density at time t.

$$T_{t + 1} \approx T_{t} + \ 3\ \times \ \alpha\left( water\_ temparature - \ T_{t} \right)$$

$${H\_ a}_{t + 1} \approx 1.1\ \times \ {H\_ a}_{t}$$

$${H\_ g}_{t + 1} \approx {1.3\ \times \ H\_ g}_{t}$$

$$P_{t + 1} \approx 0.7\ \times \ P_{t}$$

Water Injection Outside

Activate rooftop water flow. It continues for 20 hours. For each hour, current temperature, humidity of air, humidity of ground changes. $T_{t}$ is inside temperature at time t and ${H\_ a}_{t}$ is humidity of the air at time.

$$T_{t + 1} \approx T_{t} + \ 0.5$$

$${H\_ a}_{t + 1} \approx 0.9\ \times \ {H\_ a}_{t}$$

$${H\_ g}_{t + 1} \approx {0.95\ \times \ H\_ g}_{t}$$

Pesticide Injection

Spray pesticide inside the greenhouse. It continues for 10 hours. For each hour, humidity of air, humidity of ground, density of pesticide changes. ${H\_ a}_{t}$ is humidity of the air at time t and $P_{t}$ is pesticide density at time t.

$${H\_ a}_{t + 1} \approx 1.1\ \times \ {H\_ a}_{t}$$

$${H\_ g}_{t + 1} \approx {1.2\ \times \ H\_ g}_{t}$$

$$P_{t + 1} \approx 2\ \times \ P_{t}$$

Light on

Turn on the plant growth stimulating light. It continues for 50 hours. For each hour, current temperature, humidity of air, humidity of ground changes. $T_{t}$ is inside temperature at time t and ${H\_ a}_{t}$ is humidity of the air at time.

$$T_{t + 1} \approx {1.01\ \times \ T}_{t}$$

$${H\_ a}_{t + 1} \approx 0.95\ \times \ {H\_ a}_{t}$$

$${H\_ g}_{t + 1} \approx {0.98\ \times \ H\_ g}_{t}$$

Nutrient Spray

Spray nutrient inside the greenhouse. It continues for 10 hours. For each hour, humidity of air, humidity of ground, density of pesticide changes. ${H\_ a}_{t}$ is humidity of the air at time t and $P_{t}$ is pesticide density at time t.

$${H\_ a}_{t + 1} \approx 1.1\ \times \ {H\_ a}_{t}$$

$${H\_ g}_{t + 1} \approx {1.2\ \times \ H\_ g}_{t}$$

$$P_{t + 1} \approx 0.8\ \times \ P_{t}$$

Harvest

Doesn’t change any internal variable. If plant height is not sufficient to be harvested or too tall, stop the episode. Otherwise, put huge reward.

Perturbations

The environment got perturbations by outside circumstance at every time step.

By weather change

Periodically the system decided whether to change the weather or not. If whether is changed, also outside temperature is changed too. Apply gradient on inside temperature.

By rain

If it’s raining outside, set outside humidity value as maximum. Then apply gradient for inside air humidity. Also drop the temperature along the gradient.

By sunny weather

Reduce outside humidity into a random value among [10, 30], which is average humidity of south Korea. Applying gradient on temperature and increase its value +4. If temperature is increased, reduce current humidity among 0~6 random degrees. If current plant height is more than 50, apply inverse gradient on air humidity to reflect transpiration rate.

Interaction between internal parameters

Current humidity of ground is updated with gradient with current air humidity. Current plant height increases a little bit.

Correction

If any value exceeds 100, reduce it to 100. Also, any negative value is updated into 0.

Human Entrance

If human hasn’t appeared for 30 hours, a human enters the greenhouse with 1/3 probability. If a human has existed for more than 20 hours, he disappears with 4/5 probability.

Insect Attack

If pesticide has been applied in 600 hours, insect doesn’t appear. Otherwise, it appears with 1/10 probability.

Interact between the agent

Whenever an episode starts, the parameters are initialized. For each step, environment receives action from actor network and apply it. After that, a perturbation is applied to the environment too. Then the agent observes the environment.

Terminal condition

If any observable variable exceeds allowed range, the environment returns game over signal to agent and get ready for another episode. Also, the game may be finished by harvest action.