Mixed Reality Interface for Digital Twin of Plant Factory

I. Introduction

A. Digital Twin for Plant Factory

Previously, both smart farming solutions for IoT based automation [1-5] and precise environment analysis [6-11] were described as computer-based processing solutions for real-world only. However, applying digital twin models for agricultural purpose is now considered as a common inspiration among many smart farm engineers [16-20]. Digital twin methods provide precise analysis of current state and prediction of future state of industrial systems [12-15].

Hwang, et al. [16] and Alves, et al. [17] suggested a simulation model which can describe the state of real-world farm, including not only just common sensor-measured values but also the growth state of plants, and can receive control commands for actuators. As their approaches are applicable to open-field farms only, advanced solutions for indoor farm are required for most smart farm industry. Ko, et al. [18] and Monterio, et al. [19] has provided a specific service architecture for implementation of digital twin for indoor vertical farming systems, applied with existing IoT architecture. And Jeong, et al. [20] has shown that digital twin model for indoor plant factory can perform very precise real time analysis of environment state in high resolution.

Recent approaches provide useful blueprints for plant factory digital twin model. However, their 3D-rendered graphic-based model has many limitations. First, their methods always require pre-built 3D virtual farm rendering models for operation. It is only affordable for a small environment, and not appropriate for industrial-scaled plant factories. Secondly, once a 3D digital twin model is built, the users cannot move the facilities of real-word farms because it will lead to inconsistence between the twin. Also, a digital twin model should be easy to understand for old-aged users.

Therefore, I suggest a mixed reality environment with 3D scene reconstruction technologies to provide an immersive and intuitive digital twin environment.

B. 3D scene processing and reconstruction

3D scene processing method with pictures taken from a rotating camera or multi angle cameras has been very popular topic in computer vision processing area. Many researchers have suggested noble methods to overlap discrete images to build a continuous scene of 3D environment [21-22]. So various technologies or products for 360 ° scene are available now. Even some engineers now focus on how to reduce the camera cost [23] rather than image quality enhancement.

Thanks to NeRF [24], a single camera with plain angle of view can construct a 360 ° scene. NeRF model can generate the scenes between discrete pictures taken with single camera with various angle. There also are several applications of the original NeRF model such as depth-supervised model [25] and 3D scene generative model [26]. Even Nerfies model [27] can reconstruct deformable radiance fields to process a scene from pictures of moving object, or from pictures taken by moving camera. Therefore, a 360 ° scene can be generated with just one rotating camera, to provide an immersive VR-like environment.

II. Mixed Reality

A. Concept

While applying virtual elements such as buttons or indicators on real-world data is considered as augmented reality, augmented virtuality is its opposite concept: applying real-world element on virtual environment. However, in the proposed method, the real-word data is taken from remote farms and processed into 3D VR-like scene. Therefore, I think that the concept of proposed method should be categorized as mixed reality technology.

Fig. 2 shows the main concept of the interface environment. The main element of digital twin model is the virtual components such as control panel to monitor and control the remote smart farm environment

B. Interface

Users can access the interface system both via HMD or 2D display devices. The interface provides an immersive and intuitive scene synthesized with virtual interfaces for controlling and monitoring of farms, and real time streaming video taken from remote plant factory. User can interact with the environment with FPS game-like experiences. The interface continuously provides information of interactable object and options of additional actions for manipulation of active state of actuators.

The interface directly communicates with IoT servers to send and receive information and commands. Proposed method is fully compatible with previously installed IoT system.

III. Methods

A. System Architecture

Fig. 1 describes brief architecture of the system. Additional components for security options or data storage efficiency are recommended but not described on the illustration.

The sensors, actuators, IoT Node, and IoT server are not different from previously installed common-type IoT infrastructures. IoT server provides communication protocols such as rest APIs, which was previously provided with frontend applications. In this method, APIs of IoT servers are not directly applied on the frontend side. The interface server calls APIs of IoT server to draw data and provide commands from the users.

The interface server also receives real time visual data from the cameras which are located in the plant factory. It also processes the images with 3D scene synthesis algorithms in order to construct a VR-like 3D environment stream. Then the interface server mixes virtual UI with 3D streaming scene to produce a mixed reality environment.

B. User Interface Devices

Users can approach to the mixed reality environment with interface devices such as HMD or phones to monitor remote plant factory and send commands.

C. Command Stream

Users can make 3 kinds of command streams.

1) Commands for IoT Server

Commands for IoT server is just same as the user commands from the frontend interface of traditional IoT. However, the commands are not directly sent to IoT server. The user interface device first receives the commands, and it send them to the interface server. Finally, the interface server calls API of IoT servers to send user's commands.

Users can make not only commands for more information of environment state of remote plant factory, but also commands for manipulation of actuators' activation conditions, activation periods, or just activation state.

2) Commands for Interface Server

User can provide commands for the interface server too, via user interface devices. Users can request the interface server for interface pop-up, camera movements, object selection, and any other commands for virtual environment control.

3) Commands for User Interface Devices.

User also commands the interface devices for view scope angle change, zoom in and out, cursor indicator control, and any others for UI operation and experiences.

D. Scene Acquirement

Camera devices should acquire real time scene for the proposed method. Both stationary cameras and self-driving-like moving camera devices can be applied for mixed reality environment. I recommend to utilize 360° camera attached on a self-driving device, but a rotating single normal camera is compatible with this system if deformable neural radiance field [27] based image processing algorithms are available.

E. 3D Scene Synthesis

2D pictures taken from stationary camera data can be processed with traditional panoramic scene synthesis algorithms. However, if the camera device moves around the plant factory, I highly recommend applying NeRF-like applications which can remove artifacts driven by the movement of camera [27-29] or artifacts driven by LED devices [30-31] to build 3D scenes, Because of dense and irregular shapes of leaves and countless LED chips around the plant factory. I used Nerfies [27] for proposed method.

F. Interaction methods with Objects

As the proposed virtual environment displays the streaming scene of plant factory as a first person view, the interface structure for user-object interaction is very similar to FPS games. The user interface devices can provide 3 different types of interaction settings which are illustrated on Fig. 3.

Screen center method [32] are recommended for desktop computer users. The interface always selects an interactable object which is on the center of screen. So the user just can quickly change the angle of view with mouse, and simply move the view for object selection.

Additional cursor overlay method [33] is highly recommend for VR-ready devices such as HMD. User holds additional remote controller on the hand to control the indicator. While user can change the view by rotation of the head, the controller also displays a virtual indicator on the scene.

Touch screen based displays such as phones or tablet computers may provide best experience with view-cursor separation method [34]. User needs to use their fingers to change the view angle and zoom level. So complete separation between screen and cursor is recommended for comfortable UX. Users can just touch any interactable object displayed on the screen.

G. Interpolation of Sensor Values

There are limited numbers of sensors for environment monitoring. Therefore, the IoT server predicts missing values of environment states by interpolation among the sensor values. The IoT server stores (x, y, z) coordinates of sensors. And when the interface server requires the state value of (X, Y, Z), which is the location where the user interface device has received by users, the IoT server calculates the predicted values from installed sensor's locations and values.

IV. Result

A. View Synthesis

A set of 360° camera scene was taken with rotating single camera around the plant factory, taking pictures at every 45°. Then the pretrained Nerfies [27] model processed the scenes to generate new scenes. Fig. 4 displays the input images taken from medical hemp factory of Imagination Garden Inc. in Andong-si, Republic of Korea. And Fig. 5 was the synthesized scenes after postprocessing done.

B. Interaction with Objects

Fig. 6 illustrates object highlighting scenarios for user-object interaction. (a) shows normal state view. (b) shows an object activated by user's action such as click or indicator overlay. (c) describes an object with issues which users should check up.

1) Interaction Interface

Interaction UI examples are described on Fig. 7. When user activate the interactable object, summarized environment information and buttons for [Actions] or [Details] pop up. [Action] menu displays command options for actuators which the IoT servers can receive. [Details] menu shows detailed information of the object, and interpolated state variables of the location where the object is.

2) Issue Report

The environment shows issue state with color change. The severity of issue is expressed with color change.

V. Discussion

As 45° angle was too wide to build a radiance field, small amounts of pictures showed with satisfying quality. So, I think a 360° camera is still needed to provide a perfect scene, rather than processing scenes taken from a rotating camera. Also, further development for real time streaming and user-interface interaction are necessary for a commercial level application.

VI. Conclusion

User can experience 3D-like mixed reality environment with proposed system with available interface devices. The visual information from camera and sensory information from IoT nodes are processed in an intuitive way and provided to users in easy UI. Also user can make control commands of virtual objects, which lead to actuator state change. Therefore, the proposed method provides a digital twin for remote control and monitoring of a remote plant factory.

Acknowledgment

I would like to express my deep gratitude to Donghun Ryu for taking so many pictures of the plant factory.

References

1. Kibria, Md Golam, and Mohamad Tarmizi Abu Seman. "Internet of things based automated agriculture system for irrigating soil." Bulletin of Electrical Engineering and Informatics 11.3 (2022): 1752-1764.
2. Akateva, L. V., et al. "Development of an Automated Vertical Farm Module for Growing Plants Using Additive Technology." Theoretical Foundations of Chemical Engineering 56.4 (2022): 618-625.
3. Santiteerakul, Salinee, et al. "The role of smart technology in sustainable agriculture: A case study of wangree plant factory." Sustainability 12.11 (2020): 4640.
4. Ban, Byunghyun, and Soobin Kim. "Control of nonlinear, complex and black-boxed greenhouse system with reinforcement learning." 2017 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2017.
5. Ban, Byunghyun, et al. "Nutrient solution management system for smart farms and plant factory." 2020 international conference on information and communication technology convergence (ICTC). IEEE, 2020.
6. Cáceres, Gabriela, et al. "Smart farm irrigation: Model predictive control for economic optimal irrigation in agriculture." Agronomy11.9 (2021): 1810.
7. Muangprathub, Jirapond, et al. "IoT and agriculture data analysis for smart farm." Computers and electronics in agriculture 156 (2019): 467-474.
8. Ban, Byunghyun, Donghun Ryu, and Minwoo Lee. "Machine learning approach to remove ion interference effect in agricultural nutrient solutions." 2019 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2019.
9. Ban, Byunghyun, Minwoo Lee, and Donghun Ryu. "ODE network model for nonlinear and complex agricultural nutrient solution system." 2019 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2019.
10. Ban, Byunghyun. "Mathematical Model and Simulation for Nutrient-Plant Interaction Analysis." 2020 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2020.
11. Ban, Byunghyun. "Deep learning method to remove chemical, kinetic and electric artifacts on ISEs." 2020 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2020.
12. Jones, David, et al. "Characterising the Digital Twin: A systematic literature review." CIRP Journal of Manufacturing Science and Technology 29 (2020): 36-52.
13. Liu, Mengnan, et al. "Review of digital twin about concepts, technologies, and industrial applications." Journal of Manufacturing Systems 58 (2021): 346-361.
14. Tao, Fei, et al. "Digital twin in industry: State-of-the-art." IEEE Transactions on industrial informatics 15.4 (2018): 2405-2415.
15. Semeraro, Concetta, et al. "Digital twin paradigm: A systematic literature review." Computers in Industry 130 (2021): 103469.
16. Donghyun Hwang and Gyungseok Go. K.R. Patent No.10-2260011. Deajeon-si: Korean Intellectual Property Office, 2021.
17. Alves, Rafael Gomes, et al. "A digital twin for smart farming." 2019 IEEE Global Humanitarian Technology Conference (GHTC). IEEE, 2019.
18. Ko, Tae-Hwan, et al. "Design and Implementation of a Digital Twin Platform in Vertical Farming Systems." 2022 Thirteenth International Conference on Ubiquitous and Future Networks (ICUFN). IEEE, 2022.
19. Monteiro, José, et al. "Towards sustainable digital twins for vertical farming." 2018 Thirteenth International Conference on Digital Information Management (ICDIM). IEEE, 2018.
20. Jeong, Jin-Lip, et al. "Development and Validation of Digital Twin for Analysis of Plant Factory Airflow." Journal of the Korea Society for Simulation 31.1 (2022): 29-41.
21. Hussain, Ikram, Oh-Jin Kwon, and Seungcheol Choi. "Evaluating the coding performance of 360 image projection formats using objective quality metrics." Symmetry 13.1 (2021): 80.
22. Xu, Mai, et al. "State-of-the-art in 360 video/image processing: Perception, assessment and compression." IEEE Journal of Selected Topics in Signal Processing 14.1 (2020): 5-26.
23. Barazzetti, Luigi, Mattia Previtali, and Fabio Roncoroni. "Can we use low-cost 360 degree cameras to create accurate 3D models?." International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences 42.2 (2018).
24. Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM65.1 (2021): 99-106.
25. Deng, Kangle, et al. "Depth-supervised nerf: Fewer views and faster training for free." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
26. Kosiorek, Adam R., et al. "Nerf-vae: A geometry aware 3d scene generative model." International Conference on Machine Learning. PMLR, 2021.
27. Park, Keunhong, et al. "Nerfies: Deformable neural radiance fields." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
28. Klenk, Simon, et al. "E-NeRF: Neural Radiance Fields from a Moving Event Camera." arXiv preprint arXiv:2208.11300 (2022).
29. Pumarola Peris, Albert, et al. "D-NeRF: neural radiance fields for dynamic scenes." Proceding of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE), 2021.
30. Mildenhall, Ben, et al. "Nerf in the dark: High dynamic range view synthesis from noisy raw images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
31. Yang, Wenqi, et al. S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint." arXiv preprint arXiv:2210.08936 (2022).
32. Portal. Valve. 2007.
33. Oculus Home. Oculus. Accessed in 2019.
34. ArcheAge. XL Games. 2013.