Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

The numerical implementation of the controllers allows the use of very complex algorithms with ease. However, in practice, due to its proven advantages, the PID controller (and its variants) is widely used in industrial control systems as well as in many other applications that require continuous control. Most of the methods for tuning the parameters of PID controllers are based on time-invariant linear models of the processes, which in practice can lead to poor performance of the control system. The paper presents an application of reinforcement learning algorithms in tuning of PID controllers for the control of some classes of continuous nonlinear systems. Tuning the parameters of PID controllers is done with the help of Twin Delayed Deep Deterministic Policy Gradients (TD3) algorithm which presents a series of advantages compared to other similar methods from Machine Learning dedicated to continuous state and action spaces. TD3 algorithm is an off-policy Actor-Critic based method and was used as it does not require a system model. The presented technique is applied for control of a biotechnological system which has a strongly nonlinear dynamic. The proposed tuning method is compared to the classical tuning methods of PID controllers. The performance of the tuning method based on the TD3 algorithm is demonstrated through simulation illustrating the effectiveness of the proposed methodology.

The present grievous situation of the tuberculosis disease can be improved by efficient case management and timely follow-up evaluations. With the advent of digital technology this can be achieved by quick summarization of the patient-centric data. The aim of our study was to assess the effectiveness of the ODK Scan paper-to-digital system during testing period of three months. A sequential, explanatory mixed-method research approach was employed to elucidate technology use. Training, smartphones, application and 3G enabled SIMs were provided to the four field workers. At the beginning, baseline measures of the data management aspects were recorded and compared with endline measures to see the impact of ODK Scan. Additionally, at the end, users’ feedback was collected regarding app usability, user interface design and workflow changes. 122 patients’ records were retrieved from the server and analysed for quality. It was found that ODK Scan recognized 99.2% of multiple choice bubble responses and 79.4% of numerical digit responses correctly. However, the overall quality of the digital data was decreased in comparison to manually entered data. Using ODK Scan, a significant time reduction is observed in data aggregation and data transfer activities, however, data verification and form filling activities took more time. Interviews revealed that field workers saw value in using ODK Scan, however, they were more concerned about the time consuming aspects of the use of ODK Scan. Therefore, it is concluded that minimal disturbance in the existing workflow, continuous feedback and value additions are the important considerations for the implementing organization to ensure technology adoption and workflow improvements.

Classifying network flow subsequences in an Industrial IoT gateway is an effective method for identifying Industrial IoT faults. This paper proposes a subsequence classification algorithm SSGBUL-IKNN based on unsupervised learning encoding to address the problem of low classification accuracy. Firstly, we built a network module based on unsupervised learning to encode the flow sequence of the Industrial IoT gateway. In the encoding module, a subsequence calibration algorithm controls the increasing error during the coding process. Then, the specific fault type is obtained through an improved KNN classification algorithm based on the indication distance source from the integrated coding sequences converted by multi-dimensional network flow sequences. The test result on three Industrial IoT datasets of a sewage treatment plant shows that the accuracy of SSGBUL-IKNN in this paper exceeds 92%, significantly higher than the algorithm DTW and TSF. It reaches the classification requirements of Industrial IoT fault diagnosis.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

Heavy rains have created significant threats to human health and life. Floods and other natural disasters, which have a global impact annually, can be attributed to extended durations of intense precipitation. Accurate rainfall predictions are crucial in nations such as Bangladesh, where agriculture is the predominant occupation. The efficiency of machine learning (ML) methods is enhanced by the non-linearity of rainfall, surpassing the effectiveness of other alternative ways. Machine learning techniques show that individual classifiers exhibit worse accuracy than ensemble learning (EL) methodologies. Ensemble Learning techniques are utilized for rainfall prediction and estimating rainfall quantity and daily average temperature to enhance comprehension of the diverse Machine Learning algorithms. This research implements the machine learning techniques and ensemble-based classifier to predict the rainfall occurrence, along with the machine learning regressor models and ensemble-based regressor for the rainfall amount prediction and daily average temperature prediction, using Bangladesh Weather Dataset. The results of the machine learning and ensemble-based models are compared using the Accuracy and F1 score for rainfall occurrence prediction. In contrast, MAE and RMSE evaluation metrics are used for ensemble regressor and regression algorithms for the rainfall amount and daily average temperature prediction. With an accuracy of 83.41% and a recall of 78.17%, the Ensemble Classifier is the best at predicting when it will rain, but its precision of 51.16% stands in for the lowest. The Ensemble Regression model outperforms Linear Regression, Random Forest, and SVR in rainfall amount prediction, with the lowest MAE of 0.36 and RMSE of 0.90. The Ensemble Regression provides the most precise results for daily average temperature prediction with the lowest MAE 0.42 and RMSE 0.54 highlighting its superiority over the other regression models in forecasting temperature with less error. Ensemble approaches consistently lead task performance metrics.

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered remarkable success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Although, typically, the communication protocol between agents is manually specified and not altered during training, recently, some papers have shown signs of trying to emerge a communication between agents on the one hand and, on the other hand, to understand what is exchanged between agents. So, there is a growing body of literature on this topic which includes qualitative and quantitative studies and the ones that apply mixed methods. This study presents the scoping review of the methodological strategies undertaken in a total of 16 research articles. The results present the critical appraisal of quantitative methods in terms of validity and reliability and for qualitative methods considering four trustworthiness factors. In the end, relevant insights are further explored with implications and reflections on how they can benefit one's research in the field.

Background: This study aimed to identify the association between occupational stress and depression-well-being by proposing a comprehensive and flexible job burden-capital model with its corresponding hypotheses. Methods: For this research, 1618 valid samples were gathered from the electronic manufacturing service industry in Hunan Province, China; self-rated questionnaires were administered to participants for data collection after obtaining their written consent. The proposed model was fitted and tested through structural equation model analysis. Results: Single-factor correlation analysis results indicated that coefficients between all items and dimensions had statistical significance. The final model demonstrated satisfactory global goodness of fit (CMIN/DF=5.37, AGFI=0.915, NNFI=0.945, IFI=0.952, RMSEA=0.052). Both the measurement and structural models showed acceptable path loadings. Job burden and capital were directly associated with depression and well-being or indirectly related to them through personality. Multi-group structural equation model analyses indicated general applicability of the proposed model to basic features of such a population. Gender, marriage and education led to differences in the relation between occupational stress and health outcomes. Conclusions: The job burden-capital model of occupational stress-depression and well-being was found to be more systematic and comprehensive than previous models.

In this review we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.

Background. The Stanford Health Assessment Questionnaire-Disability Index (HAQ) is widely used to measure functional ability in persons with Rheumatoid Arthritis (RA). The instrument was developed with limited involvement from persons with RA, and their perception of the instrument has not been studied in depth. The aim of this study was to explore how persons with RA experience the use of the HAQ in care. Methods. Forty persons with RA were purposefully recruited to participate in semi-structured interviews. The interviews were then analyzed qualitatively using thematic analysis. Results. The participants questioned the relevance of the HAQ but nevertheless experienced that the instrument had a profound effect on their understanding of health and how care is delivered. The analysis resulted in three themes: Problems with individual items, meaning of the summative score, and effects on care and health perceptions. Conclusions. To make the HAQ relevant to persons with RA, it needs to be revised or to include an option to select items most meaningful to the respondent. To ensure relevance, the HAQ update should preferably be co-created by researchers, clinicians and persons with RA.

The modern supply chain (SC) is growing in terms of data, devices, users, and stakeholders, which introduced new security challenges and threats, especially with the reliance on centralized servers or cloud platforms. In addition, increased trust among system participants exposes the SC to a higher risk of vulnerabilities which require strong security measures. This article proposes a hybrid security framework for SC systems, BC-DRLzSC, that integrates Blockchain (BC) and Deep Reinforcement Learning (DRL) designed to operate in a zero trust (ZT) environment. In particular, we propose a decentralized BC-based approach integrated with smart contracts to manage system participant registration and authentication and to control access to system resources. BC-DRLzSC adopts a ZT architecture to reinforce SC security, which can be achieved with an advocate to verify each entity’s trustworthiness before granting or retaining access to system resources. Incorporating the ZT architecture, with BC and DRL, can potentially and significantly bolster SC system security. DRL is employed to develop a proactive attack detection model that continuously monitors the incoming traffic from authenticated nodes within the network and predicts any malicious actions. Finally, we evaluate the performance of our proposed DRL solution using the NSL-KDD dataset.

In March 2013, a novel avian influenza A H7N9 virus was emerged in China, which cause rapidly progressive pneumonia and with a high fatality rate. Serologic studies to evaluate neutralizing antibodies of infected patients and birds are invaluable tools for immunogenicity research of H7N9 and epidemiological investigation. Conventional neutralization assays are laborious and time-consuming which also hampered by biosafety requirement. In this study, We construct and produce pseudovirus bearing the full-length hemagglutinin (HA) of H7N9 virus in the Env-defective, luciferase-expressing HIV-1 backbone. The production of lentiviral pseudovirus was analysed by HA gene specific real-time reverse-transcription PCR, transmission electron microscopy (TEM), and Western Blot assay to prove the nucleic acid replication, the morphology of virus, and the expression of HA protein in pseudovirus. After that pseudovirus based inhibition assay was established to detect neutralizing antibodies of a panel of serum samples. Our results demonstrated that H7N9 pseudovirus which had single-cycle infection was generated. By comparing the neutralization antibody titers, pseudovirus based neutralization test could be recognized as an alternative of conventional microneutralization (MN). Hence, we conclude that it is possible to use pseudovirus inhibition assay to screen sera samples, as well as evaluate vaccine-induced neutralizing antibodies against H7N9 virus.

The robust control of high precision electromechanical systems, such as micropositioners, is challenging in terms of the inherent high nonlinearity, the sensitivity to external interference, and the complexity of accurate identification of the model parameters. To cope with these problems, this work investigates a disturbance observer-based deep reinforcement learning control strategy to realize high robustness and precise tracking performance. Reinforcement learning has shown great potential as optimal control scheme, however, its application in micropositioning systems is still rare. Therefore, embedded with the integral differential compensator (ID), deep deterministic policy gradient (DDPG) is utilized in this work with the ability to not only decrease the state error but also improves the transient response speed. In addition, an adaptive sliding mode disturbance observer (ASMDO) is proposed to further eliminate the collective effect caused by the lumped disturbances. The sterling performance is revealed with intensive tracking simulation experiments and demonstrates the improvement in the accuracy and response time of the controller.

This paper is primarily focused on the robust control of an inverted pendulum system based on the policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

This paper presents a system for identification of wind features, such as gusts and wind shear. These are of particular interest in the context of energy-efficient navigation of Small Unmanned Aerial Systems (UAS). The proposed system generates real-time wind vector estimates and a novel algorithm to generate wind field predictions. Estimations are based on the integration of an off-the-shelf navigation system and airspeed readings in a so-called direct approach. Wind predictions use atmospheric models to characterize the wind field with different statistical analyses. During the prediction stage, the system is able to incorporate, in a big-data approach, wind measurements from previous flights in order to enhance the approximations. Wind estimates are classified and fitted into a Weibull probability density function. A Genetic Algorithm (GA) is utilized to determine the shaping and scale parameters of the distribution, which are employed to determine the most probable wind speed at a certain position. The system uses this information to characterize a wind shear or a discrete gust and also utilizes a Gaussian Process regression to characterize continuous gusts. The knowledge of the wind features is crucial for computing energy-efficient trajectories with low cost and payload. Therefore, the system provides a solution that does not require any additional sensors. The system architecture presents a modular decentralized approach, in which the main parts of the system are separated in modules and the exchange of information is managed by a communication handler to enhance upgradeability and maintainability. Validation is done providing preliminary results of both simulations and Software-In-The-Loop testing. Telemetry data collected from real flights, performed in the Seville Metropolitan Area in Andalusia (Spain), was used for testing. Results show that wind estimation and predictions can be calculated at 1 Hz and a wind map can be updated at 0.4 Hz. Predictions show a convergence time with a 95% confidence interval of approximately 30 s.

This paper presents an optimized energy management strategy for Li-ion power batteries used on electric vehicles (EVs) at low temperatures. Under low-temperature environments, EVs suffer a sharp driving range loss resulted from the energy and power capability reduction of the battery. Simultaneously, because of Li plating, battery degradation becomes an increasing concern as temperature drops. All these factors could greatly increase the total vehicle operation cost. Prior to battery charging and vehicle operating, preheating battery to a battery-friendly temperature is an approach to promote energy utilization and reduce total cost. Based on the proposed LiFePO4 battery model, the total vehicle operation cost under certain driving cycles is quantified in the present paper. Then given a certain ambient temperature, a target temperature of preheating is optimized under the principle of minimizing total cost. As for the preheating method, a liquid heating system is also implemented on an electric bus. Simulation results show that the preheating process becomes increasingly necessary with a decreasing ambient temperature; however, the preheating demand declines as driving range grows. Vehicle tests verify that the preheating management strategy proposed in this paper is able to save total vehicle operation cost.

The electroencephalogram (EEG) is a record of brain activity. Brain Computer Interface (BCI) technology formed by the EEG signal has become one of the hotspots at present. How to extract the feature signal of EEG is the most basic research of BCI technology. In this paper, A new method of recognizing fatigue, conscious, concentrated state of human brain is proposed by the combination of discrete wavelet transform and the neural network based on EEG signal. First of all, the law signal is preprocessed by the wavelet denoising method because the law EEG signal contains a large number of high frequency noise, which is decomposed into multi-layer high frequency signal and low frequency signal. thus, δ wave, θ wave, α wave, β wave are obtained by the wavelet transform. And then, frequency band energy of the different wave is regards as the feature signal of EEG. In the experiment, the feature signal is classified by radial basic function (RBF) and annealed chaotic competitive learning network (ACCLN). RBF and ACCLN networks are trained with 500 sets of sample data and are tested by 100 sets of samples in different mental states. The experimental results show that the average accuracy of RBF network under three conditions are 88.75%, 88.25%, 88.5%, respectively, and the correct rate of ACCLN network is 97%, 98%, 98%, respectively.

The paper presents an error model of a measurement chain containing link performing discrete wavelet transform algorithm which is most often the last stage of measurement signal processing. The goal is to determine the uncertainty budget of the input quantities of the wavelet transform algorithm. The error model takes into account parts of analog, analog-to-digital and digital processing, describing the properties of subsequent fragments of the chain using their transmittance and processing functions. The proposed model enables the description of both deterministic and non-deterministic properties of signal errors. The proposed model was validated using an example measurement chain created for this purpose.

Meningitis is a severe disease associated with death in children under five with highest rates of infections under age of one. Vaccines for Neisseria meningitides and Haemophilus influenza are used to prevent the main causative agents of meningitis. Administration of H. influenzae type b (Hib) vaccine is recommended at 2, 4 and 6 months with a booster dose at 18 months. N. meningitidis has two commercially available vaccines, the pure polysaccharide is recommended at 24 months meanwhile the protein-conjugated vaccines at 12 months. We sought in this study to examine if coadministering the vaccines for the two main meningitis causing bacteria might be synergistic as a preliminary step towards the possibility of shuffling immunization schedule. So, we coadministered Hib vaccine with commercially available vaccines either quadrate (ACWY) polysaccharide meningococcal (Men) or conjugated meningococcal (Nim) vaccines in Balb/C mice (n = 6/group) and compared to each vaccine administered separately and controls. Thirty-five days post immunization, we measured specific antibodies titers. Hib vaccine increased Men antibody titers significantly for serotypes Y and W. When Hib vaccine was coadministrated with Nim, antibody titer for Y, W and A significantly increased. For serotype C, there was no significant difference in antibody titers among immunized groups. As for effect of meningococcal vaccines on Hib, Men significantly increased Hib antibody titers while Nim had no effect. Collectively, our data suggested that coadministration of Hib and Men or Nim vaccines was safe and had synergistic effect on immune responses elicited to both vaccines. Further studies are needed before immunization schedule modifications. Such immunization schedule recommendation should provide better protection against this life-threatening disease in young children.

The application of space robotic manipulators and heightened autonomy for ios represents a paramount pursuit for leading space agencies, given the substantial threat posed by space debris to operational satellites and forthcoming space endeavors. This work presents a guidance algorithm based on drl to solve for the space manipulator path-planning during the motion synchronization phase with the mission target. The goal is the trajectory generation and control of a spacecraft equipped with a 7-dof robotic manipulator, such that its end effector remains stationary with respect to the target point of capture. The ppo drl algorithm is used to optimize the manipulator’s guidance law, and the autonomous agent generates the desired joints rates of the robotic arm, which are then integrated and passed to a model-based feedback linearization controller. The agent is first trained to optimize its guidance policy and then tested extensively to validate the results against a simulated environment representing the motion synchronization scenario of an ios mission.

The attitude dynamics and orbit dynamics of large rigid space structures are coupled with each other under gravity gradient, which may affect the stability of the large rigid space structures. In this paper, the gravity gradient stability of a large rigid space structure is studied under both small and large disturbances. Based on the rigid body dynamics and orbit dynamics, an accurate dynamic model without any linearization of the large rigid space structure is established via the natural coordinate formulation (NCF), which is able to describe the large overall motions of the structures. By using the generalized-α algorithm, the gravity gradient stability of the large rigid space structure is simulated and analyzed via various examples, including the influence of large disturbance angles, the positions at the stabilization and unstabilization regions. Finally, the relationship between spinning stability and gravity gradient stability is also investigated via a large spinning space structure.

The increasing availability of longitudinal data (repeated numerical observations of same units at different times) requires the development of flexible techniques to automatically detect common errors in such data. Besides obvious and easily identifiable cases, such as missing or out-of-range data, large longitudinal dataset often present problems not easily traceable by the techniques used for generic datasets. In particular, elusive and baffling problems are i) inversion of one or more values from one unit to another; ii) anomalous jumps in the series of values, iii) errors in the timing of the values due to a recalculation operated by the data providers to compensate previous errors. This work proposes a statistical-mathematical approach based on a system of indicators that is able to capture the complexity of the described problems by working at the formal level, regardless of the specific meaning of the data. The proposed approach identifies suspect erroneous data and is applicable in a variety of contexts. We implement this approach in a relevant database of European Higher Education institutions (ETER) by analyzing Total academic staff, that is one of the most important variables, used in empirical analysis as proxy of size and also considered by policy makers at European level.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

The working environment of mining electric locomotives is wet and muddy coal mine 1 roadway. There may be idling or slipping between the wheels and rails of mining electric locomotives 2 due to low friction between the wheel and rail and insufficient utilization of creep rate. Therefore, it 3 is necessary to control the creep rate within a reasonable range. In this paper, the autonomous control 4 algorithm for mining electric locomotives based on improved ε-greedy is theoretically proven to be 5 convergent and effective firstly. Secondly, after analyzing the contact state between the wheel and rail 6 under wet and slippery road conditions, it was concluded that the value of creep rate is an important 7 factor affecting the autonomous driving of mining electric locomotives. Therefore, the autonomous 8 control method for mining electric locomotives based on creep control is proposed in this paper. 9 Finally, the effectiveness of the proposed method was verified through simulation. The problem of 10 wheel slipping and idling caused by insufficient friction of mining electric locomotives in coal mining 11 environments is effectively suppressed. Autonomous operation of vehicles with optimal driving 12 efficiency can be achieved through quantitative control and utilization of the creep rate between 13 wheels and rails

This paper proposes a novel single-stage buck-boost three-Level neutral-point-clamped (NPC) inverter with two independent dc sources coupled for the grid-tied photovoltaic (PV) application, which can effectively solve the unbalanced operational conditions generally appeared between two coupled independent PV sources induced by the unequal irradiation and temperature distribution. The proposed control scheme can simultaneously guarantee the maximum power point (MPP) operation of both PV sources and maintain the output waveform quality. Compared to the traditional two-stage PV inverter, the proposed NPC inverter could reduce the PV array voltage requirement and dc-link capacitors’ voltage rating, meanwhile show the advantage in operational efficiency. MATLAB simulations and the captured experimental results are presented to show the performance of the proposed three-level inverter.

In the past two decades, metaheuristic optimization algorithms (MOAs) have been increasingly popular, particularly in logistic, science, and engineering problems. The fundamental characteristics of such algorithms are that they are dependent on a parameter or a strategy. Some online and offline strategies are employed in order to obtain optimal configurations of the algorithms. Adaptive operator selection is one of them, and it determines whether or not to update a strategy from the strategy pool during the search process. In the filed of machine learning, Reinforcement Learning (RL) refers to goal-oriented algorithms, which learn from the environment how to achieve a goal. On MOAs, reinforcement learning has been utilised to control the operator selection process. Existing research, however, fails to show that learned information may be transferred from one problem-solving procedure to another. The primary goal of the proposed research is to determine the impact of transfer learning on RL and MOAs. As a test problem, a set union knapsack problem with 30 separate benchmark problem instances is used. The results are statistically compared in depth. The learning process, according to the findings, improved the convergence speed while significantly reducing the CPU time.

A detailed literature review is performed in this study to address solutions for the full-body design and control of an aerial manipulator. Deep Reinforcement Learning methods are growing to be utilized recently to cope with various uncertainties. The pros and cons of these theories will be explained as well as introducing the advantages of Fuzzy Reinforcement Learning methods. State-of-the-Art, possible challenges, potential approaches, and a summary of desired precision devices are discussed in this study.

Coal gasification with carbon dioxide is a process for generating clean gaseous fuels and relieving greenhouse effect. Zhundong coal has high alkali and alkali earth metals (AAEMs) content, medium volatile and low ash in nature. Isothermal CO₂ gasification of char derived from Zhundong coal (R-char) and char from acid washing R-char (AR-char) are performed in thermo-gravimetric analyzer (TGA). The effect of AAEMs is investigated on the gasification behavior in the range of temperatures 1073 K to 1273 K. The carbon conversion increases rapidly with increasing reaction temperature and CO₂ concentration. R-char has high gasification rate and carbon conversion compared with AR-char. The accuracy of the free-model approach for calculating activation energy at different conversions is validated by compared with different kinetic models (volume reaction model, distributed activation energy model). Moreover, R-char gasification with CO₂ shows a compensation effect as the Arrhenius parameters (EA and k0) increase or decrease simultaneously.

Reinforcement learning is used for air combat problems in recent years, and the idea of curriculum learning is often used for reinforcement learning, but traditional curriculum learning suffers from the problem of plasticity loss in neural networks. Plasticity loss is the difficulty of learning new knowledge after the network has converged. To this end, we propose a motivational curriculum learning distributed proximal policy optimization (MCLDPPO) algorithm, through which agents trained can significantly outperform the predictive game tree and mainstream reinforcement learning methods. The motivational curriculum learning is to help the agent gradually improve their combat ability by observing the agent's unsatisfactory performance and providing corresponding rewards as a guide. Additionally, a complete tactical maneuver is encapsulated based on existing air combat knowledge, and through the flexible use of these maneuvers, some tactics beyond human knowledge can be realized. In addition, we designed an interruption mechanism for the agent to increase the frequency of decision-making when the agent faces an emergency. When the number of threats the agent receives changes, the current action will be interrupted to reacquire observations and make decisions again. Using the interruption mechanism can significantly improve the ability of the agent. To simulate actual air combat better, we built an air combat environment based on digital twin technology to create an air combat engine core such as missiles and radars. Based on this environment, we develop a distributed intelligent training system that employs Redis as the cache and obtains the data generated in the distributed environment through a multi-threaded TCP/IP protocol, which significantly increases the training data throughput and helps the agent accelerates convergence. The experimental results demonstrate that the agent can fully exploit the situational information to make reasonable decisions and affords air combat tactical adaption, verifying the effectiveness of the algorithmic framework proposed in this paper.

Modeling is essential to understand better the generative mechanisms responsible for experimental observations gathered from complex systems. In this work, we are using such an approach to analyze the electrocardiogram (ECG). We present a systematic framework to decompose ECG signals into sums of overlapping lognormal components. We used reinforcement learning to train a deep neural network to estimate the modeling parameters from ECG recorded in babies of 1 to 24 months of age. We demonstrate this model-driven approach by showing how the extracted parameters vary with age. After correction for multiple tests, 10 of 24 modeling parameters showed statistical significance below the 0.01 threshold, with absolute Kendall rank correlation coefficients in the [0.27, 0.51] range. We presented a model-driven approach to the analysis of ECG. The impact of this framework on fundamental science and clinical applications is likely to be increased by further refining the modeling of the physiological mechanisms generating the ECG. By improving the physiological interpretability, this approach can provide a window into latent variables important for understanding the heart-beating process and its control by the autonomous nervous system.

The collision avoidance mechanism adopted by the IEEE 802.11 standard is not optimal. The mechanism employs a binary exponential backoff (BEB) algorithm in the medium access control (MAC) layer. Such an algorithm increases the backoff interval whenever a collision is detected to minimize the probability of subsequent collisions. However, the expansion of the backoff interval causes degradation of the radio spectrum utilization (i.e., bandwidth wastage). That problem worsens when the network has to manage the channel access to a dense number of stations, leading to a dramatic decrease in network performance. Furthermore, a wrong backoff setting increases the probability of collisions such that the stations experience numerous collisions before achieving the optimal backoff value. Therefore, to mitigate bandwidth wastage and, consequently, maximize the network performance, this work proposes using reinforcement learning (RL) algorithms, namely Deep Q Learning (DQN) and Deep Deterministic Policy Gradient (DDPG), to tackle such an optimization problem. As for the simulations, the NS-3 network simulator is used along with a toolkit known as NS3-gym, which integrates a reinforcement-learning (RL) framework into NS-3. The results demonstrate that DQN and DDPG have much better performance than BEB for static and dynamic scenarios, regardless of the number of stations. Moreover, the performance difference is amplified as the number of stations increases, with DQN and DDPG showing a 27% increase in throughput with 50 stations compared to BEB. Furthermore, DQN and DDPG presented similar performances.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

Deep reinforcement learning (DRL) has achieved significant results in many Machine Learning (ML) benchmarks. In this short survey we provide an overview of DRL applied to trading on financial markets, including a short meta-analysis using Google Scholar, with an emphasis on using hierarchy for dividing the problem space as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market.

Continuous greenhouse gas emissions are causing global warming and impacting the habitats of many animals. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable is challenging. To address the issue, we propose a grid dispatching technique that combines prediction technology, reinforcement learning, and optimization technology. Prediction technology can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the power system, the aggregation of distributed energy resources increases uncertainty, particularly due to the fluctuating generation of renewable energy. This requires the use of advanced predictive control techniques to ensure long-term economic and decarbonization goals. In this paper, we present a real-time dispatching framework that integrates deep learning-based prediction, reinforcement learning-based decision-making, and stochastic optimization techniques. The framework can rapidly adapt to target uncertainty caused by various factors in real-time data distribution and control processes. The proposed framework achieved global Champion in the NeurIPS Challenge 2022 competition and demonstrated its effectiveness in practical scenarios of intelligent building energy management.

As the adoption of renewable energy sources grows, ensuring a stable power balance across various time frames has become a central challenge for modern power systems. In line with the "dual carbon" objectives and the seamless integration of renewable energy sources, harnessing the advantages of various energy storage resources and coordinating the operation of long-term and short-term storage have become pivotal directions for future energy storage deployment. To address the complexities arising from the coupling of different time scales in optimizing energy storage capacity, this paper proposes a method for energy storage planning that accounts for power imbalance risks across multiple time scales. Initially, the Seasonal and Trend decomposition using the Loess (STL) decomposition method is utilized to temporally decouple actual operational data. Subsequently, power balance computations are performed based on the obtained data at various time scales to optimize the allocation of different types of energy storage capacities and assess the associated imbalance risks. Finally, the effectiveness of the proposed approach is validated through hourly applications using real-world data from a province in China over recent years.

Water quality assessment in pastoral of Tibetan Plateau, which is water sources for about 40% of world's population and the runoff-yield area of Yellow rivers, is very important. In this paper, Xiahe county which belongs to Tibetan Plateau pastoral was investigated. Six parameters(via, chloride, COD, ammonia nitrogen, nitrate, fluoride, sulfate) were selected to assess the water quality and health degree by using fuzzy comprehensive evaluation methods. The hydrochemical type in surface water was of HCO3--Mg2+-Ca2+ type. The cations and anions in surface water were mainly from weathering and dissolution of carbonate rock. Results showed that the water quality in all 69 sampling sites was all of class Ⅰ. The integrated health degree reached more than 0.85 and the health rate were 100%. Although ammonia nitrogen was regarded as the main contamination factor, but it had a little effect on the entire body of water. Overall, the surface water qualities of most samples in Xiahe County was good condition.

Sea cucumber peptides are small molecular peptides with biological function obtained from fresh sea cucumber as feedstock, which were hydrolyzed by protease, separated ,and purified. As the main product of sea cucumber refinement, it more accurately expresses the nutritional effect of sea cucumber and has a wide range of potential applications. It has been demonstrated that sea cucumber has a certain value-added effect on bone mineral density regulation, but the differences in bone mineral density regulation of extracts from different parts of sea cucumber and the techniques contributing to the efficient isolation of bone sea cucumber peptides have not been reported. Firstly, the aim of this study was to reveal the differences in osteogenic activities of enzymatic peptides from three major parts of sea cucumber: the body wall, the egg protein, and the intestinal protein. Then, the bone-enhancing sea cucumber peptides were isolated based on the characteristics of the dif-ferences in molecular weight distribution, hydrophilicity, and ionisation degree of sea cucumber, and to explore the possible mechanisms of their action. The findings indicated that sea cucumber peptides extracted from three distinct sites exhibited a dose-dependent enhancement of osteoblast proliferation and differentiation. Upon evaluating the three methods of sea cucumber peptide separation based on molecular weight distribution, hydrophilicity, and ionization, it was observed that the fraction isolated by hydrophobicity and

The prevalence of differential equations as a mathematical technique has refined the fields of control theory and constrained optimization due to the newfound ability to accurately model chaotic, unbalanced systems. However, in recent research, systems are increasingly more nonlinear and difficult to model using Differential Equations only. Thus, a newer technique is to use policy iteration and Reinforcement Learning, techniques that center around an action and reward sequence for a controller. Reinforcement Learning (RL) can be applied to control theory problems since a system can robustly apply RL in a dynamic environment such as the cartpole system (an inverted pendulum). This solution successfully avoids use of PID or other dynamics optimization systems, in favor of a more robust, reward-based control mechanism. This paper applies RL and Q-Learning to the classic cartpole problem, while also discussing the mathematical background and differential equations which are used to model the aforementioned system.

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

In the context of reinforcement learning (RL), ensuring both safety and performance is crucial, especially in real-world scenarios where mistakes can lead to severe consequences. This study aims to address this challenge by integrating temporal logic constraints into RL algorithms, thereby providing a formal mechanism for safety verification. We employ a combination of theoretical and empirical methods, including the use of temporal logic for formal verification and extensive simulations to validate our approach. Our results demonstrate that the proposed method not only maintains high levels of safety but also achieves comparable performance to traditional RL algorithms. Importantly, our approach fills a critical gap in existing literature by offering a solution that is both mathematically rigorous and empirically validated. The study concludes that the integration of temporal logic into RL offers a promising avenue for developing algorithms that are both safe and efficient. This work lays the foundation for future research aimed at generalizing this approach to various complex systems and applications.

In August 2020, the World Health Assembly launched a global initiative to eliminate cervical cancer by 2030, setting three primary targets. One key goal is to achieve a 70% screening coverage rate for cervical cancer, primarily relying on the precise analysis of Papanicolaou (Pap) or digital Pap smears. However, the responsibility of reviewing Pap smear samples to identify potentially cancerous cells primarily falls on pathologists, a task known to be exceptionally challenging and time-consuming. This paper proposes a solution to address the shortage of pathologists for cervical cancer screening. It leverages the OpenAI-GYM API to create a deep reinforcement learning environment utilizing liquid-based Pap smear images. By employing the Proximal Policy Optimization algorithm, autonomous agents navigate Pap smear images, identifying cells with the aid of rewards, penalties, and accumulated experiences. Furthermore, the use of a pre-trained convolutional neuronal network like Res-Net54 enhances the classification of detected cells based on their potential for malignancy. The ultimate goal of this study is to develop a highly efficient, automated Papanicolaou analysis system, ultimately reducing the need for human intervention in regions with limited pathologists.

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in video transmission across networks. Traditional adaptive bitrate (ABR) algorithms adjust the quality of video segments based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules within a complex environment, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel Deep Reinforcement Learning-based approach for streaming DASH, focusing on maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes the Quality of Experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate this approach through a simulation model that encompasses diverse wireless network environments and various video sequences. Additionally, we compare our proposed approach with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

For a mobile robot, navigation in a densely crowded space can be a challenging and sometimes impossible task, especially with traditional techniques. In this paper, we present a framework to train neural controllers for differential drive mobile robots which must safely navigate a crowded environment while trying to reach a target location. To learn the robot’s policy, we train a convolutional neural network using two reinforcement learning algorithms, Deep Q-Networks (DQN) and Asynchronous Advantage Actor Critic (A3C), and develop a training pipeline that allows to scale the process to several compute nodes. We show that the asynchronous training procedure in A3C can be leveraged to quickly train neural controllers and test them on a real robot in a crowded environment.

Ever-increasing need for improving the livability of a city and improve outcomes for its residents, over the last decade, the adoption of technology to develop urbanised societies around the world has given rise to the need for developing smart cities. The speed at which the world population is growing, the use of Internet of Things in smart cities have really advanced the quality of life. One significant area of concern within the smart city framework is waste management. If the waste within a city is not adequately managed, then it leads to issues in the health of the citizens. Additionally, the waste management has such a high impact on the environmental footprint, hence the need to have a smart way of managing waste is of critical importance. Through our research, we analyse the challenges of waste management within a city to understand the impact of the problem on to the citizens and overall city operations. We then investigate ways in which we can solve these problems using the emerging technologies, such as the Internet of Things, to collect valuable data of large volumes arriving at an astronomical rate, then apply multi-agent deep reinforcement learning algorithms to harness the power of big data to extract meaningful information and actionable insights. We ingest data generated by our Internet of Things into our algorithm for three main purposes including providing the notifications to an external system, for example, a map navigation engine out of the scope for this project but a future extension for route optimisation and waste vehicle tracking; extracting and reporting the actionable insights from the underlying data; and consuming the extracted data for predictive forecasting to draw out the unknown patterns of waste fill levels within various geographical locations and again send out triggers and notification to external systems for example a waste collection authority who can efficiently schedule the waste collection vehicles and optimise the route. To achieve the above mentioned outcomes, we propose a framework that is agnostic of the hardware that it connects to and can effectively interface with a wide variety of hardware keeping a level of abstraction in the architecture.

Sparse reward long horizon task is a major challenge for deep reinforcement learning algorithm. One of the key barriers is data-inefficiency. Even in the simulation environment, it usually takes weeks to training the agent. In this study, a data-efficiency training framework is proposed, where a curriculum learning is design for the agent in the simulation scenario. Different distributions of the initial state are set for the agent to get more informative reward during the whole training process. A fine-tuning of the parameters in the output layer of the neural network for value function is conduct to bridge the gap between sim-to-real. An experiment of UAV maneuver control is conducted in the proposed training framework to verify the method more efficient. We demonstrate that data-efficiency is different for the same data in different training stages.

Abstract: As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs at a larger scale. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

Recently, the application of deep reinforcement learning in recommender system is flourishing and stands out by overcoming drawbacks of traditional methods and achieving high recommendation quality. The dynamics, long-term returns and sparse data issues in recommender system have been effectively solved. But the application of deep reinforcement learning brings problems of interpretability, overfitting, complex reward function design, and user cold start. This paper proposed a tag-aware recommender system based on deep reinforcement learning without complex function design, taking advantage of tags to make up for the interpretability problems existing in recommender system. Our experiment is carried out on MovieLens dataset. The result shows that, DRL based recommender system is superior than traditional algorithms in minimum error and the application of tags has little effect on accuracy when making up for interpretability. In addition, DRL based recommender system has excellent performance on user cold start problems.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL) and Health Systems Simulations has significant potential, for both research into improving Deep RL performance and safety, and in operational practice. While individual toolkits exist for Deep RL and Health Systems Simulations, no framework to integrate the two has been established. Aim: Provide a framework for integrating Deep RL Networks with Health System Simulations, and to ensure this framework is compatible with Deep RL agents that have been developed and tested using OpenAI Gym. Methods: We developed our framework based on the OpenAI Gym framework, and demonstrate its use on a simple hospital bed capacity model. We built the Deep RL agents using PyTorch, and the Hospital Simulation using SimPy. Results: We demonstrate example models using a Double Deep Q Network or a Duelling Double Deep Q Network as the Deep RL agent. Conclusion: SimPy may be used to create Health System Simulations that are compatible with agents developed and tested on OpenAI Gym environments. GitHub repository of code: https://github.com/MichaelAllen1966/learninghospital

Three kinds of inorganic alkali are introduced into tetraethylenepentamine (TPEA) and polyethyleneimine (PEI) modified MCM-41 as the CO2 adsorbents. FT-IR and TGA are used to characterize the surface groups and the thermal stability of adsorbents. Chemical titration method is used to measure the alkali amounts of adsorbents. Thermo-gravimetric analysis with 10%CO2/N2 as the simulated flue gas is used to test the CO2 adsorption performance of adsorbents. The results show that all three kinds of inorganic alkali containing adsorbents exhibit promoted CO2 adsorption capability than traditional TPEA and PEI modified samples. Ca(OH)2 and PEI modified samples exhibit the highest adsorption capacity and stable regeneration property. The introduction of inorganic alkali changes the chemical adsorption mechanism between CO2 and increases the effective alkali amounts due to its hydroxyl groups. The CO2 adsorption capacities have a linear dependence relation with the alkali amounts of adsorbents, indicating that alkali amount is a critical factor for novel adsorbents exploring.

In response to the issue of vehicle escape guidance, this manuscript proposes a unified intelligent control strategy synthesizing optimal guidance and Meta Deep Reinforcement Learning (DRL). Optimal control with minor energy consumption is introduced to meet terminal latitude, longitude and altitude. Maneuvering escape is realized by adding longitudinal and lateral direction maneuver overloads. Maneuver command decision model is calculated based on Soft-Actor-Critic (SAC) networks. Meta learning is introduced to enhance autonomous escape capability, which improves generalization performance to time-varying scenarios not encountered in the training process. In order to obtain training samples at a faster speed, this manuscript uses the prediction method to solve reward values, which avoiding a large number of numerical integration. The simulation results manifest that the proposed intelligent strategy can achieve high precise guidance and effective escape.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

Abstract—Online learning platforms provide diverse course resources, but this often result in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, Existing course recommendation methods face the challenge in accurately recommending the target course that is most relevant to the user, because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the Actor-Critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendation effectively. Furthermore, we propose a policy gradient based on temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and improves the accuracy of the recommendation. We evaluate the proposed model on two real datasets, and the experimental results show that the proposed model is significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.

Fast convergence routing is an important issue for LEO constellation network, due to its dynamical topology changing and time varying transmission requests. Most of existing research focus on the OSPF routing algorithm, which cannot handle the frequently links state changing of network. In this paper, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL-SR) for LEO satellite networks, in which the satellite gets the network links status fast and adjusts its routing strategy. In FRL-SR, each satellite node is regarded as an agent. The agent selects the port for packet forwarding according to its own routing policy. When the satellite network state changes, agent would send ’hello’ packets to the neighbor node to update the neighbor node’s routing policy. Compared with traditional reinforcement learning, FRL-SR can perceive network information faster, and then converge faster. Also, FRL-SR can mask the dynamics of satellite network topology and adaptively adjust the forwarding strategy according to the link state. Various simulation is constructed, the results show that the proposed FRL-SR algorithm out performance the Dijkstra algorithm in performance of average delay, packet arriving ratio, network load balance.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

We consider the problem of coordinating the charging of an entire fleet of electric vehicles (EV), using a model-free approach, i.e. purely data-driven reinforcement learning (RL). The objective of the RL-based control is to optimize charging actions, while fulfilling all EV charging constraints (e.g. timely completion of the charging). In particular, we focus on batch-mode learning and adopt fitted Q-iteration (FQI). A core component in FQI is approximating the Q-function using a regression technique, from which the policy is derived. Recently, a dueling neural networks architecture was proposed and shown to lead to better policy evaluation in the presence of many similar-valued actions, as applied in a computer game context. The main research contributions of the current paper are that (i)we develop a dueling neural networks approach for the setting of joint coordination of an entire EV fleet, and (ii)we evaluate its performance and compare it to an all-knowing benchmark and an FQI approach using EXTRA trees regression technique, a popular approach currently discussed in EV related works. We present a case study where RL agents are trained with an epsilon-greedy approach for different objectives, (a)cost minimization, and (b)maximization of self-consumption of local renewable energy sources. Our results indicate that RL agents achieve significant cost reductions (70--80%) compared to a business-as-usual scenario without smart charging. Comparing the dueling neural networks regression to EXTRA trees indicates that for our case study's EV fleet parameters and training scenario, the EXTRA trees-based agents achieve higher performance in terms of both lower costs (or higher self-consumption) and stronger robustness, i.e. less variation among trained agents. This suggests that adopting dueling neural networks in this EV setting is not particularly beneficial as opposed to the Atari game context from where this idea originated.

The essence of cross-modal generation and retrieval modeling for image-text interactions lies in semantic consistency. Among these, cross-modal representation alignment serves as the foundational requirement for achieving cross-modal semantic consistency. The crux of our proposed method entails the collaborative training of an image captioning model(referred to as the ’captioner’) and a pair of contrastive encoders for image-text matching(referred to as the ’concoder’). This synergistic approach yields concurrent enhancements in concoder’s performance, captioner’s proficiency, and cross-modal semantic coherence. We coin the proposed method as ‘ReCap’. To begin, we initialize the concoder through knowledge distillation from a pre-trained model. During the training process, we use the output of the initialized concoder as input to a residual attention network, jointly training it with the captioner to achieve semantic consistency. Subsequently, we iteratively update the concoder during the image-text momentum encoding phase using the residual attention network, creating a closed-loop for semantic consistency resolution. Upon completion of training, the concoder and captioner modules can be used independently to perform tasks related to cross-modal retrieval and cross-modal generation of text and images. In order to substantiate the efficacy of our proposed method, we initially conducted empirical experiments encompassing image-text retrieval and image captioning tasks, employing a widely recognized benchmark dataset, MSCOCO. Subsequently, we assessed the fine-grained cross-modal alignment performance of the concoder through an image classification task on the iNaturalist 2018 dataset. The achieved performance metrics notably surpassed those achieved by several incumbent state-of-the-art models, thus validating the proficiency of our method. Finally, motivated by the practical requirements of handling the nature conservation image data, we performed caption annotation for the iNaturalist 2018 dataset. Subsequently, we trained the ReCap on this annotated data. Experimental results demonstrate that our proposed method can maintain semantic consistency between cross-modal retrieval and image caption generation for species with similar visual features but distinct semantics. This achievement signifies a meaningful exploration in the field of cross-modal semantic consistency representation, holding significant practical value in the domain of biological research.

The weight coefficients of the diaphragm spring depend on experiences in the traditional optimization. However, this method not only cannot guarantee the optimal solution but it is also not universal. Therefore, a new optimization target function is proposed. The new function takes the minimum of average compress force changing of the spring and the minimum force of the separation as total objectives. Based on the optimization function, the result of the clutch diaphragm spring in a car is analyzed by the non-dominated sorting genetic algorithm (NSGA-II) and the solution set of Pareto is obtained. The results show that the pressing force of the diaphragm spring is improved by 4.09%by the new algorithmand the steering separation force is improved by 6.55%, which has better stability and steering portability. The problem of the weight coefficient in the traditional empirical design is solved. The pressing force of the optimized diaphragm spring varied slightly during the abrasion range of the friction film, and the manipulation became remarkably light.

As a new plant slope protection technology, living stumps can effectively improve slope stability. The current study attemptes to investigate the mechanical characteristics of the root system of living stumps and their reinforcement mechanism on slopes by carrying out a three-dimensional living stump slope modeling test. Using ABS materials, a 3D model of a living elm stump was created by 3D printing and a slope model test was carried out. The reinforcement mechanism of the living stump was studied through a combination of model testing and numerical simulation. The results indicate that, due to the presence of living stumps in the lower and middle parts of the slope, the maximum shear stress area of the soil moved to a deeper direction. The stress state of the soil around the living stump was effectively improved due to the existence of the lateral root system.Living stumps in the lower part of the slope cross the potential sliding surface and transferred the soil shear stress gradually to the root system through root-soil interaction. Moreover, the tap roots and lateral roots of living stumps form a robust spatial structure that bears the shear stress of soil together, thus increasing the stability of the slope.

With the proliferating of video surveillance system deployment and related applications, real-time video analysis is very critical to achieve intelligent monitoring, autonomous driving, etc. It is non-trivial to achieve high accuracy and low latency video stream analysis through the traditional cloud computing. In this paper, we propose a non-orthogonal multiple access (NOMA) based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the QoE of UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimension, which respectively select the offloading and resource allocation action, the resolution scaling action. Experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of transmission and computation, and have better performance in convergence compared to other baseline.

In this study, a comprehensive review on production scheduling based on reinforcement learning (RL) techniques using bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of all pertinent publications contained in the Web of Science database. The study focuses on a wide spectrum of publications spanning the years between 1998 and 2023. The findings and recommendations of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.

We propose a deep reinforcement learning based manipulator path tracking method to solve the computationally difficult and non-unique problem of manipulator path tracking methods based on inverse kinematics. By transforming the path tracking task into a sequence decision problem, our method adopts an end-to-end learning method for closed-loop control and avoids the process of finding the inverse solution. We first explored the feasibility of the deep reinforcement learning method in the path tracking of the manipulator. After verifying the feasibility, the path tracking of the multi-degree-of-freedom(multi-DOF) manipulator was realized by combining the maximum entropy deep reinforcement learning algorithm. The experimental results show that our method has a good effect on the path tracking of the manipulator, which not only avoids the process of finding the inverse kinematics solution, but also requires no dynamic model. Therefore, we believe that our method has great significance in the study of manipulator path tracking.

In tunnel excavation with boring machines, the tunnel face is supported to avoid collapse and minimise settlement. This article proposes the use of reinforcement learning, specifically the Deep Q-Network algorithm, to predict the face support pressure. The approach is tested both analytically and numerically. By using the soil properties ahead of the tunnel face and the overburden depth as the input, the algorithm is capable of predicting the optimal tunnel face support pressure, adapting to changes in geological and geometrical conditions.

Oedaleus decorus asiaticus, Calliptamus abbreviatus, Angaracris rhodopa, and Myrmeleotettix palpalis are the main grasshoppers that harm the natural grassland in the Hexi Corridor. In this study, the MaxEnt model was employed to identify the key environmental factors affecting the distribution of the four grasshoppers habitats and to assess their distribution under current and future climate change conditions. The aim was to provide a basis for grasshopper monitoring, prediction, and precise control. In this study, 61 occurrences of O. decorus asiaticus, 68 occurrences of A. rhodopa, 58 occurrences of C. abbreviatus, and 92 occurrences of M. palpalis were recorded by GPS, and the distribution of suitable habitats for O. decorus asiaticus, C. abbreviatus, A. rhodopa, and M. palpalis were predicted under current and future climatic scenarios using a MaxEnt model. The average AUC and TSS values of the four grasshoppers were greater than 0.9, and the simulation results were excellent and highly reliable. The mean annual precipitation was the main factor limiting the current range of suitable areas for these four species. Under the current climate, O. asiaticus, C. abbreviatus, and A. rhodopa were mainly distributed in the central and eastern parts of the Hexi Corridor, and M. palpalis was distributed throughout the Hexi Corridor, with a potential distribution area of 1.44×104, 1.43×104, 1.29×104, and 2.12×104 km2, accounting for 15.3, 15.2, 13.7, and 22.5% of the total area of the grasslands in the Hexi Corridor, respectively. The highly suitable areas of O. asiaticus, C. abbreviatus, and A. rhodopa were mainly distributed in the east-central part of Zhangye City, the western part of Wuwei City, and the western and southern part of Jinchang City, with areas of 0.35×104, 0.29×104, and 0.20×104 km2, accounting for 3.7, 3, and 2.2% of the grassland area, respectively. The high habitat of M. palpalis was mainly distributed in the southeast of Jiuquan City, the west, middle, and east of Zhangye City, the west of Wuwei City, and the west and south of Jinchang City, with an area of 0.32×104 km2, accounting for 3.4% of the grassland area. In the 2030s, the range of O. asiaticus, C. abbreviatus, A. rhodopa, and M. palpalis was predicted to decrease.

The informational synthesis of neural structures, processes, parameters and characteristics that allow a unified description and modeling as neural machines of natural and artificial neural systems is presented. The general informational parameters as the global quantitative measure of the neural systems computing potential as absolute and relative neural power were proposed. Neural information organizing and processing follows the way in which nature manages neural information by developing functions, functionalities and circuits related to different internal or peripheral components and also to the whole system through a non-deterministic memorization, fragmentation and aggregation of afferent and efferent information, deep neural information processing representing multiple alternations of fragmentation and aggregation stages. The relevant neural characteristics were integrated into a neural machine type model that incorporates unitary also peripheral or interface components as the central ones. The proposed approach allows overcoming the technical constraints in artificial computational implementations of neural information processes and also provides a more relevant description of natural ones.

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.

Wireless energy harvesting (EH) communication has long been considered a sustainable networking solution. However, it has been limited in efficiency, which has been a major obstacle. Recently, strategies such as energy relaying and borrowing have been explored to overcome these difficulties and provide long-range wireless sensor connectivity. In this article, we examine the reliability of the wireless-powered communication network by maximizing the net bit rate. To accomplish our goal, we focus on enhancing the performance of hybrid access points and information sources by optimizing their transmit power. Additionally, we aim to maximize the use of harvested energy by energy-harvesting relays for both information transmission and energy relaying. However, this optimization problem is complex as it involves non-convex variables and requires combinatorial relay selection indicators optimization for decode and forward (DF) relaying. To simplify this problem, we utilize the Markov decision process and deep reinforcement learning framework based on the deep deterministic policy gradient algorithm. This approach enables us to tackle this non-tractable problem, which conventional convex optimisation techniques would be difficult to solve in complex problem environments. The proposed algorithm significantly improves the end-to-end net bit rate of the smart energy borrowing and relaying EH system by 13.22%,27.57%, and 14.12% compared to the benchmark algorithm based on borrowing energy with an adaptive reward for Quadrature Phase Shift Keying, 8-PSK, and 16-Quadrature amplitude modulation schemes, respectively.

This article focuses on developing a continuous authentication system using behavioral biometrics to recognize users accessing computing devices. The user’s distinct behavioral biometric is captured through keystroke dynamics, and reward-based reinforcement learning (RL) ideas are applied to recognize them throughout the session. The suggested system adds an extra layer of security to traditional authentication methods, forming a robust continuous authentication system that can be added to static authentication systems. The methodology involves training a RL model to detect unusual user typing patterns and flag suspicious activity. Each user has an agent trained on their historical data, which is preprocessed and used to create episodes for the agent to learn from. The environment involves fetching observations and randomly corrupting them to learn out-of-order behavior. The observation vector includes both running features and summary features. The re-ward function is binary and minimalistic. The Principal Component Analysis (PCA) model is used to encode the running features, and the Double Deep Q-Network (DDQN) algorithm with a fully connected neural network is used as the policy net. The evaluation achieved an average training accuracy and EER (equal error rate) of 94.7% and 0.0126 and test accuracy and ERR of 81.06% and 0.0323 for all users when the number of encoder features was increased. Therefore, it is concluded that by continuously learning and adapting to changing behavior patterns, this approach can provide more secure and personalized authentication, lowering the possibility of unauthorized access and cyberattacks. Overall, the use of reinforcement learning and behavioral biometrics for continuous authentication has the potential to significantly enhance security in the digital age and are effective in identifying each user.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

In this paper, we study the performance of various deep reinforcement learning algorithms to enhance the energy management system of a microgrid. We propose a novel microgrid model that consists of a wind turbine generator, an energy storage system, a set of thermostatically controlled loads, a set of price-responsive loads, and a connection to the main grid. The proposed energy management system is designed to coordinate among the different flexible sources by defining the priority resources, direct demand control signals, and electricity prices. Seven deep reinforcement learning algorithms were implemented and are empirically compared in this paper. The numerical results show that the deep reinforcement learning algorithms differ widely in their ability to converge to optimal policies. By adding an experience replay and a semi-deterministic training phase to the well-known asynchronous advantage actor-critic algorithm, we achieved the highest model performance as well as convergence to near-optimal policies.

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.

The fifth Generation (5G) mobile networks use millimeter Waves (mmWaves) to offer giga bit data rates. However, unlike microwaves, mmWave links are prone to user and topographic dynamics. They easily get blocked and end up forming irregular cell patterns for 5G. This in turn cause too early, too late, or wrong handoffs (HOs). To mitigate HO challenges, sustain connectivity and avert unnecessary HO, we propose a HO scheme based on Jump Markov Linear System (JMLS) and Deep Reinforcement Learning (DRL). JMLS is widely known to account for abrupt changes in system dynamics. DRL likewise emerges as an artificial intelligence technique for learning highly dimensional and time-varying behaviors. We combine the two techniques to account for time-varying, abrupt, and irregular changes in mmWave link behaviour by predicting likely deterioration patterns of target links. The prediction is optimized by meta training techniques that also reduces training sample size. Thus, the JMLS-DRL platform formulates intelligent and versatile HO policies for 5G. Results show our proposed prediction scheme about target link behavior post HO to be highly reliable. The scheme also averts unnecessary HOs thus ably supports longer dew time.

The transportation industry is rapidly transitioning from Internal Combustion Engine (ICE) based vehicles to Electric Vehicles (EVs) to promote clean energy. However, large-scale adoption of EVs can compromise the reliability of the power grids by introducing large uncertainty in the demand. Demand response with a controlled charge scheduling strategy for EVs can mitigate such issues. In this paper, a deep reinforcement learning- based charge scheduling strategy is developed for individual EVs by considering user’s dynamic driving behavior and charging preferences. The temporal dynamics of user’s anxiety about charging the EV battery is rigorously addressed. A dynamic weight allocation technique is applied to continuously tune user’s priority for charging and cost-saving with respect to charging duration. The sequential charging control problem is formulated as a Markov decision process, and an episodic approach to the deep deterministic policy gradient (DDPG) algorithm with target policy smoothing and delayed policy update techniques is applied to develop the optimal charge scheduling strategy. A real-world dataset that captures user’s driving behavior, such as arrival time, departure time, and charging duration, is utilized in this study. The extensive simulation results reveal the effectiveness of the proposed algorithm in minimizing energy cost while satisfying user’s charging requirements.

The IEEE 802.11ah standard is introduced to address the growing scale of Internet of Things (IoT) applications. To reduce contention and enhance energy efficiency within the system, the restricted access window (RAW) mechanism is introduced at the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, the RAW parameters need to be appropriately determined to achieve optimal network performance. In this paper, we optimize the configuration of RAW parameters in the IEEE 802.11ah-based IoT network. We analyze and propose an RAW parameters optimization problem with the objective of improving network throughput and formulate it as a Markov decision process. To cope with the complex and dynamic network conditions, we propose a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) algorithm to find the optimal RAW parameters. We construct network environments with periodic and random traffic in the NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. Simulation results reveal that using the PPO-based DRL algorithm, optimal RAW parameters can be obtained under different network conditions, thereby significantly improving network throughput.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1 and Part2, the author previously reported experimental evaluations on Genetic Algorithm and Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part3 of the series of comparative research. In this study, Differential Evolution is adopted as the third instance of major evolutionary algorithms. The experimental results show a statistically significant superiority of DE over both GA and ES (p < .01). In addition, DE exhibits its robustness to variations in hyperparameter configurations (the number of offsprings and generations). In the previous experiments, both ES and GA showed significant performance differences depending on the configurations, whereas in the experiment reported in this article, such differences are not detected for DE.

In recent years, the viability of employing multi-agent reinforcement learning technology for adaptive traffic signal control has been extensively validated. However, owing to restricted communication among agents and the partial observability of the traffic environment, the process of mapping road network states to actions encounters numerous challenges. To address this problem, this paper proposes a multi-agent deep reinforcement learning model with an emphasis on communication content (CMARL). The model decouples the complex relationships between multi-signal agents through centralized training and decentralized execution. Specifically, we first pass the traffic state through an improved deep neural network to achieve the extraction of high-dimensional semantic information and the learning of the communication matrix. Then the agents selectively interact with each other based on the learned communication matrix and generate the final state features. Finally, the features are inputted to the QMIX network to achieve the final action selection. We compare the CMARL model with 6 other baseline algorithms in real traffic networks. The results show that CMARL can significantly reduce vehicle congestion, and run stably in various scenarios.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. In this study, Genetic Algorithm is adopted as another instance of major evolutionary algorithms. Experimental results demonstrate that there was no statistically significant difference between the experimental performances of GA and ES, but the priority of generations and offsprings was different; GA performed better with a greater number of generations while ES performed better with a greater number of offsprings. Eight hidden units were the best among four variations (4, 8, 16 or 32 units), which aligns with previous study using ES.

In this study, reinforcement learning (RL) was used in factory simulation to optimize storage devices for use in Industry 4.0 and digital twins. First, we defined an RL environment, modeled it, and validated its ability to simulate a real physical system. Subsequently, we introduced a method to calculate reward signals and apply them to the environment to ensure the alignment of the behavior of the RL agent with the task objective. The stocker simulation model was used to validate the effectiveness of RL. The model is a storage device that simulates logistics in a manufacturing production area. The results revealed that RL is a useful tool for automating and optimizing complex logistics systems and increase the applicability of RL in logistics. We proposed a novel method for creating an agent through learning using the proximal policy optimization algorithm, and the agent was optimized by configuring various learning options. The application of reinforcement learning resulted in an effectiveness of 30% to 100%, and methods can be expanded to other fields.

Evolutionary algorithms find applicability in the reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the pendulum control task. In this study, the Acrobot control task is adopted as another task. Experimental results demonstrate that ES successfully trained a Multi-Layer Perceptron to achieve a remarkable height of 99.85% concerning the maximum height. However, the trained MLP failed to maintain the chain end in an upright position throughout an episode. In this study, it was observed that employing 8 hidden units in the neural network yielded better results with statistical significance compared to using 4, 16, or 32 hidden units. Furthermore, the findings indicate that a larger population size in ES led to a more extensive exploration of potential solutions over a greater number of generations, which aligns with the previous study.

Transcutaneous injection laryngoplasty is a well-known procedure for treating paralyzed vocal fold by injecting augmentation material into it. Hence, vocal fold localization plays a vital role in the preoperative planning as the fold location is required to determine the optimal injection route. In this communication, we propose a mirror environment based reinforcement learning (RL) algorithm for localizing the right and left vocal folds in preoperative neck CT. RL-based methods commonly showed noteworthy outcome in general anatomic landmark localization problem in the recent years. However, such methods suggest training individual agent for localizing each fold, though the right and left vocal folds are located in close proximity and have high feature-similarity. Utilizing the lateral symmetry between the right and left vocal folds, the proposed mirror environment allows for a single agent for localizing both the folds by treating the left fold as a flipped version of the right fold. Thus, localization of both folds can be trained using a single training session which utilizes the inter-fold correlation and avoids redundant feature learning. Experiment with 120 CT volumes showed improved localization performance and training efficiency of the proposed method compared with the standard RL method.

Aiming at the problem of model error and tracking dependence in the process of intelligent vehicle motion planning, an intelligent vehicle model transfer trajectory planning method based on deep reinforcement learning is proposed, which obtain an effective control action sequence directly. Firstly, an abstract model of the real environment is extracted. On this basis, Deep Deterministic Policy Gradient (DDPG) and vehicle dynamic model are adopted to jointly train a reinforcement learning model, and to decide the optimal intelligent driving maneuver. Secondly, the actual scene is transferred to equivalent virtual abstract scene by transfer model, furthermore, the control action and trajectory sequences are calculated according to trained deep reinforcement learning model. Thirdly, the optimal trajectory sequence is selected according to evaluation function in the real environment. Finally, the results demonstrate that the proposed method can deal with the problem of intelligent vehicle trajectory planning for continuous input and continuous output. The model transfer method improves the model generalization performance. Compared with the traditional trajectory planning, the proposed method output continuous rotation angle control sequence, meanwhile, the lateral control error is also reduced.

A reinforcement learning algorithm is proposed to improve the accuracy of short-term load forecasting (STLF) in this article. The proposed model integrates radial basis function neural network (RBFNN), support vector regression (SVR), and adaptive annealing learning algorithm (AALA). In the proposed methodology, firstly, the initial structure of RBFNN is determined by using SVR. Then, an AALA with time-varying learning rates is used to optimize the initial parameters of SVR-RBFNN (AALA-SVR-RBFNN). In order to overcome the stagnation for searching optimal RBFNN, a particle swarm optimization (PSO) is applied to simultaneously find promising learning rates in AALA. Finally, the short-term load demands are predicted by using the optimal RBFNN. The performance of the proposed methodology is verified on the actual load dataset from Taiwan Power Company (TPC). Simulation results reveal that the proposed AALA-SVR-RBFNN can achieve a better load forecasting precision as compared to various RBFNNs.

This manuscript explores unmanned aerial vehicle DC motor control performance efficacy of deterministic artificial intelligence in comparison to model-following adaptation, particularly a direct self-tuner with filtering. The deterministic artificial intelligence model made use of self-awareness statements to overcome error in response to permutations of the multi-duty cycle square wave that served as the system input. It can be seen that (despite equivalently powerful estimation techniques) deterministic artificial intelligence provided far superior results: a reduction in peak initial transient error of 55%, and a mean error reduction over 81% with over 65% reduction in error standard deviation compared to a state-of-the-art nonlinear adaptive control method. Deterministic artificial intelligence also was able to very closely track at the switching of the input control, while the benchmark nonlinear adaptive control failed to respond as quickly at these points.

Abstract: Toxicity of emerging organic pollutants to photosystems of aquatic plants are still not well clarified. This study aimed to develop a novel ecotoxicological experimental protocol based on nanoscale electrochemical mapping of photosynthetic oxygen evolution of aquatic plants by SECM (scanning electrochemical microscopy). The protocol was also checked by confocal laser scanning microscopy (CLSM), the traditional Clark oxygen electrode method and chlorophyll fluorescence technique. The typical persistent organic pollutant 2,4-dichlorophenol (2,4-DCP) in water environment and the common aquatic Microsorium pteropus (M. pteropus) were chosen as the model organic pollutant and tested plant, respectively. It was found that the SECM method could well discriminated the responses of stoma micromorphology and spatial pattens of photosynthetic oxygen evolution on single stoma. The shape of stoma blurred with increasing 2,4-DCP concentration, which was in good agreement with the CLSM images. The dose-response curves and IC50 values obtained from the SECM data were verified by the data measured by the traditional Clark oxygen electrode method and chlorophyll fluorescence test. The IC50 value of single stoma oxygen emission of 24 h exposed plant leave, which was derived from the SECM current data (32535 μg L-1), was close to those calculated from the maximum photosynthetic efficiency (Fv/Fm) measured by chlorophyll fluorescence test (33963 μg L-1), and the Clark oxygen electrode method photosynthetic oxygen evolution rate (32375 μg L-1). The 72 h and 96 h 2,4-DCP exposure data further confirmed the reliability of the nanoscale stoma oxygen emission mapping methodology for ecotoxicological assessment. In this protocol, the procedures for how to collect effective electrochemical data and how to extract useful information from the single stoma oxygen emission pattern were well established. This study showed that SECM is a feasible and reliable ecotoxicological tool for evaluation of toxicity of organic pollutants to higher plants with unique nanoscale visualization advantage over the conventional methods.

Anthropogenic development can strongly influence natural river processes, leading to environmental changes that negatively affect important habitats and biodiversity, and consequently reduce economically important natural resources. This study investigated the effects of salinity intrusion on the habitat of the clam Corbicula japonica in the Seomjin River estuarine zone. We employed the Environmental Fluid Dynamics Code (EFDC) model, which incorporates topographic data and hydrological changes, to simulate salinity concentrations. Two salinity measurement facilities were installed in Seomjin River estuarine and operated to optimize the EFDC model. The results show that reduced flow rates due to intake have a negligible impact on the increased salinity. Maintaining optimal salinity (15–20 psu) during neap tides at the Seomjin River Bridge requires constant high flow rates, which poses significant challenges. Saltwater stratification is identified as the primary cause of pronounced salinity stratification, particularly during neap tides. Addressing this issue through river discharge and intake facility operation is challenging. Structural measures, including riverbed restoration and underwater barriers, are recommended to improve resistance to seawater intrusion. Future research should aim to develop scenarios to reduce salinity concentrations, quantify the reduction efficiency, and propose region-specific measures.

Collaborative robotics is a major topic in current robotics research posing new challenges, especially in Human-Robot Interaction. The crucial aspect in this area of research focuses on understanding the behavior of robots when engaging with humans, where reinforcement learning is a key discipline that allows us to explore sophisticated emerging reactions. This review aims to delve into the relevance of different sensors and techniques, with special attention to EEG (electroencephalography data on brain activity) and its influence on the behavior of robots interacting with humans. In addition, mechanisms available to mitigate potential risks during the experimentation process such as virtual reality will also be addressed. In the final part of the paper, future lines of research combining the areas of collaborative robotics, reinforcement learning, virtual reality and human factors will be explored, as this last aspect is vital to ensure safe and effective Human-Robot Interactions.

The development of synthetic biology has enabled us to make massive progress on biotechnology and to approach research questions from a brand new perspective. In particular, the design and study of gene regulatory networks in vitro, in vivo and in silico, have played an increasingly indispensable role in understanding and controlling biological phenomena. Among them, it is of great interest to understand how associative learning is formed at the molecular circuit level. Noticeably, mathematical models have been increasingly used to predict the behaviors of molecular circuits. The Fernando’s model, which is thought to be one of the first works in this line of research using the Hill equation, attempted to design a synthetic circuit that mimics Hebbian learning in the neural network architecture. In this article, we carry out in-depth computational analysis of the model and demonstrate that the reinforcement effect can be achieved by choosing the proper parameter values. We also construct a novel circuit that can demonstrate forced dissociation, which was not observed in the Fernando’s model. Our work can be readily used as reference for synthetic biologists who consider implementing the circuits of this kind in biological systems.

Only a handful of microbial mosquito larval pathogens have been described to date. Sampling of several natural enzootic infections of mosquito larvae in southwestern Florida indicated the presence of microbial pathogens capable of extensive larval mortality. Microscopic analysis of one sample site revealed extensive apparent growth of a Pythium-like microbe on mosquito lar-vae, with highest infection observed on the siphon and head regions. Structures consistent with sporangia were seen on infected insects after lactophenol blue staining, and higher resolution scanning electron micrographs (SEM) showed sporangia and encysted zoospores targeting the head and siphon regions. The isolate was single colony purified and molecular identification targeting the ITS and COX1 loci coupled to phylogenetic reconstruction indicated that the isolate belonged to the Pythium genus, but was distinct from its closest characterized species, P. angusta-tum. Morphological features were characterized, with the isolate showing rapid growth on all mycological media tested and relatively high thermotolerance, capable of robust growth at 37oC, and hence was designated, P. thermoculicivorax. Sampling from a second series of natural infec-tions of mosquito larvae resulted in the molecular identification of three Trichoderma isolates, one with high similarity to T. strigosum, and the other two clustering with, but distinct from, T. asperellum. These data highlight the occurrence of natural enzootic infections of mosquito larvae, potentially as a resource for the identification of new mosquito pathogens.

With the development of higher voltage power grid, the high and low voltage parallel loops are emerging, which lead to energy losses, even threaten the security and stability of power system. The multi-infeed HVDC configurations widely appearing in AC/DC interconnected power system make this situation even worse. Aimed at energy saving and system security, a decision optimization method for power grid operating condition with high and low voltage parallel loops is proposed in this paper. Firstly, considering hub substation distribution and power grid structure, parallel loop opening schemes are generated with GN algorithm. Then, candidate opening schemes are preliminarily selected from all these generated schemes based on a filtering index. Finally, with the influence on power system security, stability and operation economy in consideration, an evaluation model for candidate opening schemes is founded based on analytic hierarchy process (AHP). And a fuzzy evaluation algorithm is used to find the optimal scheme. Simulation results of New England 39-bus system and an actual power system validate the effectiveness and superiority of this proposed method.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, the author previously reported experimental evaluations on Evolution Strategy　for reinforcement learning of binary neural networks, utilizing the Pendulum control task. This article constitutes Part2 of the series of comparative research. In this study, Genetic Algorithm is adopted as another evolutionary algorithm. Wilcoxon signed rank test revealed that there was no statistically significant difference between the fitness scores obtained using GA and those using ES. However, the p-value 0.11 indicated that GA worked better than ES on this training task. As the values of binary weights, {-1, 1} were significantly superior to {0, 1} (p < .01). The motion of the pendulum controlled by the binary MLP after the training showed that the binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion.

The climate crisis necessitates a global shift to achieve a secure, sustainable, and affordable energy system toward a green energy transition reaching climate neutrality by 2050. Because of this, renewable energy sources have come to the forefront and the research interest in microgrids which rely on distributed generation and storage systems has exploded. Furthermore, many new markets for energy trading, ancillary services, and frequency reserve markets have provided attractive investment opportunities in exchange for balancing the supply and demand of electricity. Artificial intelligence can be utilized to locally optimize energy consumption, trade energy with the main grid, and participate in these markets. Reinforcement learning (RL) is one of the most promising approaches to achieve this goal because it enables an agent to learn optimal behavior in a microgrid by executing specific actions that maximize the long-term reward signal/function. The study focuses on testing two optimization algorithms: logic-based optimization and reinforcement learning. This paper proposes an optimization algorithm based on reinforcement learning in an industrial microgrid that is capable of trading energy with the main grid, providing significant cost savings. The RL-based approach is implemented in Python based on real data from the site and in combination with MATLAB-Simulink to validate its results. The application of the RL algorithm achieved an average monthly cost saving of 20% compared to logic-based optimization and 86% savings compared to not using any optimization. These findings contribute to digitalization and decarbonization of energy technology and support fundamental goals and policies of the European Green Deal.

Stagnation at local optima represents a significant challenge in bio–inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca Predator Algorithm with Deep Q–Learning. Orca Predator Algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q–Learning is a reinforcement learning technique that combines Q–Learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique’s performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of Orca Predator Algorithm to identify promising regions nearby in the search space, complemented by the fine–tuning capabilities of Deep Q–Learning to navigate these areas precisely. The practical application of this approach is evaluated using the high–dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of Orca Predator Algorithm against its native version and a set of state–of–the–art algorithms.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, 2 and 3, the author previously reported experimental evaluations on Evolution Strategy, Genetic Algorithm, and Differential Evolution for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part4 of the series of comparative research. In this study, Particle Swarm Optimization is adopted as an instance of major swarm intelligence algorithms. The experimental result shows that PSO performed worse than all of DE, GA and ES. The difference between PSO and DE was statistically significant (p<.01). In addition, PSO exhibited lower capability in exploring solutions in high-dimensional search spaces than DE, GA, and ES did. A larger swarm size compensated for the weakness of PSO in global exploration, thus making itself more beneficial than a larger number of swarm search iterations.

The author previously reported an experimental result of evolutionary reinforcement learning of neural network controllers for the pendulum task. In the previous work, a conventional multilayer perceptron was employed in which connection weights were real numbers. In this study, the author experimentally applies an evolutionary algorithm to the reinforcement training of binary neural networks. In both studies, the same task and the same evolutionary algorithm are utilized, i.e. the pendulum control problem and Evolution Strategy respectively. The only differences lie in the memory size per connection weight and the model size of the neural network. The findings from this study are (1) the performance of the binary MLP with 32 hidden units was inferior to that of the real-valued MLP with 16 hidden units; however, this difference was not statistically significant (p >.05); (2) the trained binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion, as the real-valued MLP had done; and (3) the memory size required to record the binary MLP with 32 hidden units is 3.1% or 6.2% of the memory size required to record the real-valued MLP with 16 hidden units.

German noise action plans aim to reduce negative health outcomes from noise exposure and identify quiet areas free of noise pollution. Quiet area identification in German noise action plans is based primarily on noise mapping and spatial analysis and not empirical or qualitative data about acoustic environments, thus leaving a gap in the understanding of the quality of formally recognized quiet areas in noise action plans. This work presents a comparative empirical case study in Dortmund, Germany, with the aim to describe the diurnal dB(A) and biophonic properties of quiet areas versus noise 'hot spots'. Sound observations were collected (n=282,764) in five different natural or recreational land use patch types larger than four acres within 33 proposed quiet areas in Dortmund (n=70) and 23 noise hot spots between April 27^th, 2022 and March 2^nd, 2023. We find that quiet areas are on average more than 20 dB(A) quieter than noise hot spots almost every hour of the day. Forests, managed tree stands, cemeteries, and agriculture diel patterns are dominated by dawn dusk chorus in spring and summer, whereas sport and recreation as well as noise hot spots are dominated by traffic and human noise. A novel composite biophony mapping procedure is presented that finds distinct temporal distribution of biophony in forested and agriculture peri-urban locations positively associated with patch size, distance away from LDEN > 55, proximity to water, and the number of vegetation layers in the plant community. Anthrophony distribution dominates urban land uses in all hours of the day, but expands during day and evening and contracts at night and in dusk hours. The procedures presented here illustrate how qualitative information regarding quiet areas can be integrated into German noise action planning.

Urban rail transit hub platform is the most important area for passenger flow distribution. In order to calculate passenger flow volume in platform and evaluate platform service level during rush hours, this paper presents a method for modeling and simulation of passenger flow distribution in platform. Passenger flow distribution model (PFDM) is proposed based on the basic analysis and the superposition principle of passenger flow. Simulation design for PFDM is proposed by Anylogic, which contains simulation process and simulation model. Experiment results show that PFDM and simulation design are effective and accordant with the reality scenario, and the simulation precision is comparatively ideal. This research could provide a beneficial reference for train scheduling and operation management under the viewpoint of traffic safety and service level.

Habituation is the most ancient and fundamental form of learning. As a firewall that protects our brain from sensory overload, it is indispensable for higher cognitive processes. Studies in humans and animal models provide a growing body of evidence that habituation is affected in autism and related monogenic neurodevelopmental disorders (NDDs). An integrated application of habituation assessment in NDDs and their animal models has currently unexploited potential for fundamental neuroscience and medical care.With the aim to gain mechanistic insights, we systematically retrieved genes that have been demonstrated in the literature to underlie habituation. We identified 258 evolutionarily conserved genes across species, describe the biological processes they converge on, and highlight regulatory pathways and drugs that may alleviate the habituation deficits associated with their dysregulation. We also summarize current habituation paradigms and extract the most decisive arguments from the literature that support the crucial role of habituation for cognition in health and disease. We conclude that habituation is a conserved, quantitative, cognition- and disease-relevant process that can connect preclinical and clinical work, and hence is a powerful tool to advance research, diagnostics,’’ and treatment of NDDs.

Wearable technology has been proposed as a potential tool to change the way of human life, such as the smart bracelet and the Google Glass. In the wearable technology, the inertial sensor has great significance in tracking the object movements. The paper focused on detecting the movements of user’s finger based on the inertial sensor to give the control signals. Firstly, the attitude matrix, which represented the transformation relation of carrier coordinate system and the navigation coordinate system, was obtained. Secondly, the attitude matrix was expressed based on the quaternions. Thirdly, the finger gesture was processed by the attitude matrix to get the attitude angle. Finally, the robot was controlled by attitude angle to make the moving action. The experimental results showed the detection of the finger movement is effective.

While researchers have made notable progress in bio-inspired swimming robot development, a persistent challenge lies in creating propulsive gaits tailored to these robotic systems. The California sea lion achieves its robust swimming abilities through a careful coordination of foreflippers and body segments. In this paper, reinforcement learning (RL) was used to develop a novel sea lion foreflipper gait for a bio-robotic swimmer using a numerically modelled computational representation of the robot. This model integration enabled reinforcement learning to develop desired swimming gaits in the challenging underwater domain. The novel RL gait outperformed the characteristic sea lion foreflipper gait in the simulated underwater domain. When applied to the real-world robot, the RL constructed novel gait performed as well or better than the characteristic sea lion gait in many factors. This work shows the potential for using complimentary bio-robotic and numerical models with reinforcement learning to enable the development of effective gaits and maneuvers for underwater swimming vehicles.