Managing a portfolio whose value closely tracks an index by trading only in a subset of the index constituents involves an NP-hard optimisation problem. In the prior literature, it has been suggested to solve this problem using sequential Monte Carlo (SMC, a.k.a. particle filter) methods. However, this literature does not take transaction costs into account, although transaction costs are the primary motivation for attempting to replicate the index by trading in a subset, rather than the full set of index constituents. This paper extends the SMC approach to index tracking to allow for proportional transaction costs and implements this extended method on empirical data for a variety stock indices. In addition to providing a more practically useful tracking strategy by allowing for transaction costs, we find that including a penalty for transaction costs in the optimisation objective can actually lead to better tracking performance.

This paper compares the finite sample performance of three non-parametric threshold estimators via Monte Carlo method. Our results show that the finite sample performance of the three estimators is not robust to the relative position of the threshold level along the distribution of threshold variable, especially when a structural change occurs at the tail part of the distribution.

To sample from complex, high-dimensional distributions, one may choose algorithms based on the Hybrid Monte Carlo (HMC) method. HMC-based algorithms generate nonlocal moves alleviating diffusive behavior. Here, I build on an already defined HMC framework, Hybrid Monte Carlo on Hilbert spaces [A. Beskos, F.J. Pinski, J.-M. Sanz-Serna and A.M. Stuart, Stoch. Proc. Applic. 121, 2201 - 2230 (2011); doi:10.1016/j.spa.2011.06.003] that provides finite-dimensional approximations of measures π which have density with respect to a Gaussian measure on an infinite-dimensional Hilbert (path) space. In all HMC algorithms, one has some freedom to choose the mass operator. The novel feature of the algorithm described in this article lies in the choice of this operator. This new choice defines a Markov Chain Monte Carlo (MCMC) method which is well defined on the Hilbert space itself. As before, the algorithm described herein uses an enlarged phase space Π having the target π as a marginal, together with a Hamiltonian flow that preserves Π. In the previous method, the phase space π was augmented with Brownian bridges. With the new choice for the mass operator, π is augmented with Ornstein-Uhlenbeck (OU) bridges. The covariance of Brownian bridges grows with its length, which has negative effects on the Metropolis-Hasting acceptance rate. This contrasts with the covariance of OU bridges which is independent of the path length. The ingredients of the new algorithm include the definition of the mass operator, the equations for the Hamiltonian flow, the (approximate) numerical integration of the evolution equations, and finally the Metropolis-Hastings acceptance rule. Taken together, these constitute a robust method for sampling the target distribution in an almost dimension-free manner. The behavior of this novel algorithm is demonstrated by computer experiments for a particle moving in two dimensions, between two free-energy basins separated by an entropic barrier.

An iterative outlier elimination procedure based on hypothesis testing, commonly known as Iterative Data Snooping (IDS) among geodesists, is often used for the quality control of the modern measurement systems in geodesy and surveying. The test statistic associated with IDS is the extreme normalised least-squares residual. It is well-known in the literature that critical values (quantile values) of such a test statistic cannot be derived from well-known test distributions, but must be computed numerically by means of Monte Carlo. This paper provides the first results about Monte Carlo-based critical value inserted to different scenarios of correlation between the outlier statistics. From the Monte Carlo evaluation, we compute the probabilities of correct identification, missed detection, wrong exclusion, overidentifications and statistical overlap associated with IDS in the presence of a single outlier. Based on such probability levels we obtain the Minimal Detectable Bias (MDB) and Minimal Identifiable Bias (MIB) for the case where IDS is in play. MDB and MIB are sensitivity indicators for outlier detection and identification, respectively. The results show that there are circumstances that the larger the Type I decision error (smaller critical value), the higher the rates of outlier detection, but the lower the rates of outlier identification. For that case, the larger the Type I Error, the larger the ratio between MIB and MDB. We also highlight that an outlier becomes identifiable when the contribution of the measures to the wrong exclusion rate decline simultaneously. In that case, we verify that the effect of the correlation between the outlier statistics on the wrong exclusion rates becomes insignificant from a certain outlier magnitude, which increases the probability of identification.

Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

Every firm is devoted to polishing and fine-tuning its maintenance strategy. More exactly, when we develop robust and efficient preventive and conditional maintenance approaches, our objective is to combine performance and robustness to increase maintenance decision-making. This, in turn, leads to a reduction in system failures and a subsequent drop in related expenses. This impulse drives the origin of this essay: to provide fresh criteria that allow us to improve and optimize maintenance techniques. Furthermore, our study has been realistically utilized via the deployment of condition-based maintenance (CBM) techniques, which are commonly regarded as a critical lever that allows firms to gain a competitive advantage in the context of the fourth industrial revolution. Combining both perfect and imperfect maintenance actions into CBM strategies appears to be an effective technique to obtain high economic performance and robustness in the maintenance strategies of the companies. Following this idea, we propose and optimize in this paper a set of criteria enabling the combined assessment of the mean economic performance and the resilience of various kinds of maintenance techniques. The advantage of the proposed criterion is that it adapts to different types of maintenance strategies and provides access to a simple and relevant evaluation model. Thanks to this new criterion, we will be able to choose the better maintenance strategies that have more performance and robustness between these strategies for a system with more or less stable behavior. The Monte Carlo Method is incorporated into the comparative assessment of maintenance methods, adding a layer to the evaluation of the performance and robustness of each adaptation in decision-making.

In Brazil, the production of corn and soybeans has been growing significantly in recent years. Considering that the strategies for investing in the production of these crops are conditioned by production costs and risk, the objective of this article is to propose a simulation model that indicates the trend of production costs for these commodities, considering the dispersion and correlation of selected key variable prices and corn and soybean production costs practiced between the years of 2018 and 2022. Fifty specialized companies in the state of Sao Paulo, Brazil, provided data for this study, and Monte Carlo simulations showed that the total cost of corn production/ha is between USD 600.00 and USD 1150.00, with a level of certainty of 84.7%, and soybeans are in the range of USD 260.00 to USD 420.00, with a level of certainty of 86.4%. The model evidenced a trend of decreasing production costs for the crops for the 2023/24 and 2024/25 harvests, as the input with the greatest impact (fertilizer) is showing a downward trend. On the other hand, costs related to labor, soybean seed, and fungicides are showing an upward trend, while dolomitic limestone corrective remains stable.

Long-range interactions are relevant for a large variety of quantum systems in quantum optics and condensed matter physics. In particular, the control of quantum-optical platforms promises to gain deep insights in quantum-critical properties induced by the long-range nature of interactions. From a theoretical perspective, long-range interactions are notoriously complicated to treat. Here, we give an overview of recent advancements to investigate quantum magnets with long-range interactions focusing on two techniques based on Monte Carlo integration. First, the method of perturbative continuous unitary transformations where classical Monte Carlo integration is applied within the embedding scheme of white graphs. This linked-cluster expansion allows to extract high-order series expansions of energies and observables in the thermodynamic limit. Second, stochastic series expansion quantum Monte Carlo which enables calculations on large finite systems. Finite-size scaling can then be used to determine physical properties of the infinite system. In recent years, both techniques have been applied successfully to one- and two-dimensional quantum magnets involving long-range Ising, XY, and Heisenberg interactions on various bipartite and non-bipartite lattices. Here, we summarise the obtained quantum-critical properties including critical exponents for all these systems in a coherent way. Further, we review how long-range interactions are used to study quantum phase transitions above the upper critical dimension and the scaling techniques to extract these quantum critical properties from the numerical calculations.

We consider a class of density-driven flow problems. We are particularly interested in the problem of the salinization of coastal aquifers. We consider the Henry saltwater intrusion problem with uncertain porosity, permeability, and recharge parameters as a test case. The reason for the presence of uncertainties is the lack of knowledge, inaccurate measurements, and inability to measure parameters at each spatial or time location. This problem is nonlinear and time-dependent. The solution is the salt mass fraction, which is uncertain and changes in time. Uncertainties in porosity, permeability, recharge, and mass fraction are modeled using random fields. This work investigates the applicability of the well-known multilevel Monte Carlo (MLMC) method for such problems. The MLMC method can reduce the total computational and storage costs. Moreover, the MLMC method runs multiple scenarios on different spatial and time meshes and then estimates the mean value of the mass fraction. The parallelization is performed in both the physical space and stochastic space. To solve every deterministic scenario, we run the parallel multigrid solver ug4 in a black-box fashion. We use the solution obtained from the quasi-Monte Carlo method as a reference solution.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

This paper concentrates on computing the efficiency of heliostat field systems. By conducting a comprehensive analysis of the mathematical model for optical efficiency, we have effectively simplified the complex three-dimensional illumination problem of heliostat fields into a two-dimensional issue. Our work primarily investigates the calculation methods for the average optical efficiency and output power of heliostat. Traditionally, calculating shadow and blocking efficiency, key factors affecting the optical performance of mirror fields, has posed a significant technical challenge. Through innovative improvements to the Monte Carlo simulation technique, this study has successfully reduced the spatial dimensionality of the problem. We employed a series of novel coordinate transformations to simulate light tracing on a two-dimensional plane, thereby efficiently calculating the efficiency of shadows and obstructions. Not only have we solved the computational challenges of shadow and obstruction efficiency, but our unique algorithm design also allows us to precisely obtain the overall average optical efficiency of the heliostat field while enhancing computational efficiency.

The entropy models have been recently adopted in many studies to evaluate the distribution of the shear stress in circular channels. However, the uncertainty in their predictions and their reliability remains an open question. We present a novel method to evaluate the uncertainty of four popular entropy models, including Shannon, Shannon-Power Low (PL), Tsallis, and Renyi, in shear stress estimation in circular channels. The Bayesian Monte-Carlo (BMC) uncertainty method is simplified considering a 95% Confidence Bound (CB). We developed a new statistic index called as FREE_opt-based OCB (FOCB) using the statistical indices Forecasting Range of Error Estimation (FREE) and the percentage of observed data in the CB (N_in), which integrates their combined effect. The Shannon and Shannon PL entropies had close values of the FOCB equal to 8.781 and 9.808, respectively, had the highest certainty in the calculation of shear stress values in circular channels followed by traditional uniform flow shear stress and Tsallis models with close values of 14.491 and 14.895, respectively. However, Renyi entropy with much higher values of FOCB equal to 57.726 has less certainty in the estimation of shear stress than other models. Using the presented results in this study, the amount of confidence in entropy methods in the calculation of shear stress to design and implement different types of open channels and their stability is determined.

This paper investigates the inverse option problems (IOP) in the extended Black--Scholes model arising in financial markets. We identify the volatility and the drift coefficient from the measured data in financial markets using a Bayesian inference approach, which is presented as an IOP solution. The posterior probability density function of the parameters is computed from the measured data. The statistics of the unknown parameters are estimated by a Markov Chain Monte Carlo (MCMC) algorithm, which exploits the posterior state space. The efficient sampling strategy of the MCMC algorithm enables us to solve inverse problems by the Bayesian inference technique. Our numerical results indicate that the Bayesian inference approach can simultaneously estimate the unknown trend and volatility coefficients from the measured data.

Dynamic hedging has been adopted by many insurance companies to mitigate the financial risks associated with variable annuity guarantees. In order to simulate the performance of dynamic hedging for variable annuity products, insurance companies rely on nested stochastic projections, which is highly computationally intensive and often prohibitive for large variable annuity portfolios. Metamodeling techniques have recently been proposed to address the computational issues. However, it is difficult for researchers to obtain real datasets from insurance companies to test metamodeling techniques and publish the results in academic journals. In this paper, we create synthetic datasets that can be used for the purpose of addressing the computational issues associated with the nested stochastic valuation of large variable annuity portfolios. The runtime used to create these synthetic datasets would be about 3 years if a single CPU were used. These datasets are readily available to researchers and practitioners so that they can focus on testing metamodeling techniques.

The total enthalpies of the 16 different spin configurations that can be realized in the unit cell of the archetype spin crossover complex [Fe(phen)₂(NCS)₂] (phen=1,2-phenanthroline) were calculated applying periodic density functional theory combined with the Hubbard model and the Grimme-D2 dispersion correction (DFT+U/D2). The obtained enthalpy differences between the individual spin configurations were used to determine spin couplings of an Ising-like model and subsequent Monte Carlo simulations for this model allowed to estimate the phenomenological interaction parameter Г of the Slichter-Drickamer model, which is commonly used to describe the cooperativity of the spin transition. The calculational procedure described here, which led to an estimate of about 3 kJ/mol⁻¹ for Г, in good agreement with experiment, may be used to predict from first principles how modifications of spin crossover complexes can change the character of the spin transition from gradual to abrupt and vice versa.

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered remarkable success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Although, typically, the communication protocol between agents is manually specified and not altered during training, recently, some papers have shown signs of trying to emerge a communication between agents on the one hand and, on the other hand, to understand what is exchanged between agents. So, there is a growing body of literature on this topic which includes qualitative and quantitative studies and the ones that apply mixed methods. This study presents the scoping review of the methodological strategies undertaken in a total of 16 research articles. The results present the critical appraisal of quantitative methods in terms of validity and reliability and for qualitative methods considering four trustworthiness factors. In the end, relevant insights are further explored with implications and reflections on how they can benefit one's research in the field.

Flexible implementations of Monte Carlo Tree Search (MCTS), combined with domain specific knowledge and hybridization with other search algorithms, can be a very powerful for the solution of problems in complex planning. We introduce mctreesearch4j, a standard MCTS implementation written as a standard JVM library following key design principles of object oriented programming. We define key class abstractions allowing the MCTS library to flexibly adapt to any well defined Markov Decision Process or turn-based adversarial game. Furthermore, our library is designed to be modular and extensible, utilizing class inheritance and generic typing to standardize custom algorithm definitions. We demon- strate that the design of the MCTS implementation provides ease of adaptation for unique heuristics and customization across varying Markov Decision Process (MDP) domains. In addition, the implementation is reasonably performant and accurate for standard MDP’s. In addition, via the implementation of mctreesearch4j, the nuances of different types of MCTS algorithms are discussed.

Sword-duels are interesting, from a game-theoretical perspective, as they correspond to sudden-death interactions between players (i.e., defeat means removal from the game), and where victory/defeat depend on not just skill, but also luck. Analyzing probabilities of victory streaks, given a certain level of self and others' skill, is thus relevant for any application domain that can be modeled in the same way. This paper takes inspiration from Miyamoto Musashi, famously undefeated for 61 duels, and implements a Markov-chain Monte-Carlo simulation approach to evaluate this scenario. Results suggest that a 61 victory streak can be probabilistically observed when skill level is roughly 6.5 times that of the average duelist, and with 95\% confidence when skill level is roughly 1000 times that of the average duelist. More generally, this paper provides a method for determining chances of victory streaks in game-theoretical sudden-death encounters, when both "skill" and "luck" contribute to the outcome of the encounter. Specific scenarios can be modeled by modifying the utilized Markov chain and adjusting sampled distributions as required.

We numerically study surface models defined on hexagonal disks with a free boundary. 2D surface models for planer surfaces have recently attracted interest due to the engineering applications of functional materials such as graphene and its composite with polymers. These 2D composite meta-materials are strongly influenced by external stimuli such as thermal fluctuations if they are sufficiently thin. For this reason, it is very interesting to study the shape stability/instability of thin 2D materials against thermal fluctuations. In this paper, we study three types of surface models including Landau-Ginzburg (LG) and Helfirch-Polyakov models defined on triangulated hexagonal disks using the parallel tempering Monte Carlo simulation technique. We find that the planer surfaces undergo a first-order transition between the smooth and crumpled phases in the LG model and continuous transitions in the other two models. The first-order transition is relatively weaker compared to the transition on spherical surfaces already reported. The continuous nature of the transition is consistent with the reported results, although the transitions are stronger than that of the reported ones.

After 2010, the source model of the microSelectron HDR Afterloader System was slightly modified from the previous model. Granero et al. named the modified source model “mHDR-v2r (revised model mHDR-v2)” and the previous model “mHDR-v2”. They concluded that the dosimetric differences arising from the dimensional changes between the mHDR-v2 and mHDR-v2r designs were negligible at almost all locations (within 0.5 % for r ≥ 0.25 cm), the two-dimensional anisotropy function difference between the two sources is found 2.1 % at r = 1.0 cm when compared with the results of the other experimental group. To confirm this difference, we performed a full Monte Carlo simulation without the energy fluence approximation. This is useful near the radiation source where charged-particle equilibrium does not hold. The two-dimensional anisotropy function of the TG-43U1 dataset showed a few percent difference between the mHDR-v2r and mHDR-v2 sources. There was no agreement in the immediate vicinity from the source (0.10 cm and 0.25 cm), when compared to Granero et al. in mHDR-v2r sources. The differences in these two-dimensional anisotropy functions were identified.

Model selection and model averaging have been the popular approaches in handling modelling uncertainties. Fan and Li(2006) laid out a unified frame work for variable selection via penalized likelihood. The tuning parameter selection is vital in the optimization problem for the penalized estimators in achieving consistent selection and optimal estimation. Since the OLSpost-LASSO estimator by Belloni and Chernozhukov (2013), few studies have focused on the finite sample performances of the class of OLS post-penalty estimators with the tuning parameter choice determined by different tuning parameter selection approaches. We aim to supplement the existing model selection literature by studying such a class of OLS post-selection estimators. Inspired by the Shrinkage Averaging Estimator (SAE) by Schomaker(2012) and the Mallows Model Averaging (MMA) criterion by Hansen (2007), we further propose a Shrinkage Mallows Model Averaging (SMMA) estimator for averaging high dimensional sparse models. Based on the Monte Carlo design by Wang et al. (2009) which features an expanding sparse parameter space as the sample size increases, our Monte Carlo design further considers the effect of the effective sample size and the degree of model sparsity on the finite sample performances of model selection and model averaging estimators. From our data examples, we find that the OLS post-SCAD(BIC) estimator in finite sample outperforms most of the current penalized least squares estimators as long as the number of parameters does not exceed the sample size. In addition, the SMMA performs better given sparser models. This supports the use of the SMMA estimator when averaging high dimensional sparse models.

The description of protein disordered states is important for understanding protein folding mechanisms and their functions. In this short review, we briefly describe a simulation approach to modeling disordered protein interactions and unfolded states of globular proteins. It is based on the CABS coarse-grained protein model that uses a Monte Carlo (MC) sampling scheme and a knowledge-based statistical force field. We review several case studies showing that description of protein disordered states resulting from CABS simulations is consistent with experimental data. The case studies comprise investigations of protein-peptide binding and protein folding processes. The CABS model has been recently made available as the simulation engine of multiscale modeling tools enabling studies of protein-peptide docking and protein flexibility. Those tools offer customization of the modeling process, driving the conformational search using distance restraints, reconstruction of selected models to all-atom resolution and studies of large protein systems in a reasonable computational time. Therefore, CABS can be combined in integrative modeling pipelines incorporating experimental data and other modeling tools of various resolution.

Estimates of extreme water level return periods in river systems are crucial for hydraulic engineering design and planning. Recorded historical water level data of Taiwan’s rivers are not long enough for traditional frequency analyses when predicting extreme water levels for different return periods. In this study, the integration of a one-dimensional flash flood routing hydrodynamic model with the Monte Carlo simulation was developed to predict extreme water levels in the Danshuei River system of northern Taiwan. The numerical model was calibrated and verified with observed water levels using four typhoon events. The results indicated a reasonable agreement between the model simulation and observation data. Seven parameters, including the astronomical tide and surge height at the mouth of the Danshuei River and the river discharge at five gauge stations, were adopted to calculate the joint probability and generate stochastic scenarios via the Monte Carlo simulation. The validated hydrodynamic model driven by the stochastic scenarios was then used to simulate extreme water levels for further frequency analysis. The design water level was estimated using different probability distributions in the frequency analysis at five stations. The design high-water levels for a 200-year return period at Guandu Bridge, Taipei Bridge, Hsin-Hai Bridge, Da-Zhi Bridge, and Chung-Cheng Bridge were 2.90 m, 5.13 m, 6.38 m, 6.05 m, and 9.94 m, respectively. The estimated design water levels plus the freeboard are proposed and recommended for further engineering design and planning.

The numerical implementation of the controllers allows the use of very complex algorithms with ease. However, in practice, due to its proven advantages, the PID controller (and its variants) is widely used in industrial control systems as well as in many other applications that require continuous control. Most of the methods for tuning the parameters of PID controllers are based on time-invariant linear models of the processes, which in practice can lead to poor performance of the control system. The paper presents an application of reinforcement learning algorithms in tuning of PID controllers for the control of some classes of continuous nonlinear systems. Tuning the parameters of PID controllers is done with the help of Twin Delayed Deep Deterministic Policy Gradients (TD3) algorithm which presents a series of advantages compared to other similar methods from Machine Learning dedicated to continuous state and action spaces. TD3 algorithm is an off-policy Actor-Critic based method and was used as it does not require a system model. The presented technique is applied for control of a biotechnological system which has a strongly nonlinear dynamic. The proposed tuning method is compared to the classical tuning methods of PID controllers. The performance of the tuning method based on the TD3 algorithm is demonstrated through simulation illustrating the effectiveness of the proposed methodology.

Background. The Stanford Health Assessment Questionnaire-Disability Index (HAQ) is widely used to measure functional ability in persons with Rheumatoid Arthritis (RA). The instrument was developed with limited involvement from persons with RA, and their perception of the instrument has not been studied in depth. The aim of this study was to explore how persons with RA experience the use of the HAQ in care. Methods. Forty persons with RA were purposefully recruited to participate in semi-structured interviews. The interviews were then analyzed qualitatively using thematic analysis. Results. The participants questioned the relevance of the HAQ but nevertheless experienced that the instrument had a profound effect on their understanding of health and how care is delivered. The analysis resulted in three themes: Problems with individual items, meaning of the summative score, and effects on care and health perceptions. Conclusions. To make the HAQ relevant to persons with RA, it needs to be revised or to include an option to select items most meaningful to the respondent. To ensure relevance, the HAQ update should preferably be co-created by researchers, clinicians and persons with RA.

The modern supply chain (SC) is growing in terms of data, devices, users, and stakeholders, which introduced new security challenges and threats, especially with the reliance on centralized servers or cloud platforms. In addition, increased trust among system participants exposes the SC to a higher risk of vulnerabilities which require strong security measures. This article proposes a hybrid security framework for SC systems, BC-DRLzSC, that integrates Blockchain (BC) and Deep Reinforcement Learning (DRL) designed to operate in a zero trust (ZT) environment. In particular, we propose a decentralized BC-based approach integrated with smart contracts to manage system participant registration and authentication and to control access to system resources. BC-DRLzSC adopts a ZT architecture to reinforce SC security, which can be achieved with an advocate to verify each entity’s trustworthiness before granting or retaining access to system resources. Incorporating the ZT architecture, with BC and DRL, can potentially and significantly bolster SC system security. DRL is employed to develop a proactive attack detection model that continuously monitors the incoming traffic from authenticated nodes within the network and predicts any malicious actions. Finally, we evaluate the performance of our proposed DRL solution using the NSL-KDD dataset.

Monte Carlo Filtering (MCF) is one of the methods of Experimental Statistical Energy Analysis (E-SEA), which allows the correction of a negative LF (Loss Factor). In this article, a modification of the MCF method, called DESA (Diagonal Extension of the Search Area), is proposed. The technique applies a non-uniform extension of the search area when generating a population of normalized energy matrices. The degree of expansion of the search area is controlled by the Diagonal Penalty Factor (DPF). The authors demonstrated the method's effectiveness on a system that could not be identified in several frequency bands by the classical MCF method. After applying DESA, it was possible to fill in the problematic bands that were missing CLF and DLF values. The paper also proposes a way to minimize the errors introduced by using overly high DPF values.

The rational fabrication of low-dimensional materials with a well-defined topology and functions is an incredibly important aspect of nanotechnology. In particular, the on-surface synthesis (OSS) methods based on the bottom-up approach enable facile construction of sophisticated molecular architectures unattainable by traditional methods of wet chemistry. Among such supramolecular constructs especially interesting are surface-supported metal-organic networks (SMONs), composed of low-coordinated metal atoms and π-aromatic bridging linkers differing in size, shape and functionalization scheme. In this work, the lattice Monte Carlo (MC) simulation technique was used to extract the chemical information encoded in a family of Y-shaped positional isomers coadsorbed with trivalent metal atoms on a flat metallic surface with (111) geometry. For this purpose the investigated tritopic molecules/metal atoms were carefully mapped on a triangular lattice as interconnected/isolated segments with appropriatelly embedded anisotropic interactions. Depending on the intramolecular distribution of active centers (functional groups) within the simulated molecular bricks, we observed the metal-directed self-assembly of two-dimensional (2D) openwork patterns, aperiodic mosaics and metal-organic ladders. The obtained theoretical findings could be especially relevant for the scanning tunneling microscopy (STM) experimentalists interested in a surface-assisted construction of complex nanomaterials stabilized by directional coordination bonds.

Computationally modelling a nuclear reactor startup core for a benchmark against the existing models is highly desirable for an independent assessment informing nuclear engineers and energy policymakers. This work presents a startup core model of the UK’s first Evolutionary Pressurised Water Reactor (EPR) based on Monte Carlo simulations of particle collisions using Serpent 2, a continuous-energy Monte Carlo reactor physics burnup code. Coupling between neutronics and thermal-hydraulic conditions with the fuel depletion is incorporated into the multi-dimensional branches, obtaining the thermal flux and fission rate (power) distributions radially and axially from the three dimensional (3D) single assembly level to a 3D full core. Shannon entropy is employed to characterise the convergence of the fission source distribution, with 3 billion neutron histories tracked by parallel computing. Source biasing is applied for the variance reduction. Benchmarking the proposed Monte Carlo 3D full-core model against the traditional deterministic transport computation suite used by the UK Office for Nuclear Regulation (ONR), a reasonably good agreement within statistics is demonstrated for the safety-related reactivity coefficients, which creates trust in the EPR safety report.

In this paper the authors present a method which facilitates computationally efficient parameter estimation of dynamical systems from a continuously growing set of measurement data. It is shown that the proposed method, which utilises Sequential Monte Carlo samplers, is guaranteed to be fully parallelisable (in contrast to Markov chain Monte Carlo methods) and can be applied to a wide variety of scenarios within structural dynamics. Its ability to allow convergence of one's parameter estimates, as more data is analysed, sets it apart from other sequential methods (such as the particle filter).

The present grievous situation of the tuberculosis disease can be improved by efficient case management and timely follow-up evaluations. With the advent of digital technology this can be achieved by quick summarization of the patient-centric data. The aim of our study was to assess the effectiveness of the ODK Scan paper-to-digital system during testing period of three months. A sequential, explanatory mixed-method research approach was employed to elucidate technology use. Training, smartphones, application and 3G enabled SIMs were provided to the four field workers. At the beginning, baseline measures of the data management aspects were recorded and compared with endline measures to see the impact of ODK Scan. Additionally, at the end, users’ feedback was collected regarding app usability, user interface design and workflow changes. 122 patients’ records were retrieved from the server and analysed for quality. It was found that ODK Scan recognized 99.2% of multiple choice bubble responses and 79.4% of numerical digit responses correctly. However, the overall quality of the digital data was decreased in comparison to manually entered data. Using ODK Scan, a significant time reduction is observed in data aggregation and data transfer activities, however, data verification and form filling activities took more time. Interviews revealed that field workers saw value in using ODK Scan, however, they were more concerned about the time consuming aspects of the use of ODK Scan. Therefore, it is concluded that minimal disturbance in the existing workflow, continuous feedback and value additions are the important considerations for the implementing organization to ensure technology adoption and workflow improvements.

Accurate treatment planning in radiotherapy essentially decreases secondary damage to healthy tissue surrounding the tumor. Due to plans to use a direct, highly collimated, narrow beam with high intensity to treat small area tumors, researchers have studied microbeam radiation therapy extensively. Using a synchrotron beam as the radiation source may help to limit secondary damage, but treatment planning using computerized simulations and dosimetry is still necessary to achieve optimal results. For this purpose, PDA-gel dosimeters were developed and their sensitivity around a 150 keV induced synchrotron X-ray radiation beam was examined via Monte Carlo simulations using the EGS5 code system. The microbeam development is now at the animal study stage. In this study, we simulate the irradiation of a rat's brain. The simulation results obtained spectra for two types of PDA-gel dosimeters that were compared with the spectrum obtained in a modelized brain tumor of a rat. Additionally, percentage depth dose curves were calculated for the brain tissue and the two gels. Correction equations for the dosimeters were obtained from the dose-difference plots. For further references, these equations can be used to calculate the actual dose in a brain tumor in a rat. The Monte Carlo simulations demonstrate that PDA-gel dosimeters can be used as a reference technique in planning treatment using synchrotron irradiations and in other medical treatment planning methods.

Breaching and liquefaction are two types of coastal flow slides proposed in the literature. While their final results are similar, liquefaction and breaching failure are two different processes, with breaching failure characterized by a different progression and sand movement mechanisms. In densely packed sand, breaching causes delayed sand grain discharge and negative excess pore pressures. Liquefaction, when a mass of soil suddenly behaves like a liquid, flow out over overly gentle slopes. Flow slides typically go unnoticed until the slope fails above ground because they start below the water. Without modern technology, diving equipment, risk assessment, and probabilistic and sensitivity assessments, flow slides inherently challenging. The authors developed a new sensitivity index to detect the susceptibility of coastal flow slides. It was established a sophisticated hybrid model that accommodates flow slides in sync with unpredictable variables employed in this new sensitivity index. Innovative hybrid model has three distinctive models. The Hybrid Hydrodynamic Model addresses wave, wind, current, climate change, and sediment transfer. The Monte Carlo Simulation analyses sensitivity, and the Bayesian Network calculates joint probability of coastal flow slide parameters of this new index that includes environmental characteristics, including climate change. With the assistance of these three models, researchers aim at a) presenting a methodology for coupling coastal flow slide projections with outcomes; b) distinguishing two flow slides; c) examining variables influencing flow slides. The use of such a hybrid model and risk index offers a robust and computationally efficient approach to evaluating the critical angle slope for breaching failure.

We report the results of the characterization of local Monte Carlo (MC) dynamics of an equilibrium bond fluctuation model polymer matrix (BFM), in time interval typical for MC simulations of non-linear optical phenomena in host-guest systems. The study contributes to the physical picture of the dynamical aspects of quasi-binary mosaic states characterized previously in the static regime. The polymer dynamics was studied at three temperatures (below, above and close to the glass transition), using time-dependent generalization of the static parameters which characterize local free volume and local mobility of the matrix. Those parameters play the central role in the kinetic MC model of host-guest systems. The analysis was done in terms of the probability distributions of instantaneous and time-averaged local parameters. The main result is the characterization of time scales characteristic of various local structural processes. Slowing down effects close to the glass transition are clearly marked. The approach yields an elegant geometric criterion for the glass transition temperature. A simplified quantitative physical picture of the dynamics of guest molecules dispersed in BFM matrix at low temperatures offers a starting point for stochastic modeling of host-guest systems.

The application of space robotic manipulators and heightened autonomy for ios represents a paramount pursuit for leading space agencies, given the substantial threat posed by space debris to operational satellites and forthcoming space endeavors. This work presents a guidance algorithm based on drl to solve for the space manipulator path-planning during the motion synchronization phase with the mission target. The goal is the trajectory generation and control of a spacecraft equipped with a 7-dof robotic manipulator, such that its end effector remains stationary with respect to the target point of capture. The ppo drl algorithm is used to optimize the manipulator’s guidance law, and the autonomous agent generates the desired joints rates of the robotic arm, which are then integrated and passed to a model-based feedback linearization controller. The agent is first trained to optimize its guidance policy and then tested extensively to validate the results against a simulated environment representing the motion synchronization scenario of an ios mission.

The aim of this study is to present a method for developing fragility curves for soil liquefaction that are consistent with the seismic hazard using Monte Carlo simulation. This method can account for all the uncertainties and variabilities of the input parameters. The seismic parameters of earthquake magnitude (M) and the associated peak ground acceleration (PGA) are considered together for the liquefaction assessment. The liquefaction potential and damages obtained by this method are more realistic. A case study is performed using data from a sand-boil site in Yuanlin, Changhua county, where liquefaction occurred during the Chi-Chi earthquake in Taiwan in 1999. The results show that the liquefaction potential index, IL, the post-liquefaction settlement, St, and the liquefaction probability index, PW, are all suitable parameters for evaluating the liquefaction damages. The fragility curves for soil liquefaction developed by this method can support the performance-based earthquake engineering (PBEE) approach, provide guidance for liquefaction evaluation to the ‘Taiwan Earthquake Loss Estimation System’-TELES, and serve as a basis for scenario simulation and an earthquake early warning system for liquefaction damages.

In this paper, we study the black hole solution of self-similar gravitational collapse in the Einstein-axion-dilaton system for the elliptic class in four dimensions. The solution is invariant under space-time dilation, which is combined with internal SL(2,R) transformations. Due to the complex and highly nonlinear pattern of the equations of motion in the physics of black holes, researchers typically have to use various numerical techniques to make the equations tractable to estimate the parameters and the critical solutions. To this end, they have to ignore the numerical measurement errors in estimating the parameters. To our knowledge, for the first time in the literature on axion-dilation systems, we propose to estimate the critical collapse functions in a Bayesian framework. We develop a novel methodology to translate the modelling of the complexity of the elliptic black hole to a sampling problem using Hamiltonian Monte Carlo with stacked neural networks. Unlike methods in the literature, this probabilistic approach enables us not only to recover the available deterministic solution but also to explore possibly all physically distinguishable self-similar solutions that may occur due to numerical measurement errors.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

Nowadays, the uses of laser and optics in the medical areas are extremely vivid, especially low-level laser therapy. The light with the wavelength of 633 nm to 1200 nm could penetrate and propagate deep in biological tissue. To develop the low-level laser therapy device, optimizing light delivery is critical to accurately stimulate the biological effects inside the biological tissue. Nevertheless, each form of the tissues at each zone on the body had various refractive optic, absorption, scattering, and anisotropy coefficients. This paper describes the simulation results of low-level laser propagation from skin surface at the lower spine, the knee, the femur and the prostate gland with four wavelengths (633 nm, 780 nm, 850 nm, and 940 nm) by the Monte Carlo method. These simulation results are the base for developing the low-level laser therapy device, that could be used in clinical for treating the fracture, knee osteoarthritis, spinal degeneration, and benign prostatic hypertrophy.

COVID-19 transmission models have conferred great value in informing public health understanding, planning, and response. However, the pandemic also demonstrated the infeasibility of basing public health decision-making on transmission models with pre-set assumptions. No matter how favourably evidenced when built, a model with fixed assumptions is challenged by numerous factors that are difficult to predict. Ongoing planning associated with rolling back and re-instituting measures, initiating surge planning, and issuing public health advisories can benefit from approaches that allow state estimates for transmission models to be continuously updated in light of unfolding time series. A model being continuously regrounded by empirical data in this way can provide a consistent, integrated depiction of the evolving underlying epidemiology and acute care demand, offer the ability to project forward such a depiction in a fashion suitable for triggering the deployment of acute care surge capacity or public health measures, support quantative evaluation of tradeoffs associated with prospective interventions in light of the latest estimates of the underlying epidemiology. We describe here the design, implementation and multi-year daily use for public health and clinical support decision-making of a particle filtered COVID-19 compartmental model, which served Canadian federal and provincial governments via regular reporting starting in June 2020. The use of the Bayesian Sequential Monte Carlo algorithm of Particle Filtering allows the model to be re-grounded daily and adapt to new trends within daily incoming data – including test volumes and positivity rates, endogenous and travel-related cases, hospital census and admissions flows, daily counts dose-specific vaccinations administered, measured concentration of SARS-CoV-2 in wastewater, and mortality. Important model outputs include estimates (via sampling) of the count of undiagnosed infectives, the count of individuals at different stages of the natural history of frankly and pauci-symptomatic infection, the current force of infection, effective reproductive number, and current and cumulative infection prevalence. Following a brief description of model design, we describe how the machine learning algorithm of particle filtering is used to continually reground estimates of dynamic model state, support probabilistic model projection of epidemiology and health system capacity utilization and service demand and probabilistically evaluate trade-offs between potential intervention scenarios. We further note aspects of model use in practice as an effective reporting tool in a manner that is parameterized by jurisdiction, including support of a scripting pipeline that permits a fully automated reporting pipeline other than security-restricted new data retrieval, including automated model deployment, data validity checks, and automatic post-scenario scripting and reporting. As demonstrated by this multi-year deployment of Bayesian machine learning algorithm of particle filtering to provide industrial-strength reporting to inform public health decision making across Canada, such methods offer strong support for evidence-based public health decision making informed by ever-current articulated transmission models whose probabilistic state and parameter estimates are continually regrounded by diverse data streams.

X-ray fluorescence is largely employed in the measurement of the thickness of coatings. Despite of its diffusion, the task is not straightforward because of the complex physics involved that results in high dependence on matrix effects. Thickness quantification is in practice accomplished using the Fundamental Parameters approach, adjusted with empirical measurements of standards with known composition and thickness. This approach has two major drawbacks: i) there are no standards for any possible coating and coating architecture and ii) even relying on standards, the quantification of unknown samples requires the precise knowledge of the matrix nature (e.g., in case of multilayer coatings the thickness and the composition of each underlayer). In this work, we describe a semiquantitative approach to coatings thickness measurement based on the construction of calibration curves through simulated XRF spectra built with Monte Carlo simulations. Simulations have been performed with the freeware software XMI-MSIM. We have assessed the accuracy of the methods by comparing the results with those obtained by i) XRF thickness determination with standards and ii) FIB-SEM cross-sectioning. Then we evaluated which parameters are critical in this kind of indirect thickness measurements.

This work analyzes the severity and risk associated with Automatic Dependent Surveillance-Broadcast (ADS-B) message drop out in Detect and Avoid (DAA) function of Unmanned Aircraft System (UAS). Performance assessment of the Universal Access Transceiver (UAT) ADS-B message implies that in some cases ADS-B fails to update within a specified update interval, which is referred to as drop out in this work. ADS-B is a fundamental surveillance sensor for both class 1 and class 2 DAA systems. Message loss or drop out has been found as one of the common limitations of the ADS-B system. The key feature of this study is incorporating the update rate of real ADS-B data transmitted from the manned aircraft. The data were received from the Grand Forks International Airport, North Dakota. Monte Carlo method has been adopted to resolve encounter scenarios in the presence of drop out. The change in the alert triggered by the UAS DAA in the presence of ADS-B drop out has been investigated. Furthermore, the risk matrices are created to quantify the associated risk with drop out affected alerts. Simulation results depict that both the duration of drop out and DAA look-ahead time affect the alert-triggering function of UAS. With a small look- ahead window and longer duration of drop out, the number of warning alerts increases. Also, alerts are affected more during an overtaking encounter than that of a head-to-head encounter. A system-level analysis is also carried out to recognize the potential reasons behind the ADS-B drop out.

In the past two decades, metaheuristic optimization algorithms (MOAs) have been increasingly popular, particularly in logistic, science, and engineering problems. The fundamental characteristics of such algorithms are that they are dependent on a parameter or a strategy. Some online and offline strategies are employed in order to obtain optimal configurations of the algorithms. Adaptive operator selection is one of them, and it determines whether or not to update a strategy from the strategy pool during the search process. In the filed of machine learning, Reinforcement Learning (RL) refers to goal-oriented algorithms, which learn from the environment how to achieve a goal. On MOAs, reinforcement learning has been utilised to control the operator selection process. Existing research, however, fails to show that learned information may be transferred from one problem-solving procedure to another. The primary goal of the proposed research is to determine the impact of transfer learning on RL and MOAs. As a test problem, a set union knapsack problem with 30 separate benchmark problem instances is used. The results are statistically compared in depth. The learning process, according to the findings, improved the convergence speed while significantly reducing the CPU time.

Flomoxef (FMX) may be a potential alternative to carbapenems for dogs infected with Enterobacterales producing extended-spectrum β-lactamase (ESBL-E). However, the appropriate dosage of FMX in dogs with ESBL-E infections has yet to be established. This study was carried out to establish appropriate treatment regimens for FMX against ESBL-E infections in dogs by a pharmacokinetics–pharmacodynamics (PK–PD) approach. Five dogs received an intravenous bolus dose of FMX (40 mg/kg). Serum FMX concentrations were evaluated using high-performance liquid chromatography-tandem mass spectrometry, and PK indices were determined based on non-compartmental model. The cumulative fraction of response (CFR) was calculated based on the distribution of minimum inhibitory concentrations among wild-type ESBL-E from companion animals. As the results, the dosage regimens of 40 mg/kg q8h and q6h were estimated to achieve a CFR of >90% for wild-type isolates of ESBL-producing Escherichia coli, Klebsiella pneumoniae, and Proteus mirabilis for dogs. In contrast, all regimens exhibited a CFR of <80% for ESBL-producing Enterobacter cloacae. Our results indicated that dosage regimens of 40 mg/kg FMX q8h and q6h can be a non-carbapenem treatment for dogs infected with ESBL-producing isolates of Escherichia coli, Klebsiella pneumoniae, and Proteus mirabilis, but not for those infected with ESBL-producing Enterobacter cloacae.

With the growth of air traffic demand in busy airspace, there is an urgent need for airspace sectorization to increase air traffic throughput and ease the pressure on controllers. The purpose of this paper is to develop a method framework that can perform airspace sectorization automatically, reasonably, which can be used as an advisory tool for controllers as an automatic system, especially for eliminating irregular sector shapes generated by simulated annealing algorithm (SAA) based on region growth method. Two graph cutting method, dynamic Monte Carlo method by changing location of flexible vertices (MC-CLFV) and Monte Carlo method by radius changing (MC-RC) were developed to eliminating irregular sector shapes generated by SAA in post-processing. The experimental results show that the proposed method framework of AS can automatically and reasonably generate sector design schemes that meet the design criteria. Our methodology framework and software can provide assistant design and analysis tools for airspace planners to design airspace, improve the reliability and efficiency of airspace design, and reduce the burden of airspace planners. In addition, this lays the foundation for reconstructing airspace with more intelligent method.

Background: The genetic variants identified by three large genome-wide association studies (GWAS) of educational attainment were used to test a polygenic selection model. Methods: Average frequencies of alleles with positive effect (polygenic scores or PS) were compared across populations (N=26) using data from 1000 Genomes. A null model was created using frequencies of random SNPs. Results: Polygenic selection signal of educational attainment GWAS hits is high among a handful of SNPs within genomic regions replicated across GWAS publications. A polygenic score comprising 9 SNPs predicts population IQ (r=0.88), outperforming 99% of the polygenic scores obtained from sets of random SNPs (Monte Carlo p= 0.011). Its predictive power remains unaffected after controlling for spatial autocorrelation (Beta= 0.83). The largest polygenic score (161 SNPs) exhibits similar predictive power (Beta=0.8). Random polygenic scores are moderate predictors of population IQ (thanks to spatial autocorrelation), and their predictive power increases logarithmically with the number of SNPs, indicating an exponential reduction in noise. Conclusion: This study provides guidance for using GWAS hits together with random SNPs for testing polygenic selection using Monte Carlo simulations.

Reinforcement learning is used for air combat problems in recent years, and the idea of curriculum learning is often used for reinforcement learning, but traditional curriculum learning suffers from the problem of plasticity loss in neural networks. Plasticity loss is the difficulty of learning new knowledge after the network has converged. To this end, we propose a motivational curriculum learning distributed proximal policy optimization (MCLDPPO) algorithm, through which agents trained can significantly outperform the predictive game tree and mainstream reinforcement learning methods. The motivational curriculum learning is to help the agent gradually improve their combat ability by observing the agent's unsatisfactory performance and providing corresponding rewards as a guide. Additionally, a complete tactical maneuver is encapsulated based on existing air combat knowledge, and through the flexible use of these maneuvers, some tactics beyond human knowledge can be realized. In addition, we designed an interruption mechanism for the agent to increase the frequency of decision-making when the agent faces an emergency. When the number of threats the agent receives changes, the current action will be interrupted to reacquire observations and make decisions again. Using the interruption mechanism can significantly improve the ability of the agent. To simulate actual air combat better, we built an air combat environment based on digital twin technology to create an air combat engine core such as missiles and radars. Based on this environment, we develop a distributed intelligent training system that employs Redis as the cache and obtains the data generated in the distributed environment through a multi-threaded TCP/IP protocol, which significantly increases the training data throughput and helps the agent accelerates convergence. The experimental results demonstrate that the agent can fully exploit the situational information to make reasonable decisions and affords air combat tactical adaption, verifying the effectiveness of the algorithmic framework proposed in this paper.

Modeling is essential to understand better the generative mechanisms responsible for experimental observations gathered from complex systems. In this work, we are using such an approach to analyze the electrocardiogram (ECG). We present a systematic framework to decompose ECG signals into sums of overlapping lognormal components. We used reinforcement learning to train a deep neural network to estimate the modeling parameters from ECG recorded in babies of 1 to 24 months of age. We demonstrate this model-driven approach by showing how the extracted parameters vary with age. After correction for multiple tests, 10 of 24 modeling parameters showed statistical significance below the 0.01 threshold, with absolute Kendall rank correlation coefficients in the [0.27, 0.51] range. We presented a model-driven approach to the analysis of ECG. The impact of this framework on fundamental science and clinical applications is likely to be increased by further refining the modeling of the physiological mechanisms generating the ECG. By improving the physiological interpretability, this approach can provide a window into latent variables important for understanding the heart-beating process and its control by the autonomous nervous system.

The collision avoidance mechanism adopted by the IEEE 802.11 standard is not optimal. The mechanism employs a binary exponential backoff (BEB) algorithm in the medium access control (MAC) layer. Such an algorithm increases the backoff interval whenever a collision is detected to minimize the probability of subsequent collisions. However, the expansion of the backoff interval causes degradation of the radio spectrum utilization (i.e., bandwidth wastage). That problem worsens when the network has to manage the channel access to a dense number of stations, leading to a dramatic decrease in network performance. Furthermore, a wrong backoff setting increases the probability of collisions such that the stations experience numerous collisions before achieving the optimal backoff value. Therefore, to mitigate bandwidth wastage and, consequently, maximize the network performance, this work proposes using reinforcement learning (RL) algorithms, namely Deep Q Learning (DQN) and Deep Deterministic Policy Gradient (DDPG), to tackle such an optimization problem. As for the simulations, the NS-3 network simulator is used along with a toolkit known as NS3-gym, which integrates a reinforcement-learning (RL) framework into NS-3. The results demonstrate that DQN and DDPG have much better performance than BEB for static and dynamic scenarios, regardless of the number of stations. Moreover, the performance difference is amplified as the number of stations increases, with DQN and DDPG showing a 27% increase in throughput with 50 stations compared to BEB. Furthermore, DQN and DDPG presented similar performances.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

Deep reinforcement learning (DRL) has achieved significant results in many Machine Learning (ML) benchmarks. In this short survey we provide an overview of DRL applied to trading on financial markets, including a short meta-analysis using Google Scholar, with an emphasis on using hierarchy for dividing the problem space as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market.

Continuous greenhouse gas emissions are causing global warming and impacting the habitats of many animals. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable is challenging. To address the issue, we propose a grid dispatching technique that combines prediction technology, reinforcement learning, and optimization technology. Prediction technology can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the power system, the aggregation of distributed energy resources increases uncertainty, particularly due to the fluctuating generation of renewable energy. This requires the use of advanced predictive control techniques to ensure long-term economic and decarbonization goals. In this paper, we present a real-time dispatching framework that integrates deep learning-based prediction, reinforcement learning-based decision-making, and stochastic optimization techniques. The framework can rapidly adapt to target uncertainty caused by various factors in real-time data distribution and control processes. The proposed framework achieved global Champion in the NeurIPS Challenge 2022 competition and demonstrated its effectiveness in practical scenarios of intelligent building energy management.

The prevalence of differential equations as a mathematical technique has refined the fields of control theory and constrained optimization due to the newfound ability to accurately model chaotic, unbalanced systems. However, in recent research, systems are increasingly more nonlinear and difficult to model using Differential Equations only. Thus, a newer technique is to use policy iteration and Reinforcement Learning, techniques that center around an action and reward sequence for a controller. Reinforcement Learning (RL) can be applied to control theory problems since a system can robustly apply RL in a dynamic environment such as the cartpole system (an inverted pendulum). This solution successfully avoids use of PID or other dynamics optimization systems, in favor of a more robust, reward-based control mechanism. This paper applies RL and Q-Learning to the classic cartpole problem, while also discussing the mathematical background and differential equations which are used to model the aforementioned system.

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

In the context of reinforcement learning (RL), ensuring both safety and performance is crucial, especially in real-world scenarios where mistakes can lead to severe consequences. This study aims to address this challenge by integrating temporal logic constraints into RL algorithms, thereby providing a formal mechanism for safety verification. We employ a combination of theoretical and empirical methods, including the use of temporal logic for formal verification and extensive simulations to validate our approach. Our results demonstrate that the proposed method not only maintains high levels of safety but also achieves comparable performance to traditional RL algorithms. Importantly, our approach fills a critical gap in existing literature by offering a solution that is both mathematically rigorous and empirically validated. The study concludes that the integration of temporal logic into RL offers a promising avenue for developing algorithms that are both safe and efficient. This work lays the foundation for future research aimed at generalizing this approach to various complex systems and applications.

In August 2020, the World Health Assembly launched a global initiative to eliminate cervical cancer by 2030, setting three primary targets. One key goal is to achieve a 70% screening coverage rate for cervical cancer, primarily relying on the precise analysis of Papanicolaou (Pap) or digital Pap smears. However, the responsibility of reviewing Pap smear samples to identify potentially cancerous cells primarily falls on pathologists, a task known to be exceptionally challenging and time-consuming. This paper proposes a solution to address the shortage of pathologists for cervical cancer screening. It leverages the OpenAI-GYM API to create a deep reinforcement learning environment utilizing liquid-based Pap smear images. By employing the Proximal Policy Optimization algorithm, autonomous agents navigate Pap smear images, identifying cells with the aid of rewards, penalties, and accumulated experiences. Furthermore, the use of a pre-trained convolutional neuronal network like Res-Net54 enhances the classification of detected cells based on their potential for malignancy. The ultimate goal of this study is to develop a highly efficient, automated Papanicolaou analysis system, ultimately reducing the need for human intervention in regions with limited pathologists.

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in video transmission across networks. Traditional adaptive bitrate (ABR) algorithms adjust the quality of video segments based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules within a complex environment, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel Deep Reinforcement Learning-based approach for streaming DASH, focusing on maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes the Quality of Experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate this approach through a simulation model that encompasses diverse wireless network environments and various video sequences. Additionally, we compare our proposed approach with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

For a mobile robot, navigation in a densely crowded space can be a challenging and sometimes impossible task, especially with traditional techniques. In this paper, we present a framework to train neural controllers for differential drive mobile robots which must safely navigate a crowded environment while trying to reach a target location. To learn the robot’s policy, we train a convolutional neural network using two reinforcement learning algorithms, Deep Q-Networks (DQN) and Asynchronous Advantage Actor Critic (A3C), and develop a training pipeline that allows to scale the process to several compute nodes. We show that the asynchronous training procedure in A3C can be leveraged to quickly train neural controllers and test them on a real robot in a crowded environment.

Ever-increasing need for improving the livability of a city and improve outcomes for its residents, over the last decade, the adoption of technology to develop urbanised societies around the world has given rise to the need for developing smart cities. The speed at which the world population is growing, the use of Internet of Things in smart cities have really advanced the quality of life. One significant area of concern within the smart city framework is waste management. If the waste within a city is not adequately managed, then it leads to issues in the health of the citizens. Additionally, the waste management has such a high impact on the environmental footprint, hence the need to have a smart way of managing waste is of critical importance. Through our research, we analyse the challenges of waste management within a city to understand the impact of the problem on to the citizens and overall city operations. We then investigate ways in which we can solve these problems using the emerging technologies, such as the Internet of Things, to collect valuable data of large volumes arriving at an astronomical rate, then apply multi-agent deep reinforcement learning algorithms to harness the power of big data to extract meaningful information and actionable insights. We ingest data generated by our Internet of Things into our algorithm for three main purposes including providing the notifications to an external system, for example, a map navigation engine out of the scope for this project but a future extension for route optimisation and waste vehicle tracking; extracting and reporting the actionable insights from the underlying data; and consuming the extracted data for predictive forecasting to draw out the unknown patterns of waste fill levels within various geographical locations and again send out triggers and notification to external systems for example a waste collection authority who can efficiently schedule the waste collection vehicles and optimise the route. To achieve the above mentioned outcomes, we propose a framework that is agnostic of the hardware that it connects to and can effectively interface with a wide variety of hardware keeping a level of abstraction in the architecture.

Sparse reward long horizon task is a major challenge for deep reinforcement learning algorithm. One of the key barriers is data-inefficiency. Even in the simulation environment, it usually takes weeks to training the agent. In this study, a data-efficiency training framework is proposed, where a curriculum learning is design for the agent in the simulation scenario. Different distributions of the initial state are set for the agent to get more informative reward during the whole training process. A fine-tuning of the parameters in the output layer of the neural network for value function is conduct to bridge the gap between sim-to-real. An experiment of UAV maneuver control is conducted in the proposed training framework to verify the method more efficient. We demonstrate that data-efficiency is different for the same data in different training stages.

Abstract: As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs at a larger scale. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

Recently, the application of deep reinforcement learning in recommender system is flourishing and stands out by overcoming drawbacks of traditional methods and achieving high recommendation quality. The dynamics, long-term returns and sparse data issues in recommender system have been effectively solved. But the application of deep reinforcement learning brings problems of interpretability, overfitting, complex reward function design, and user cold start. This paper proposed a tag-aware recommender system based on deep reinforcement learning without complex function design, taking advantage of tags to make up for the interpretability problems existing in recommender system. Our experiment is carried out on MovieLens dataset. The result shows that, DRL based recommender system is superior than traditional algorithms in minimum error and the application of tags has little effect on accuracy when making up for interpretability. In addition, DRL based recommender system has excellent performance on user cold start problems.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL) and Health Systems Simulations has significant potential, for both research into improving Deep RL performance and safety, and in operational practice. While individual toolkits exist for Deep RL and Health Systems Simulations, no framework to integrate the two has been established. Aim: Provide a framework for integrating Deep RL Networks with Health System Simulations, and to ensure this framework is compatible with Deep RL agents that have been developed and tested using OpenAI Gym. Methods: We developed our framework based on the OpenAI Gym framework, and demonstrate its use on a simple hospital bed capacity model. We built the Deep RL agents using PyTorch, and the Hospital Simulation using SimPy. Results: We demonstrate example models using a Double Deep Q Network or a Duelling Double Deep Q Network as the Deep RL agent. Conclusion: SimPy may be used to create Health System Simulations that are compatible with agents developed and tested on OpenAI Gym environments. GitHub repository of code: https://github.com/MichaelAllen1966/learninghospital

This paper explores the application of Random Matrix Theory (RMT) as a methodological enhancement for portfolio selection within financial markets. Traditional approaches to portfolio optimization often rely on historical estimates of correlation matrices, particularly susceptible to instability. To address this challenge, we combine a data preprocessing technique based on the Hilbert transformation of returns with RMT to refine the accuracy and robustness of correlation matrix estimation. By contrasting empirical correlations with those derived from random matrices, we uncover non-random properties and underlying relationships within financial data. We then utilize this methodology to construct the correlation network dependence structure used in portfolio optimization. The empirical analysis presented in this paper validates the effectiveness of RMT in enhancing portfolio diversification and risk management strategies. This research contributes by offering investors and portfolio managers with methodological insights to construct portfolios that are more stable, robust, and diversified. At the same time, it advances our comprehension of the intricate statistical principles underlying multivariate financial data.

Up to now, many studies have been performed on particle radiations before or during earthquakes (EQs). In our previous study, with the help of piezoelectricity relationships and the elastic energy formula, the MCNPX simulation code was applied to find the amount of created atomic/ nuclear particles, the dominant interactions; and the energy of the particles for various sizes of quartz and granite blocks. In this study, using the MCNPX simulation code, we have estimated the flux of the particles (created from under-stressed granitic rocks) at different distances from the EQ hypocenter inside the fractures, filled with air, water, and CO2. It was found that inside a water-filled fracture, the particles do not show the flux far from the EQ hypocenter, but inside the gases like air and CO2 with the normal condition density, different types of particles can have a flux far from the source (more than a kilometer) and they might reach themselves to the surface in the case that the EQ hypocenter is very shallow (0- 5 km). However, for deep EQs, it seems that the most detected nuclear particles on the surface should pass via the vacuum-filled fractures and reach themselves to the surface. Moreover, it was concluded that the more density of the fracture’s filling fluid, the less distance that the particles can have a flux.

In response to the issue of vehicle escape guidance, this manuscript proposes a unified intelligent control strategy synthesizing optimal guidance and Meta Deep Reinforcement Learning (DRL). Optimal control with minor energy consumption is introduced to meet terminal latitude, longitude and altitude. Maneuvering escape is realized by adding longitudinal and lateral direction maneuver overloads. Maneuver command decision model is calculated based on Soft-Actor-Critic (SAC) networks. Meta learning is introduced to enhance autonomous escape capability, which improves generalization performance to time-varying scenarios not encountered in the training process. In order to obtain training samples at a faster speed, this manuscript uses the prediction method to solve reward values, which avoiding a large number of numerical integration. The simulation results manifest that the proposed intelligent strategy can achieve high precise guidance and effective escape.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

Abstract—Online learning platforms provide diverse course resources, but this often result in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, Existing course recommendation methods face the challenge in accurately recommending the target course that is most relevant to the user, because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the Actor-Critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendation effectively. Furthermore, we propose a policy gradient based on temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and improves the accuracy of the recommendation. We evaluate the proposed model on two real datasets, and the experimental results show that the proposed model is significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.

Fast convergence routing is an important issue for LEO constellation network, due to its dynamical topology changing and time varying transmission requests. Most of existing research focus on the OSPF routing algorithm, which cannot handle the frequently links state changing of network. In this paper, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL-SR) for LEO satellite networks, in which the satellite gets the network links status fast and adjusts its routing strategy. In FRL-SR, each satellite node is regarded as an agent. The agent selects the port for packet forwarding according to its own routing policy. When the satellite network state changes, agent would send ’hello’ packets to the neighbor node to update the neighbor node’s routing policy. Compared with traditional reinforcement learning, FRL-SR can perceive network information faster, and then converge faster. Also, FRL-SR can mask the dynamics of satellite network topology and adaptively adjust the forwarding strategy according to the link state. Various simulation is constructed, the results show that the proposed FRL-SR algorithm out performance the Dijkstra algorithm in performance of average delay, packet arriving ratio, network load balance.

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

We consider the problem of coordinating the charging of an entire fleet of electric vehicles (EV), using a model-free approach, i.e. purely data-driven reinforcement learning (RL). The objective of the RL-based control is to optimize charging actions, while fulfilling all EV charging constraints (e.g. timely completion of the charging). In particular, we focus on batch-mode learning and adopt fitted Q-iteration (FQI). A core component in FQI is approximating the Q-function using a regression technique, from which the policy is derived. Recently, a dueling neural networks architecture was proposed and shown to lead to better policy evaluation in the presence of many similar-valued actions, as applied in a computer game context. The main research contributions of the current paper are that (i)we develop a dueling neural networks approach for the setting of joint coordination of an entire EV fleet, and (ii)we evaluate its performance and compare it to an all-knowing benchmark and an FQI approach using EXTRA trees regression technique, a popular approach currently discussed in EV related works. We present a case study where RL agents are trained with an epsilon-greedy approach for different objectives, (a)cost minimization, and (b)maximization of self-consumption of local renewable energy sources. Our results indicate that RL agents achieve significant cost reductions (70--80%) compared to a business-as-usual scenario without smart charging. Comparing the dueling neural networks regression to EXTRA trees indicates that for our case study's EV fleet parameters and training scenario, the EXTRA trees-based agents achieve higher performance in terms of both lower costs (or higher self-consumption) and stronger robustness, i.e. less variation among trained agents. This suggests that adopting dueling neural networks in this EV setting is not particularly beneficial as opposed to the Atari game context from where this idea originated.

As a new plant slope protection technology, living stumps can effectively improve slope stability. The current study attemptes to investigate the mechanical characteristics of the root system of living stumps and their reinforcement mechanism on slopes by carrying out a three-dimensional living stump slope modeling test. Using ABS materials, a 3D model of a living elm stump was created by 3D printing and a slope model test was carried out. The reinforcement mechanism of the living stump was studied through a combination of model testing and numerical simulation. The results indicate that, due to the presence of living stumps in the lower and middle parts of the slope, the maximum shear stress area of the soil moved to a deeper direction. The stress state of the soil around the living stump was effectively improved due to the existence of the lateral root system.Living stumps in the lower part of the slope cross the potential sliding surface and transferred the soil shear stress gradually to the root system through root-soil interaction. Moreover, the tap roots and lateral roots of living stumps form a robust spatial structure that bears the shear stress of soil together, thus increasing the stability of the slope.

With the proliferating of video surveillance system deployment and related applications, real-time video analysis is very critical to achieve intelligent monitoring, autonomous driving, etc. It is non-trivial to achieve high accuracy and low latency video stream analysis through the traditional cloud computing. In this paper, we propose a non-orthogonal multiple access (NOMA) based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the QoE of UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimension, which respectively select the offloading and resource allocation action, the resolution scaling action. Experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of transmission and computation, and have better performance in convergence compared to other baseline.

In this review we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.

Introduction: maxillary hypoplasia and mandibular asymmetry may be corrected with orthognathic surgery at the completion of growth. Many patients require segmental Le Fort I osteotomies to obtain the most stable results. Unfortunately, if Invisalign is chosen for pre-surgical preparation, ClinCheck © software does not allow for prediction when these complex surgeries are planned. The objective of this paper was to show the potential of aligners beyond its software in most complex cases like multiple-piece maxillary osteotomies in cleft and non-cleft patients. Method: Thirteen patients who underwent pre-surgical treatment with clear aligners-Invisalign (study group) were matched in terms of diagnosis, surgical procedure, and orthodontic complexity with thirteen patients treated using fixed appliances (control group). A technique based on simple superimposition was devised to enhance the software for creating a suitable occlusion for segmental pre-surgical treatments. Virtual curves following the lower arch were employed to guide the correct pre-surgical positions of the upper teeth. The amount of impressions required in both groups to achieve satisfactory pre-surgical alignment of the segmented arches was compared. Results: one or no refinement phase were needed in the Invisalign group to reach an acceptable pre-surgical occlusion while the amount of pre-surgical impressions taken in order to reach adequate coordination with fixed appliance treatment was slightly higher. The average overall treatment duration did not show a statistically significant variation between the groups. Conclusions: it appears that clear aligner could serve as an effective treatment for individuals with skeletal discrepancies, including cleft and craniofacial anomalies, necessitating segmental Le Fort I osteotomies when aided by the suggested superimposition approach during digital pre-surgical movement simulations. More sophisticated softwares will certainly be implemented in the future.

Sparse training (ST) aims to ameliorate deep learning by replacing fully connected artificial neural networks (ANNs) with sparse or ultra-sparse ones, such as brain networks are, therefore it might benefit to borrow brain-inspired learning paradigms from complex network intelligence theory. Here, we launch the ultra-sparse advantage challenge, whose goal is to offer evidence on the extent to which ultra-sparse (around 1% connection retained) topologies can achieve any leaning advantage against fully connected. Epitopological learning is a field of network science and complex network intelligence that studies how to implement learning on complex networks by changing the shape of their connectivity structure (epitopological plasticity). One way to implement Epitopological (epi- means new) Learning is via link prediction: predicting the likelihood of non-observed links to appear in the network. Cannistraci-Hebb learning theory inspired the CH3-L3 network automata rule for link prediction which is effective for general-purpose link prediction. Here, starting from CH3-L3 we propose Epitopological Sparse Meta-deep Learning (ESML) to apply Epitopological Learning to sparse training. In empirical experiments, we find that ESML learns ANNs with ultra-sparse hyperbolic (epi-)topology in which emerges a community layer organization that is meta-deep (meaning that each layer also has an internal depth due to power-law node hierarchy). Furthermore, we discover that ESML can in many cases automatically sparse the neurons during training (arriving even to 30% neurons left in hidden layers), this process of node dynamic removal is called percolation. Starting from this network science evidence, we design Cannistraci-Hebb training (CHT), a 4-step training methodology that puts ESML at its heart. We conduct experiments on 7 datasets and 5 network structures comparing CHT to dynamic sparse training SOTA algorithms and the fully connected counterparts. The results indicate that, with a mere 1% of links retained during training, CHT surpasses fully connected networks on VGG16, GoogLeNet, ResNet50, and ResNet152. This key finding is an evidence for ultra-sparse advantage and signs a milestone in deep learning. CHT acts akin to a gradient-free oracle that adopts CH3-L3-based epitopological learning to guide the placement of new links in the ultra-sparse network topology to facilitate sparse-weight gradient learning, and this in turn reduces the convergence time of ultra-sparse training. Finally, CHT offers the first examples of parsimony dynamic sparse training because, in many datasets, it can retain network performance by percolating and significantly reducing the node network size. Our code is available at: https://github.com/biomedical-cybernetics/Cannistraci-Hebb-training.

In this study, a comprehensive review on production scheduling based on reinforcement learning (RL) techniques using bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of all pertinent publications contained in the Web of Science database. The study focuses on a wide spectrum of publications spanning the years between 1998 and 2023. The findings and recommendations of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.

We propose a deep reinforcement learning based manipulator path tracking method to solve the computationally difficult and non-unique problem of manipulator path tracking methods based on inverse kinematics. By transforming the path tracking task into a sequence decision problem, our method adopts an end-to-end learning method for closed-loop control and avoids the process of finding the inverse solution. We first explored the feasibility of the deep reinforcement learning method in the path tracking of the manipulator. After verifying the feasibility, the path tracking of the multi-degree-of-freedom(multi-DOF) manipulator was realized by combining the maximum entropy deep reinforcement learning algorithm. The experimental results show that our method has a good effect on the path tracking of the manipulator, which not only avoids the process of finding the inverse kinematics solution, but also requires no dynamic model. Therefore, we believe that our method has great significance in the study of manipulator path tracking.

In tunnel excavation with boring machines, the tunnel face is supported to avoid collapse and minimise settlement. This article proposes the use of reinforcement learning, specifically the Deep Q-Network algorithm, to predict the face support pressure. The approach is tested both analytically and numerically. By using the soil properties ahead of the tunnel face and the overburden depth as the input, the algorithm is capable of predicting the optimal tunnel face support pressure, adapting to changes in geological and geometrical conditions.

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.

Wireless energy harvesting (EH) communication has long been considered a sustainable networking solution. However, it has been limited in efficiency, which has been a major obstacle. Recently, strategies such as energy relaying and borrowing have been explored to overcome these difficulties and provide long-range wireless sensor connectivity. In this article, we examine the reliability of the wireless-powered communication network by maximizing the net bit rate. To accomplish our goal, we focus on enhancing the performance of hybrid access points and information sources by optimizing their transmit power. Additionally, we aim to maximize the use of harvested energy by energy-harvesting relays for both information transmission and energy relaying. However, this optimization problem is complex as it involves non-convex variables and requires combinatorial relay selection indicators optimization for decode and forward (DF) relaying. To simplify this problem, we utilize the Markov decision process and deep reinforcement learning framework based on the deep deterministic policy gradient algorithm. This approach enables us to tackle this non-tractable problem, which conventional convex optimisation techniques would be difficult to solve in complex problem environments. The proposed algorithm significantly improves the end-to-end net bit rate of the smart energy borrowing and relaying EH system by 13.22%,27.57%, and 14.12% compared to the benchmark algorithm based on borrowing energy with an adaptive reward for Quadrature Phase Shift Keying, 8-PSK, and 16-Quadrature amplitude modulation schemes, respectively.

This article focuses on developing a continuous authentication system using behavioral biometrics to recognize users accessing computing devices. The user’s distinct behavioral biometric is captured through keystroke dynamics, and reward-based reinforcement learning (RL) ideas are applied to recognize them throughout the session. The suggested system adds an extra layer of security to traditional authentication methods, forming a robust continuous authentication system that can be added to static authentication systems. The methodology involves training a RL model to detect unusual user typing patterns and flag suspicious activity. Each user has an agent trained on their historical data, which is preprocessed and used to create episodes for the agent to learn from. The environment involves fetching observations and randomly corrupting them to learn out-of-order behavior. The observation vector includes both running features and summary features. The re-ward function is binary and minimalistic. The Principal Component Analysis (PCA) model is used to encode the running features, and the Double Deep Q-Network (DDQN) algorithm with a fully connected neural network is used as the policy net. The evaluation achieved an average training accuracy and EER (equal error rate) of 94.7% and 0.0126 and test accuracy and ERR of 81.06% and 0.0323 for all users when the number of encoder features was increased. Therefore, it is concluded that by continuously learning and adapting to changing behavior patterns, this approach can provide more secure and personalized authentication, lowering the possibility of unauthorized access and cyberattacks. Overall, the use of reinforcement learning and behavioral biometrics for continuous authentication has the potential to significantly enhance security in the digital age and are effective in identifying each user.

To identify objects in images, a complex set of skills is needed that includes understanding the context and being able to determine the borders of objects. In computer vision, this task is known as semantic segmentation and it involves categorizing each pixel in an image. It is crucial in many real-world situations: for autonomous vehicles, it enables the identification of objects in the surrounding area; in medical diagnosis, it enhances the ability to detect dangerous pathologies early, thereby reducing the risk of serious consequences. In this study, we compare the performance of various ensembles of convolutional and transformer neural networks. Ensembles can be created, e.g, by varying the loss function, data augmentation method or the learning rate strategy. Our proposed ensemble, which is based on the simple average rule, demonstrates exceptional performance on several datasets. All the resources used in this study are available online at the following GitHub repository: https://github.com/LorisNanni.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

In this paper, we study the performance of various deep reinforcement learning algorithms to enhance the energy management system of a microgrid. We propose a novel microgrid model that consists of a wind turbine generator, an energy storage system, a set of thermostatically controlled loads, a set of price-responsive loads, and a connection to the main grid. The proposed energy management system is designed to coordinate among the different flexible sources by defining the priority resources, direct demand control signals, and electricity prices. Seven deep reinforcement learning algorithms were implemented and are empirically compared in this paper. The numerical results show that the deep reinforcement learning algorithms differ widely in their ability to converge to optimal policies. By adding an experience replay and a semi-deterministic training phase to the well-known asynchronous advantage actor-critic algorithm, we achieved the highest model performance as well as convergence to near-optimal policies.

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.

The presented results of simulations take into account the optical parameters of the selected sea region (from literature data on the southern Baltic Sea) and two optically extreme types of crude oil (from historical data) which exist in the form of a highly diluted oil-in-water emulsion (10 ppm). The spectral index was analyzed based on the results of modelling the radiance reflectance distribution for almost an entire hemisphere of the sky (zenith angle from 0 to 80o). The spectral index was selected and is universal for all optically different types of oil (wavelengths 650 and 412 nm). The possibility of detecting pollution in the conditions of the wavy sea surface (as a result of wind of up to 10 m/s) was studied. It has been also shown that if the viewing direction is close to a direction perpendicular to the sea surface, observations aimed at determining the spectral index are less effective than observation under the zenith angle of incidence of sunlight for all azimuths excluding the direction of sunlight specular reflection.

The robust control of high precision electromechanical systems, such as micropositioners, is challenging in terms of the inherent high nonlinearity, the sensitivity to external interference, and the complexity of accurate identification of the model parameters. To cope with these problems, this work investigates a disturbance observer-based deep reinforcement learning control strategy to realize high robustness and precise tracking performance. Reinforcement learning has shown great potential as optimal control scheme, however, its application in micropositioning systems is still rare. Therefore, embedded with the integral differential compensator (ID), deep deterministic policy gradient (DDPG) is utilized in this work with the ability to not only decrease the state error but also improves the transient response speed. In addition, an adaptive sliding mode disturbance observer (ASMDO) is proposed to further eliminate the collective effect caused by the lumped disturbances. The sterling performance is revealed with intensive tracking simulation experiments and demonstrates the improvement in the accuracy and response time of the controller.

The fifth Generation (5G) mobile networks use millimeter Waves (mmWaves) to offer giga bit data rates. However, unlike microwaves, mmWave links are prone to user and topographic dynamics. They easily get blocked and end up forming irregular cell patterns for 5G. This in turn cause too early, too late, or wrong handoffs (HOs). To mitigate HO challenges, sustain connectivity and avert unnecessary HO, we propose a HO scheme based on Jump Markov Linear System (JMLS) and Deep Reinforcement Learning (DRL). JMLS is widely known to account for abrupt changes in system dynamics. DRL likewise emerges as an artificial intelligence technique for learning highly dimensional and time-varying behaviors. We combine the two techniques to account for time-varying, abrupt, and irregular changes in mmWave link behaviour by predicting likely deterioration patterns of target links. The prediction is optimized by meta training techniques that also reduces training sample size. Thus, the JMLS-DRL platform formulates intelligent and versatile HO policies for 5G. Results show our proposed prediction scheme about target link behavior post HO to be highly reliable. The scheme also averts unnecessary HOs thus ably supports longer dew time.

Precision medicine emphasizes fine-grained diagnostics, taking individual variability into account to enhance treatment effectiveness. Parkinson's Disease (PD) heterogeneity among individuals is a proof that disease subtypes exist, and assigning individuals to subgroups is necessary for a better understanding of disease mechanisms and designing precise treatment approaches. The purpose of this study was to identify PD subtypes using RNA-Seq data in a combined pipeline including unsupervised machine learning, bioinformatics, and network analysis. 210 post mortem brain RNA-Seq samples from PD (n = 115) and Normal Controls (NC, n = 95) were obtained with a systematic data retrieval following PRISMA statements and a fully data-driven clustering pipeline was performed to identify PD subtypes. Bioinformatics and Network analyses were performed to characterize the disease mechanisms of the identified PD subtypes and to identify target genes for drug repurposing. Two PD clusters were identified and 42 DEGs were found (p.adjusted ≤ 0.01). PD clusters had significantly different gene network structures (p < 0.0001) and phenotype-specific disease mechanisms, highlighting the differential involvement of the Wnt/β-catenin pathway regulating adult neurogenesis. NEUROD1 was identified as a key regulator of gene networks and ISX9 and PD98059 were identified as NEUROD1-interacting compounds with disease-modifying potential, reducing the effects of dopaminergic neurodegeneration. This hybrid data analysis approach could enable precision medicine applications by providing insights for the identification and characterization of pathological subtypes. This workflow has proven useful on PD brain RNA-Seq, but its application to other neurodegenerative diseases is encouraged.

Data collection and analysis are becoming more and more important in a variety of application domains as long as the novel technologies advance. At the same time, we are experiencing a growing need for human-machine interaction with expert systems pushing research through new knowledge representation models and interaction paradigms. In particular, in the last years eHealth - that indicates all the health-care practices supported by electronic elaboration and remote communications - calls for the availability of smart environment and big computational resources. The aim of this paper is to introduce the HOLMeS (Health On-Line Medical Suggestions) framework. The introduced system proposes to change the eHealth paradigm where a trained machine learning algorithm, deployed on a cluster-computing environment, provides medical suggestion via both chat-bot and web-app modules. The chat-bot, based on deep learning approaches, is able to overcome the limitation of biased interaction between users and software, exhibiting a human-like behavior. Results demonstrate the effectiveness of the machine learning algorithms showing 74.65% of Area Under ROC Curve (AUC) when first-level features are used to assess the occurrence of different prevention pathways. When disease-specific features are added, HOLMeS shows 86.78% of AUC achieving a more specific prevention pathway evaluation.

The paper proposes, through an experimental method, a conceptualisation of theoretical-methodological and practical guidelines for the project on the system of collective open spaces, residual areas, abandoned areas that characterise the contemporary city, both in the urban contexts of the historical city, and in the more marginal areas of the modern suburbs. This methodology was experimented on an historical neighbourhood in Rome (San Lorenzo discrict) starting from the results of some researches conducted in collaboration between Sapienza University of Rome and the Escuela de Architettura, Universidad San Jorge and some design experiments conducted by Gràvalosdimonte design studio. This methodology and experimentation combines the theme of the inter-scalarity of the project (territorial, urban and local scale) with the themes of the flexibility and proximity of functions with respect to local instances and contingencies caused by the pandemic situation and by the sustainability perspectives, with particular reference to the need for urban health, the dialectic between public and private space in the perspective of the notion of common good, the bottom-up project for the construction of proximity spaces and plurality in the decision-making process.

Postural deficits such as hyperlordosis (hollow back) or hyperkyphosis (hunchback) are relevant health issues. Diagnoses depend on the experience of the examiner and are therefore often subjective and prone to errors. Machine learning (ML) methods in combination with explainable ar-tificial intelligence (XAI) tools have proven useful for providing an objective, data-based orien-tation. However, only a few works have considered posture parameters, leaving the potential of more human-friendly XAI interpretations still untouched. Therefore, the present work proposes an objective, data-driven ML system for medical decision support that enables especially human-friendly interpretations using counterfactual explanations (CFs). Posture data for 1151 subjects were recorded by means of stereophotogrammetry. An expert-based classification of the subjects regarding the presence of hyperlordosis or hyperkyphosis was initially performed. Using a Gaussian progress classifier, the models were trained and interpreted using CFs. Label errors were flagged and re-evaluated using confident learning. Very good classification performances for both hyperlordosis and hyperkyphosis were found, whereby the re-evaluation and correction of the test labels led to a significant improvement (MPRAUC = 0.97). A statistical evaluation showed that the CFs seemed to be plausible in general. In the context of personalized medicine, the present study’s approach could be of importance for reducing diagnostic errors and thereby improving the individual adaptation of therapeutic measures. Likewise, it could be a basis for the development of apps for preventive posture assessment.

The transportation industry is rapidly transitioning from Internal Combustion Engine (ICE) based vehicles to Electric Vehicles (EVs) to promote clean energy. However, large-scale adoption of EVs can compromise the reliability of the power grids by introducing large uncertainty in the demand. Demand response with a controlled charge scheduling strategy for EVs can mitigate such issues. In this paper, a deep reinforcement learning- based charge scheduling strategy is developed for individual EVs by considering user’s dynamic driving behavior and charging preferences. The temporal dynamics of user’s anxiety about charging the EV battery is rigorously addressed. A dynamic weight allocation technique is applied to continuously tune user’s priority for charging and cost-saving with respect to charging duration. The sequential charging control problem is formulated as a Markov decision process, and an episodic approach to the deep deterministic policy gradient (DDPG) algorithm with target policy smoothing and delayed policy update techniques is applied to develop the optimal charge scheduling strategy. A real-world dataset that captures user’s driving behavior, such as arrival time, departure time, and charging duration, is utilized in this study. The extensive simulation results reveal the effectiveness of the proposed algorithm in minimizing energy cost while satisfying user’s charging requirements.

The IEEE 802.11ah standard is introduced to address the growing scale of Internet of Things (IoT) applications. To reduce contention and enhance energy efficiency within the system, the restricted access window (RAW) mechanism is introduced at the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, the RAW parameters need to be appropriately determined to achieve optimal network performance. In this paper, we optimize the configuration of RAW parameters in the IEEE 802.11ah-based IoT network. We analyze and propose an RAW parameters optimization problem with the objective of improving network throughput and formulate it as a Markov decision process. To cope with the complex and dynamic network conditions, we propose a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) algorithm to find the optimal RAW parameters. We construct network environments with periodic and random traffic in the NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. Simulation results reveal that using the PPO-based DRL algorithm, optimal RAW parameters can be obtained under different network conditions, thereby significantly improving network throughput.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1 and Part2, the author previously reported experimental evaluations on Genetic Algorithm and Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part3 of the series of comparative research. In this study, Differential Evolution is adopted as the third instance of major evolutionary algorithms. The experimental results show a statistically significant superiority of DE over both GA and ES (p < .01). In addition, DE exhibits its robustness to variations in hyperparameter configurations (the number of offsprings and generations). In the previous experiments, both ES and GA showed significant performance differences depending on the configurations, whereas in the experiment reported in this article, such differences are not detected for DE.

In recent years, the viability of employing multi-agent reinforcement learning technology for adaptive traffic signal control has been extensively validated. However, owing to restricted communication among agents and the partial observability of the traffic environment, the process of mapping road network states to actions encounters numerous challenges. To address this problem, this paper proposes a multi-agent deep reinforcement learning model with an emphasis on communication content (CMARL). The model decouples the complex relationships between multi-signal agents through centralized training and decentralized execution. Specifically, we first pass the traffic state through an improved deep neural network to achieve the extraction of high-dimensional semantic information and the learning of the communication matrix. Then the agents selectively interact with each other based on the learned communication matrix and generate the final state features. Finally, the features are inputted to the QMIX network to achieve the final action selection. We compare the CMARL model with 6 other baseline algorithms in real traffic networks. The results show that CMARL can significantly reduce vehicle congestion, and run stably in various scenarios.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. In this study, Genetic Algorithm is adopted as another instance of major evolutionary algorithms. Experimental results demonstrate that there was no statistically significant difference between the experimental performances of GA and ES, but the priority of generations and offsprings was different; GA performed better with a greater number of generations while ES performed better with a greater number of offsprings. Eight hidden units were the best among four variations (4, 8, 16 or 32 units), which aligns with previous study using ES.