The aim of this study was to investigate the physicochemical characterization and biological activities of chitosan (CH) based composite films with the incorporation of chestnut flower essential oil (CFEO). The composite films were prepared by casting method and characterized in terms of structural, morphological and mechanical properties via FT-IR, XRD, UV, SEM and TGA. The results proved the significantly increased water vapor permeability (WVP), heat resistance, antibacterial and antioxidant capabilities of CH-CFEO films. The incorporation of CH and CFEO enhanced the UV blocking, which made the film shield almost all UV light. Films with the tensile strength of 6.37 ± 0.41 MPa and elongation at break of 22.57 ± 0.35% were got with 6 mg mL-1 CFEO. Subsequently, the fruit preservation experiments also confirmed that the composite films could effectively extend the shelf life of bananas through their excellent water vapor and oxygen barrier. These desirable performances enable our newly developed composite films to be a remarkable packaging material to solve the fresh fruit preservation dilemma.

Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

The numerical implementation of the controllers allows the use of very complex algorithms with ease. However, in practice, due to its proven advantages, the PID controller (and its variants) is widely used in industrial control systems as well as in many other applications that require continuous control. Most of the methods for tuning the parameters of PID controllers are based on time-invariant linear models of the processes, which in practice can lead to poor performance of the control system. The paper presents an application of reinforcement learning algorithms in tuning of PID controllers for the control of some classes of continuous nonlinear systems. Tuning the parameters of PID controllers is done with the help of Twin Delayed Deep Deterministic Policy Gradients (TD3) algorithm which presents a series of advantages compared to other similar methods from Machine Learning dedicated to continuous state and action spaces. TD3 algorithm is an off-policy Actor-Critic based method and was used as it does not require a system model. The presented technique is applied for control of a biotechnological system which has a strongly nonlinear dynamic. The proposed tuning method is compared to the classical tuning methods of PID controllers. The performance of the tuning method based on the TD3 algorithm is demonstrated through simulation illustrating the effectiveness of the proposed methodology.

The modern supply chain (SC) is growing in terms of data, devices, users, and stakeholders, which introduced new security challenges and threats, especially with the reliance on centralized servers or cloud platforms. In addition, increased trust among system participants exposes the SC to a higher risk of vulnerabilities which require strong security measures. This article proposes a hybrid security framework for SC systems, BC-DRLzSC, that integrates Blockchain (BC) and Deep Reinforcement Learning (DRL) designed to operate in a zero trust (ZT) environment. In particular, we propose a decentralized BC-based approach integrated with smart contracts to manage system participant registration and authentication and to control access to system resources. BC-DRLzSC adopts a ZT architecture to reinforce SC security, which can be achieved with an advocate to verify each entity’s trustworthiness before granting or retaining access to system resources. Incorporating the ZT architecture, with BC and DRL, can potentially and significantly bolster SC system security. DRL is employed to develop a proactive attack detection model that continuously monitors the incoming traffic from authenticated nodes within the network and predicts any malicious actions. Finally, we evaluate the performance of our proposed DRL solution using the NSL-KDD dataset.

As a new plant slope protection technology, living stumps can effectively improve slope stability. The current study attemptes to investigate the mechanical characteristics of the root system of living stumps and their reinforcement mechanism on slopes by carrying out a three-dimensional living stump slope modeling test. Using ABS materials, a 3D model of a living elm stump was created by 3D printing and a slope model test was carried out. The reinforcement mechanism of the living stump was studied through a combination of model testing and numerical simulation. The results indicate that, due to the presence of living stumps in the lower and middle parts of the slope, the maximum shear stress area of the soil moved to a deeper direction. The stress state of the soil around the living stump was effectively improved due to the existence of the lateral root system.Living stumps in the lower part of the slope cross the potential sliding surface and transferred the soil shear stress gradually to the root system through root-soil interaction. Moreover, the tap roots and lateral roots of living stumps form a robust spatial structure that bears the shear stress of soil together, thus increasing the stability of the slope.

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered remarkable success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Although, typically, the communication protocol between agents is manually specified and not altered during training, recently, some papers have shown signs of trying to emerge a communication between agents on the one hand and, on the other hand, to understand what is exchanged between agents. So, there is a growing body of literature on this topic which includes qualitative and quantitative studies and the ones that apply mixed methods. This study presents the scoping review of the methodological strategies undertaken in a total of 16 research articles. The results present the critical appraisal of quantitative methods in terms of validity and reliability and for qualitative methods considering four trustworthiness factors. In the end, relevant insights are further explored with implications and reflections on how they can benefit one's research in the field.

The Active Distribution Networks (ADNs) are Multi-Input Multi-Output (MIMO) systems with coupled dynamics which cause interactions among the control loops of the Distributed Energy Resources (DERs). This undesired effect lead to performance degradation in the voltage control. To mitigate the effects of these unavoidable coupling, the present paper proposes a systematic design procedure based on the analysis of the interaction's sources. In details, each DER is equipped with a double loop PI to control the active and reactive powers output by the Voltage Source Converter (VSC) which connects the DER to the network's node. Furthermore, to guarantee Ancillary Services (ASs), the two loops are coupled by a simple mechanism of cooperation of the active power to voltage regulation realized by a filtered droop law. To achieve voltage regulation with reduced loop interactions, the PI parameters and the filter's pulse are designed according to a two sequential steps procedure based on the (Internal Model Control) IMC technique. Simulation studies are finally presented to demonstrate that the proposed design method achieves both reduction of the loop interaction and robust voltage control in presence of model parameter uncertainty in the MIMO model.

(1) Background: The rapid expansion of 4G and 5G networks has raised public concerns about potential health risks associated with electromagnetic radiation. This study addresses these fears by investigating electromagnetic measurements in crowded squares, where human exposure is heightened. Despite rigorous scientific methodologies showing generally low levels of radiation from these technologies, public apprehension persists; (2) Methods: We conducted 496 measurements in Larissa's major squares, using a portable field strength analyzer and GIS mapping. Squares were categorized by traffic and population density; (3) Results: Low-traffic squares exhibited exposure ratios consistently below ICNIRP limits, with median values of 0.0012 and 0.00173 in non-active and active mobile phone cases, respectively. In high-traffic squares, the median exposure ratio was 0.00335 in non-active mobile phone cases, remaining below recommended limits; (4) Conclusions: These findings indicate no significant risk of electromagnetic radiation in crowded squares in Larissa, emphasizing the importance of factual communication and dispelling public misconceptions.

The application of space robotic manipulators and heightened autonomy for ios represents a paramount pursuit for leading space agencies, given the substantial threat posed by space debris to operational satellites and forthcoming space endeavors. This work presents a guidance algorithm based on drl to solve for the space manipulator path-planning during the motion synchronization phase with the mission target. The goal is the trajectory generation and control of a spacecraft equipped with a 7-dof robotic manipulator, such that its end effector remains stationary with respect to the target point of capture. The ppo drl algorithm is used to optimize the manipulator’s guidance law, and the autonomous agent generates the desired joints rates of the robotic arm, which are then integrated and passed to a model-based feedback linearization controller. The agent is first trained to optimize its guidance policy and then tested extensively to validate the results against a simulated environment representing the motion synchronization scenario of an ios mission.

An active power dispatch method for a microgrid (MG) with multi-type loads, renewable energy sources (RESs) and distributed energy storage devices (DESDs) is the focus of this paper. The MG operates in a grid-connected model, and distributed power sources contribute to the service for load demands. The outputs of multiple DESDs are controlled to optimize the active power dispatch. Our goal with optimization is to reduce the economic cost under time-of-use (TOU) price, and to adjust the excessively high or low load rate of distributed transformers (DTs) caused by the peak-valley demand and load uncertainties. To simulate a practical environment, the stochastic characteristics of multi-type loads are formulated. The transition matrix of system state is provided. Then, a finite-horizon Markov decision process (FHMDP) model is established to describe the dispatch optimization problem. A learning-based technique is adopted to search the optimal joint control policy of multiple DESDs. Finally, simulation experiments are performed to validate the effectiveness of the proposed method, and the fuzzification analysis of the method is presented.

Thin people who undergo implant insertion into the chest wall face a high risk of implant exposure to the external environment. Five months after an 8-year-old boy underwent implantable cardioverter-defibrillator (ICD) implantation in a subcutaneous pocket in the left anterolateral chest wall to manage long QT syndrome, ICD replacement became necessary owing to exposure risk from distal and lateral thinning of the ICD pocket. Pocket rupture and exposure would increase the risk of infection; therefore, we performed ICD removal and primary pocket closure. Two weeks later, a new suprafascial pocket was created, acellular dermal matrix (ADM) was attached to the inner wall to prevent ICD protrusion, and a new ICD was inserted. One year postoperatively, the ADM was engrafted, and no complications were observed. A thin subcutaneous layer increases the risk of ICD implantation complications. Inner wall strengthening with ADM can help prevent pocket rupture.

This paper proposes an innovative mechatronic piezo-actuated module to control vibrations in modern machine tools. Vibrations represent one of the main issues that compromise seriously the quality of the workpiece. The active vibration control (AVC) device is composed by a host part integrated with sensors and actuators synchronized by a regulator, able to make a self-assessment and adjust to the environmental alteration. This study presents the mechatronic model based on the kinematic and dynamic analysis of the AVC device. To ensure a real time performance, a H2-LQG controller has been developed and validated by simulations involving machine tool, PZT actuator and controller models. The Hardware-in-the-loop (HIL) architecture is adopted to control and attenuate the vibrations. A set of experimental tests has been performed to validate the AVC module on a commercial machine tool. The feasibility of the real time vibration damping is demonstrated and the simulation accuracy is evaluated.

Sea cucumber peptides are small molecular peptides with biological function obtained from fresh sea cucumber as feedstock, which were hydrolyzed by protease, separated ,and purified. As the main product of sea cucumber refinement, it more accurately expresses the nutritional effect of sea cucumber and has a wide range of potential applications. It has been demonstrated that sea cucumber has a certain value-added effect on bone mineral density regulation, but the differences in bone mineral density regulation of extracts from different parts of sea cucumber and the techniques contributing to the efficient isolation of bone sea cucumber peptides have not been reported. Firstly, the aim of this study was to reveal the differences in osteogenic activities of enzymatic peptides from three major parts of sea cucumber: the body wall, the egg protein, and the intestinal protein. Then, the bone-enhancing sea cucumber peptides were isolated based on the characteristics of the dif-ferences in molecular weight distribution, hydrophilicity, and ionisation degree of sea cucumber, and to explore the possible mechanisms of their action. The findings indicated that sea cucumber peptides extracted from three distinct sites exhibited a dose-dependent enhancement of osteoblast proliferation and differentiation. Upon evaluating the three methods of sea cucumber peptide separation based on molecular weight distribution, hydrophilicity, and ionization, it was observed that the fraction isolated by hydrophobicity and

The pharmacological effects of the fruiting body of Ophiocordyceps sinensis (Os) exerted antioxidant, anti-virus, and immunomodulatory, whereas the anti-inflammatory effects and active components of Os mycelial submerged culture (OMSC) are not known. This study aimed to investigate the active compounds in the fermented liquid (FLOS), hot water (WEOS), and 50-95% (EEOS-50, EEOS-95) ethanol extracts of OMSC and their anti-inflammatory effects and potential mechanisms in LPS-stimulated microglial BV2 cells. The results demonstrated that all of the OMSC extracts could inhibit NO production in BV2 cells. EEOS-95 exhibited the strongest inhibitory effects (41 % inhibitory ability at 500 µg/mL), and its ergosterol, GABA, total phenolic, and total flavonoid contents were significantly higher than those of the other extracts (18.60, 18.60, 2.28, and 2.14 mg/g, p < 0.05, respectively). EEOS-95 also has a strong inhibitory ability against IL-6, IL-1β and TNF-α with IC50 of 617, 277 and 507 µg/mL, respectively, which is higher than that of 1 mM melatonin. The anti-inflammatory mechanism of EEOS-95 seems to be associated with up-regulation of PPAR-γ/Nrf-2/HO-1 antioxidant-related expression and down-regulation of NF-κB/COX-2/iNOS pro-inflammatory expression signaling. Therefore, we hypothesized that EEOS-95 can prevent neurodegenerative diseases, but the effect on the application in vivo should be further investigated.

This study presents the utilization of the Microwave InfraRed Thermography (MIRT) technique to identify and analyze the defects in the carbon-fiber-reinforced polymer (CFRP) composite rein-forcement of concrete specimens. At first, a set of the numerical models was created, comprising of the broadband pyramidal horn antenna and the analyzed specimen. The utilization the system operating at a power of 1000W in a continuous mode, operating at frequency of 2.45 GHz was analyzed. The specimen under examination comprised a compact concrete slab that was covered with an adhesive layer, thereafter, topped with a layer of CFRP. An air gap represented a defect at the interface between the concrete and CFRP within the adhesive layer. In the modelling stage the study investigated three separate scenarios: a sample with no defects, a sample with a defect located at the center, and a sample with a numerous additional random defects located at the rim of the CFRP matte, to analyze the effect of the natural reinforcement degradation in this area. The next phase of the study involved conducting experiments to confirm the results obtained from nu-merical modeling. In the experiments the concrete sample aged for 10 years with the defect in the center and naturally developed defects at the CFRP rim was used. The study employed numerical modeling to explore the phenomenon of microwave heating in complex structures. The aim was to assess the chosen antenna design and identify the most effective experimental setup. These con-clusions were subsequently confirmed through experimentation. The observations made during the heating process were particularly remarkable, since they deviated from earlier studies that solely conducted measurements of the sample post-heating phase. The findings demonstrate that MIRT has the capacity to be employed as a technique for detecting flaws in concrete structures reinforced with CFRP.

In contemporary construction, the prevalence of vibration serviceability issues in lightweight and slender structures has become increasingly common, owing to advancements in building materials and construction methods. While these structures often meet the criteria for ultimate limit states, they can still elicit complaints due to excessive vibrations induced by human activity. To address this challenge, the Integral Resonant Control (IRC) technique has emerged as a favored approach for actively damping vibrations in various systems. This study introduces a fresh perspective on implementing a multi-input multi-output (MIMO) IRC scheme for active vibration control (AVC) specifically tailored for pedestrian structures utilizing inertial mass actuators. Building upon a common framework and design methodology outlined in previous research, this work presents a novel application of MIMO IRC for AVC. The designed controller is rigorously tested and implemented on a laboratory floor structure to validate its efficacy.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

The objective of this study was to carry out the extraction and characterization of chitin and chitosan from the exoskeletons of the Farfantepenaeus californiensis species of shrimp, as well as its application in the formulation of polymeric films plasticized with glycerol used in the process of food packaging. The characterization of the extracted polymer was made by Fourier transform infrared spectroscopy (FTIR), Scanning electron microscopy (SEM), and physicochemical analysis of chitin and chitosan, resulting in a theoretical performance of 29.33% and 16.46%, respectively. In terms of quality, it was determined that the extracted chitosan had a molecular weight of 1.10 x105 g/mol and a deacetylation degree of 80.23%, which classifies it as a low-molecular weight chitosan, suitable for food packaging. It was shown that the extracted chitosan is fit for the production of the films by using the casting method, presenting a tension resistance of 1 MPa. The Antibacterial analysis was carried out using Staphylococcus aureus (S. aureus) and Escherichia coli (E. coli), bacterial models commonly found in food. The results showed an effective inhibition capability of the polymeric materials. Overall, the results for both chitosan and the films indicate a high potential for application in the food packaging sector.

Huperzine A is an herbal reversible inhibitor of Acetylcholinesterase (AChE). A molecular docking analysis on Huperzine A molecule has been carried out to understand its structure, conformational flexibility, intermolecular interaction and the binding affinity in the active site of AChE enzyme. Further, the charge density distribution of huperzine A molecule (lifted from the active site of AChE) was determined from the high level quantum chemical calculations coupled with charge density analysis. The binding affinity of Huperzine A towards AChE was calculated from the molecular docking; the lowest docked energy is -8.46 kcal/mol. In the active site, huperzine A molecule interacts with acyl binding pocket-Phe330 of AChE, that is, the bicyclo ring group of huperzine A forms an intermolecular interaction with the oxygen atom of main chain of the amino acid residue Phe330 at the distances 3.02 and 3.25 Å respectively. On the other hand, a gas phase study on huperzine A molecule also performed using HF and DFT (B3LYP) methods with the basis set 6-311G**. The molecular structure, conformation, and the charge density distribution of huperzine A molecule in the gas phase have determined using quantum chemical calculations and the charge density analysis. The comparative studies between the gas phase and the active site forms of huperzine A molecule, explicitly reveals the degree of conformational modification and the charge density redistribution of huperzine A when present in the active site. The dipole moment of the molecule in the active site is 6.85 D, which is slightly higher than its gas phase value (5.91 D). The electrostatic potential (ESP) surface of active site molecule clearly shows the strong electronegative and positive ESP regions of the molecule, which are the expected strong reactive locations of the molecule.

Obesity is related to the establishment of chronic inflammation and metabolic diseases, but it can be positively influenced by the regular practice of physical activity. The study aimed to compare the anthropometric, metabolic, and inflammatory parameters of physically active Military Police Officers (MPOs) with those who are less physically active. Sixty male MPOs, low activity (n=28) and physically active (n=32) participated. The following parameters were measured: plasma cytokine levels, C-reactive protein (CRP) levels, circulating glucose triglyceride (TAG) and high-density lipoprotein cholesterol (HDL-C) levels, and plasma glutamic oxalacetate transaminase (GOT), glutamic-pyruvate transaminase (GPT), and gamma-glutamyl transferase (GGT) activities. The physically active group presented lower body fat and reduced TAG and IL-8 levels compared to the low activity group. Moreover, a negative correlation between SPE and SBP, DBP, BPM was detected for the physically active group (p<0.05) but not in the low activity group. Furthermore, the physically active group's work time (WT) values were not correlated with the important metabolic markers SBP, DBP, BPM, GLU, TAG (p>0.05) but could be in the low activity group (p<0.05). These findings highlight the fundamental protective role of physical activity in controlling body composition, subclinical inflammation, and cardiovascular risk in MPOs.

Open source software such as OpenDSS has given a lot of help to distribution network researchers and educators. With high penetration of distributed renewable energy resources into distribution network, tradition distribution steady state analysis software such as OpenDSS is faced with difficulty in handling distributed generators. Three-phase distributed generators are often modeled in sequence frame while unbalanced distribution network are usually modeled in phase frame. So a load flow in sequence-phase coupled frame is proposed to handle models described in both frames. Voltage controlled DGs which are difficult to cope with in OpenDSS are handled in proposed program. The steady state analysis platform is programmed with open source Modelica language and the main aim of this paper is to introduce an open source platform for active distribution network steady analysis include load flow and short circuit analysis which can be easily adopted and improved by other educators and researchers.

During public health events, Active Mobility (walking and cycling) plays an important role in securing residents' travel. To enhance the resilience of urban transportation, this article integrates the travel environment elements for walking and cycling and constructs a travel environment for Active Mobility evaluation index system. This article uses survey data from the main urban area of Harbin in the context of the COVID-19 pandemic in May 2022, and combines the Gradient Boosting Decision Tree (GBDT) and Impact-Asymmetry Analysis (IAA) methods to analyze the satisfaction of the travel environment for Active Mobility. It proposes strategies for optimizing the travel environment for Active Mobility, ensuring residents' travel during public health events, guiding them to adopt Active Mobility, and improving urban transportation resilience. This research shows that Active Mobility can enhance urban transportation resilience. In addition, there are differences in resident demand for walking and cycling environments in the travel environment for Active Mobility. Residents are more concerned about the comfort of the walking environment, while they prioritize the convenience of the cycling environment.

Under the influence of various factors, the bronze ware unearthed by archaeology has various diseases of different degrees, and the mineralization problem is more prominent. The physical properties of mineralized bronzes have been destroyed, and the whole is loose and fragile, which is in urgent need of reinforcement and protection. Common reinforcement and protection materials mainly consider surface sealing, matrix bonding reinforcement and penetration reinforcement. Considering the low matrix strength and complex disease types of fragile bronzeware, it is difficult to achieve targeted protection with commonly used reinforcement and protection materials. In view of the problems such as poor permeability and easy discoloration of commonly used protection materials, this paper uses self-developed reinforcement and protection material AMC to compare its reinforcement and protection performance with B72, which has a high comprehensive evaluation on the performance of commonly used reinforcement and protection materials. The matrix strength was increased, and the appearance intervention degree was less than B72.

Originally developed to image the shadow region of the central black hole in Sagittarius A* and in the nearby galaxy M87, the Event Horizon Telescope (EHT) provides deep, very high angular resolution data on other AGN sources too. The challenges of working with EHT data have spurred the development of new image reconstruction algorithms. This work briefly reviews the status of the EHT and its utility for observing AGN sources, with emphasis on novel imaging techniques that offer the promise of better reconstructions at 1.3 mm and other wavelengths.

Active ageing is defined as the process of optimizing opportunities for physical, social and mental health to enable older people to take an active part in society without discrimination and to enjoy an independent and good quality of life. The World Health Organization assumed this as a process for increasing and maintaining an individual’s participation in activities to enhance his/her quality of life. In this survey, the authors addressed the following question: “Is assistive technology (AT) for mobility contributing to enhancement of lifelong capacity and performance?”. From June 2015 until February 2016, 96 community dwelling adults, AT users for mobility (powered wheelchairs, manual wheelchairs, lower limb prostheses, walkers, crutches and canes), aged 45-97, mean 67.02 +/- 14.24 years old, 56.3% female, were interviewed using the Psychosocial Impact of Assistive Devices Scale (P-PIADS), the Activities and Participation Profile related to Mobility (APPM) and demographics, clinical and questions about AT use and training. The participants’ profiles revealed moderate limitation and restrictions in participation, measured by the APPM (2.03). Most participants showed positive impact of AT; average scores obtained from the P-PIADS subscales were: Self-esteem 0.62, Competency 1.11 and Adaptability 1.10. P-PIADS total was 0.96, with the powered wheelchair users scoring the highest (1.53) and the walker users scoring the lowest (0.73). All subscales and P-PIADS total were positively correlated with the activities and participation profile. There was no relation between age and the psychosocial impact of AT or activities and participation profile. These results encourage the authors to follow these participants up for a lifelong intervention. To accomplish that aim, currently, the protocol is implemented at the AT prescribing centers in Coimbra, Portugal in order to assess the impact of AT on participation in society, one of the domains of the Active Ageing Index, a new analytical tool to help policy makers in developing policies for active and healthy ageing.

The prevalence of differential equations as a mathematical technique has refined the fields of control theory and constrained optimization due to the newfound ability to accurately model chaotic, unbalanced systems. However, in recent research, systems are increasingly more nonlinear and difficult to model using Differential Equations only. Thus, a newer technique is to use policy iteration and Reinforcement Learning, techniques that center around an action and reward sequence for a controller. Reinforcement Learning (RL) can be applied to control theory problems since a system can robustly apply RL in a dynamic environment such as the cartpole system (an inverted pendulum). This solution successfully avoids use of PID or other dynamics optimization systems, in favor of a more robust, reward-based control mechanism. This paper applies RL and Q-Learning to the classic cartpole problem, while also discussing the mathematical background and differential equations which are used to model the aforementioned system.

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

Currently, electromagnetic and radar methods are the most common non-destructive tests for non-destructive rebar location in structures.. Both methods have some advantages, disadvantages and limitations. This article is an attempt to describe possibilities of particular methods with emphasizing their advantages and limitations. The first stage of tests included results from tests conducted on a lintel beam made of autoclaved aerated concrete (AAC) and reinforced with small-diameter rebars and rebars placed close to each other. The second stage consisted in testing nine lightweight concrete specimens, each with three bars having a diameter of 10, 16 and 20 mm at three different space arrangements, to determine the effect of bar diameter and spacing on the measurement accuracy. The reinforcement of lintels and specimens was tested with two different electromagnetic scanners and a ground-penetrating radar (GPR) device. Received results from direct tests were compared with results from non-destructive tests (NDT). NDT tests conducted in such an element were found to perform the correct assessment of the concrete cover, and at the same time to give ambiguous results with reference to shape and number of rebars.

Recycled tire fibers (RTF) are currently one of the most abundant waste not being recovered due to several processing hurdles and the presence of high amounts of residual ground rubber particles (GR). Therefore, this study proposes a simple approach to separate most of rubber particles from fibers and to determine their rubber content using thermogravimetric analysis (TGA)/calcination. Afterwards, clean fiber (CF) and GR are used as fillers for recycled post-consumer low density polyethylene (rLDPE), and their effects on the physical properties are investigated. Accordingly, a series of composites with CF and GR is prepared at different filler concentrations (0-30%) via extrusion compounding before using compression molding and injection molding for comparison. In all cases, injection molding leads to higher strength and modulus, but lower elongation at break. The results show that incorporating 30 wt.% of CF into rLDPE yields a remarkable improvement in tensile strength (15%), tensile modulus (192%) and flexural modulus (142%). On the other hand, the incorporation of up to 30 wt.% of GR results in a reduction of both tensile strength and flexural modulus by 15%, confirming the critical role of the cleaning process for RTF in achieving the best results.

Reinforcement learning is used for air combat problems in recent years, and the idea of curriculum learning is often used for reinforcement learning, but traditional curriculum learning suffers from the problem of plasticity loss in neural networks. Plasticity loss is the difficulty of learning new knowledge after the network has converged. To this end, we propose a motivational curriculum learning distributed proximal policy optimization (MCLDPPO) algorithm, through which agents trained can significantly outperform the predictive game tree and mainstream reinforcement learning methods. The motivational curriculum learning is to help the agent gradually improve their combat ability by observing the agent's unsatisfactory performance and providing corresponding rewards as a guide. Additionally, a complete tactical maneuver is encapsulated based on existing air combat knowledge, and through the flexible use of these maneuvers, some tactics beyond human knowledge can be realized. In addition, we designed an interruption mechanism for the agent to increase the frequency of decision-making when the agent faces an emergency. When the number of threats the agent receives changes, the current action will be interrupted to reacquire observations and make decisions again. Using the interruption mechanism can significantly improve the ability of the agent. To simulate actual air combat better, we built an air combat environment based on digital twin technology to create an air combat engine core such as missiles and radars. Based on this environment, we develop a distributed intelligent training system that employs Redis as the cache and obtains the data generated in the distributed environment through a multi-threaded TCP/IP protocol, which significantly increases the training data throughput and helps the agent accelerates convergence. The experimental results demonstrate that the agent can fully exploit the situational information to make reasonable decisions and affords air combat tactical adaption, verifying the effectiveness of the algorithmic framework proposed in this paper.

Modeling is essential to understand better the generative mechanisms responsible for experimental observations gathered from complex systems. In this work, we are using such an approach to analyze the electrocardiogram (ECG). We present a systematic framework to decompose ECG signals into sums of overlapping lognormal components. We used reinforcement learning to train a deep neural network to estimate the modeling parameters from ECG recorded in babies of 1 to 24 months of age. We demonstrate this model-driven approach by showing how the extracted parameters vary with age. After correction for multiple tests, 10 of 24 modeling parameters showed statistical significance below the 0.01 threshold, with absolute Kendall rank correlation coefficients in the [0.27, 0.51] range. We presented a model-driven approach to the analysis of ECG. The impact of this framework on fundamental science and clinical applications is likely to be increased by further refining the modeling of the physiological mechanisms generating the ECG. By improving the physiological interpretability, this approach can provide a window into latent variables important for understanding the heart-beating process and its control by the autonomous nervous system.

The collision avoidance mechanism adopted by the IEEE 802.11 standard is not optimal. The mechanism employs a binary exponential backoff (BEB) algorithm in the medium access control (MAC) layer. Such an algorithm increases the backoff interval whenever a collision is detected to minimize the probability of subsequent collisions. However, the expansion of the backoff interval causes degradation of the radio spectrum utilization (i.e., bandwidth wastage). That problem worsens when the network has to manage the channel access to a dense number of stations, leading to a dramatic decrease in network performance. Furthermore, a wrong backoff setting increases the probability of collisions such that the stations experience numerous collisions before achieving the optimal backoff value. Therefore, to mitigate bandwidth wastage and, consequently, maximize the network performance, this work proposes using reinforcement learning (RL) algorithms, namely Deep Q Learning (DQN) and Deep Deterministic Policy Gradient (DDPG), to tackle such an optimization problem. As for the simulations, the NS-3 network simulator is used along with a toolkit known as NS3-gym, which integrates a reinforcement-learning (RL) framework into NS-3. The results demonstrate that DQN and DDPG have much better performance than BEB for static and dynamic scenarios, regardless of the number of stations. Moreover, the performance difference is amplified as the number of stations increases, with DQN and DDPG showing a 27% increase in throughput with 50 stations compared to BEB. Furthermore, DQN and DDPG presented similar performances.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

Deep reinforcement learning (DRL) has achieved significant results in many Machine Learning (ML) benchmarks. In this short survey we provide an overview of DRL applied to trading on financial markets, including a short meta-analysis using Google Scholar, with an emphasis on using hierarchy for dividing the problem space as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market.

The growing attention for plug-in electric vehicles, and the associated high-performance demands, have initiated a development trend towards highly efficient and compact on-board battery chargers. These isolated ac-dc converters are most commonly realized using two conversion stages, combining a non-isolated power factor correction (PFC) rectifier with an isolated dc-dc converter. This, however, involves two loss stages and a relatively high component count, limiting the achievable efficiency and power density and resulting in high costs. In this paper a single-stage converter approach is analyzed to realize a single-phase ac-dc converter, combining all functionalities into one conversion stage and thus enabling a cost-effective efficiency and power density increase. The converter topology consists of a quasi-lossless synchronous rectifier followed by an isolated dual active bridge (DAB) dc-dc converter, putting a small filter capacitor in between. To show the performance potential of this bidirectional, isolated ac-dc converter, a comprehensive design procedure and multi-objective optimization with respect to efficiency and power density is presented, using detailed loss and volume models. The models and procedures are verified by a 3.7 kW hardware demonstrator, interfacing a 400 V dc-bus with the single-phase 230 V, 50 Hz utility grid. Measurement results indicate a state-of-the-art efficiency of 96.1% and power density of 2.2 kW/dm³, confirming the competitiveness of the investigated single-stage DAB ac-dc converter.

In the last 20 years, an increasing number of studies have been reported on membrane active peptides, which exert their biological activity by interacting with the cell membrane either to disrupt it and lead to cell lysis or to translocate through it to deliver cargos into the cell and reach their target. These peptides are attractive alternatives to currently used pharmaceuticals. Antimicrobial peptides (AMPs) and peptides designed for drug and gene delivery currently in the drug pipeline suggest that these membrane active peptides will soon constitute a significant percentage of the drug market. Here, we focus on two most prominent classes of membrane active peptides; AMPs and cell-penetrating peptides (CPPs). AMPs are a group of membrane active peptides that disrupt the membrane integrity or inhibit the cellular functions of bacteria, virus and fungi. CPPs are another group of membrane active peptides that mainly function as cargo-carriers even though they may also show antimicrobial activity to some extent. Biophysical techniques to understand how they interact with the membrane have shed light on the peptide–membrane interaction at various levels of detail. Structural investigation of membrane active peptides in the presence of the membrane provides important clues on the effect of the membrane environment on peptide conformations. Advances in live imaging techniques have allowed examination of peptide action at a single cell or single molecule level. In addition to these experimental biophysical techniques, molecular dynamics simulations provided clues on the peptide-lipid interactions and dynamics of the cell entry process at atomic detail. In this review, we summarize the recent advances in experimental and computational investigation of membrane active peptides with particular emphasis on two amphipathic membrane active peptides, the AMP melittin and the CPP pVEC.

In this paper we explore the known connection among sustainability, resilience, and well-being within the framework of active inference. Initially, we revisit how the notions of well-being and resilience intersect within active inference before defining sustainability. We adopt a holistic concept of sustainability denoting the enduring capacity to meet needs over time without depleting crucial resources. It extends beyond material wealth to encompass community networks, labor, and knowledge. Using the Free Energy Principle, we can emphasize the role of fostering resource renewal, harmonious system-entity exchanges, and practices that encourage self-organization and resilience as pathways to achieving sustainability, both in an agent and in collectives. We start by connecting Active Inference with well-being, building on exsiting work. We then attempt to link resilience with sustainability, asserting that resilience alone is insufficient for sustainable outcomes. While crucial for absorbing shocks and stresses, resilience must be intrinsically linked with sustainability to ensure that adaptive capacities do not merely perpetuate existing vulnerabilities. Rather, it should facilitate transformative processes that address the root causes of unsustainability. Sustainability, therefore, must manifest across extended timescales and all system strata, from individual components to the broader system, to uphold ecological integrity, economic stability, and social well-being. We explain how sustainability manifests at the level of an agent, and then at the level of collectives and systems. To model and quantify the interdependencies between resources and their impact on overall system sustainability, we introduce the application of network theory and dynamical systems theory. We emphasize the optimization of precision or learning rates through the active inference framework, advocating for an approach that fosters the elastic and plastic resilience necessary for long-term sustainability and abundance.

In response to the issue of vehicle escape guidance, this manuscript proposes a unified intelligent control strategy synthesizing optimal guidance and Meta Deep Reinforcement Learning (DRL). Optimal control with minor energy consumption is introduced to meet terminal latitude, longitude and altitude. Maneuvering escape is realized by adding longitudinal and lateral direction maneuver overloads. Maneuver command decision model is calculated based on Soft-Actor-Critic (SAC) networks. Meta learning is introduced to enhance autonomous escape capability, which improves generalization performance to time-varying scenarios not encountered in the training process. In order to obtain training samples at a faster speed, this manuscript uses the prediction method to solve reward values, which avoiding a large number of numerical integration. The simulation results manifest that the proposed intelligent strategy can achieve high precise guidance and effective escape.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

In the context of reinforcement learning (RL), ensuring both safety and performance is crucial, especially in real-world scenarios where mistakes can lead to severe consequences. This study aims to address this challenge by integrating temporal logic constraints into RL algorithms, thereby providing a formal mechanism for safety verification. We employ a combination of theoretical and empirical methods, including the use of temporal logic for formal verification and extensive simulations to validate our approach. Our results demonstrate that the proposed method not only maintains high levels of safety but also achieves comparable performance to traditional RL algorithms. Importantly, our approach fills a critical gap in existing literature by offering a solution that is both mathematically rigorous and empirically validated. The study concludes that the integration of temporal logic into RL offers a promising avenue for developing algorithms that are both safe and efficient. This work lays the foundation for future research aimed at generalizing this approach to various complex systems and applications.

In August 2020, the World Health Assembly launched a global initiative to eliminate cervical cancer by 2030, setting three primary targets. One key goal is to achieve a 70% screening coverage rate for cervical cancer, primarily relying on the precise analysis of Papanicolaou (Pap) or digital Pap smears. However, the responsibility of reviewing Pap smear samples to identify potentially cancerous cells primarily falls on pathologists, a task known to be exceptionally challenging and time-consuming. This paper proposes a solution to address the shortage of pathologists for cervical cancer screening. It leverages the OpenAI-GYM API to create a deep reinforcement learning environment utilizing liquid-based Pap smear images. By employing the Proximal Policy Optimization algorithm, autonomous agents navigate Pap smear images, identifying cells with the aid of rewards, penalties, and accumulated experiences. Furthermore, the use of a pre-trained convolutional neuronal network like Res-Net54 enhances the classification of detected cells based on their potential for malignancy. The ultimate goal of this study is to develop a highly efficient, automated Papanicolaou analysis system, ultimately reducing the need for human intervention in regions with limited pathologists.

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in video transmission across networks. Traditional adaptive bitrate (ABR) algorithms adjust the quality of video segments based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules within a complex environment, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel Deep Reinforcement Learning-based approach for streaming DASH, focusing on maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes the Quality of Experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate this approach through a simulation model that encompasses diverse wireless network environments and various video sequences. Additionally, we compare our proposed approach with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

A frequent problem in Geotechnics is the soils with inadequate physical-mechanical properties before the efforts to which they will be subjected by the constructions, incurring cost overruns caused by their engineering improvement. The need to improve the engineering properties of soils is not recent, currently observing that the most common alternatives are binders such as cement and lime. The climate change observed in recent decades and the uncontrolled emission of greenhouse gases have motivated Geotechnical and Geoenvironmental researchers to seek mechanisms for soil reinforcement from a more sustainable and environmentally friendly approach by proposing the use of recycled and waste materials. An alternative is natural fibers, which can be obtained as waste from many agro-industrial processes, implying their high availability and low cost. Sawdust as a by-product of wood processing has a rough texture that can generate high friction between the fiber and the matrix of the soils, leading to a significant increase in its shearing strength and bearing capacity. This concept of improving the properties of soils using natural fibers distributed randomly is inspired by the natural phenomenon of grass and/or plants that when growing on a slope can effectively stabilize a said slope.

For a mobile robot, navigation in a densely crowded space can be a challenging and sometimes impossible task, especially with traditional techniques. In this paper, we present a framework to train neural controllers for differential drive mobile robots which must safely navigate a crowded environment while trying to reach a target location. To learn the robot’s policy, we train a convolutional neural network using two reinforcement learning algorithms, Deep Q-Networks (DQN) and Asynchronous Advantage Actor Critic (A3C), and develop a training pipeline that allows to scale the process to several compute nodes. We show that the asynchronous training procedure in A3C can be leveraged to quickly train neural controllers and test them on a real robot in a crowded environment.

Ever-increasing need for improving the livability of a city and improve outcomes for its residents, over the last decade, the adoption of technology to develop urbanised societies around the world has given rise to the need for developing smart cities. The speed at which the world population is growing, the use of Internet of Things in smart cities have really advanced the quality of life. One significant area of concern within the smart city framework is waste management. If the waste within a city is not adequately managed, then it leads to issues in the health of the citizens. Additionally, the waste management has such a high impact on the environmental footprint, hence the need to have a smart way of managing waste is of critical importance. Through our research, we analyse the challenges of waste management within a city to understand the impact of the problem on to the citizens and overall city operations. We then investigate ways in which we can solve these problems using the emerging technologies, such as the Internet of Things, to collect valuable data of large volumes arriving at an astronomical rate, then apply multi-agent deep reinforcement learning algorithms to harness the power of big data to extract meaningful information and actionable insights. We ingest data generated by our Internet of Things into our algorithm for three main purposes including providing the notifications to an external system, for example, a map navigation engine out of the scope for this project but a future extension for route optimisation and waste vehicle tracking; extracting and reporting the actionable insights from the underlying data; and consuming the extracted data for predictive forecasting to draw out the unknown patterns of waste fill levels within various geographical locations and again send out triggers and notification to external systems for example a waste collection authority who can efficiently schedule the waste collection vehicles and optimise the route. To achieve the above mentioned outcomes, we propose a framework that is agnostic of the hardware that it connects to and can effectively interface with a wide variety of hardware keeping a level of abstraction in the architecture.

Sparse reward long horizon task is a major challenge for deep reinforcement learning algorithm. One of the key barriers is data-inefficiency. Even in the simulation environment, it usually takes weeks to training the agent. In this study, a data-efficiency training framework is proposed, where a curriculum learning is design for the agent in the simulation scenario. Different distributions of the initial state are set for the agent to get more informative reward during the whole training process. A fine-tuning of the parameters in the output layer of the neural network for value function is conduct to bridge the gap between sim-to-real. An experiment of UAV maneuver control is conducted in the proposed training framework to verify the method more efficient. We demonstrate that data-efficiency is different for the same data in different training stages.

Abstract: As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs at a larger scale. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

In the past two decades, metaheuristic optimization algorithms (MOAs) have been increasingly popular, particularly in logistic, science, and engineering problems. The fundamental characteristics of such algorithms are that they are dependent on a parameter or a strategy. Some online and offline strategies are employed in order to obtain optimal configurations of the algorithms. Adaptive operator selection is one of them, and it determines whether or not to update a strategy from the strategy pool during the search process. In the filed of machine learning, Reinforcement Learning (RL) refers to goal-oriented algorithms, which learn from the environment how to achieve a goal. On MOAs, reinforcement learning has been utilised to control the operator selection process. Existing research, however, fails to show that learned information may be transferred from one problem-solving procedure to another. The primary goal of the proposed research is to determine the impact of transfer learning on RL and MOAs. As a test problem, a set union knapsack problem with 30 separate benchmark problem instances is used. The results are statistically compared in depth. The learning process, according to the findings, improved the convergence speed while significantly reducing the CPU time.

Recently, the application of deep reinforcement learning in recommender system is flourishing and stands out by overcoming drawbacks of traditional methods and achieving high recommendation quality. The dynamics, long-term returns and sparse data issues in recommender system have been effectively solved. But the application of deep reinforcement learning brings problems of interpretability, overfitting, complex reward function design, and user cold start. This paper proposed a tag-aware recommender system based on deep reinforcement learning without complex function design, taking advantage of tags to make up for the interpretability problems existing in recommender system. Our experiment is carried out on MovieLens dataset. The result shows that, DRL based recommender system is superior than traditional algorithms in minimum error and the application of tags has little effect on accuracy when making up for interpretability. In addition, DRL based recommender system has excellent performance on user cold start problems.

Henry Vidal first introduced the concept of using strips, grids, and sheets for reinforcing soil masses. Since then, a large variety of materials such as steel bars, tire shreds, polypropylene, polyester, glass fibers, coir, jute fibers etc. have been widely added to the soil mass randomly or in a regular, oriented manner. In this investigation, a new concept of multi-oriented plastic reinforcement (hexa-pods), is discussed. A systematic and comprehensive laboratory tests were conducted on unreinforced and reinforced soil samples. Laboratory tests such as direct shear teat and California bearing ratio (CBR) test were analyzed on soil samples consisting of only soil samples, soil sample with random inclusion of hexapods and soil samples with layered inclusion of hexapods. From the results obtained through direct shear test it could be observed that cohesion value of both the soil sample has increased and the angle of internal friction has been decreased after reinforcing it with inclusions in both randomly and layered conditions. CBR test indicates that for same amount of compactive effort, both random and layered inclusions of hexapods show improvement in strength and stiffness. Random inclusions of hexapods give better resistance to penetration as compared to layered inclusions. The hexa-pods also changed the brittle behavior of unreinforced sand samples to ductile ones.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL) and Health Systems Simulations has significant potential, for both research into improving Deep RL performance and safety, and in operational practice. While individual toolkits exist for Deep RL and Health Systems Simulations, no framework to integrate the two has been established. Aim: Provide a framework for integrating Deep RL Networks with Health System Simulations, and to ensure this framework is compatible with Deep RL agents that have been developed and tested using OpenAI Gym. Methods: We developed our framework based on the OpenAI Gym framework, and demonstrate its use on a simple hospital bed capacity model. We built the Deep RL agents using PyTorch, and the Hospital Simulation using SimPy. Results: We demonstrate example models using a Double Deep Q Network or a Duelling Double Deep Q Network as the Deep RL agent. Conclusion: SimPy may be used to create Health System Simulations that are compatible with agents developed and tested on OpenAI Gym environments. GitHub repository of code: https://github.com/MichaelAllen1966/learninghospital

Steel reinforcement in concrete has the tendency to corrode and this process can lead to structural damage. FRP reinforcement represents a viable alternative for structures exposed to aggressive environments and has many possible applications where superior corrosion resistance properties are required. The use of FRP rebars as internal reinforcements for concrete, however, is limited to specific structural elements and does not yet extend to the whole structure. The reasons for this relate to the limited availability of curved or shaped reinforcing elements on the market and their reduced structural performance. Various studies, in fact, have shown that the mechanical performance of bent portions of composite bars is reduced significantly under a multiaxial combination of stresses and that the tensile strength can be as low as 25% of the maximum tensile strength that can be developed in the straight part. In a significant number of cases, the current design recommendations for concrete structures reinforced with FRP, however, were found to overestimate the bend capacity of FRP rebar. This paper presents the state-of-the art review of the research works on the strength degradation in curved FRP composites and highlighted the performance of exiting predictive models for the bend capacity of FRP reinforcement. Recent practical predictive model based on the Tsai-Hill failure criteria by considering the material at marcromechanical level is also discussed and highlighted. The review also identifies the challenges and highlights the future directions of research to explore the use of shaped FRP composites in civil engineering applications and the trends for future research in this area.

Background: Here, we investigated the inflammaging and physical performance in older adults before and after the COVID-19 pandemic. Methods: Women(n=18) and men(n=7) older were evaluated before the COVID-19 pandemic(PRE), 12 months after the lockdown(POST), and 10 months after resuming exercise training(POS-TR). Physical tests [gait speed(GS) and timed-up-and-go(TUG)]; muscle strength (handgrip-HG), and serum cytokine levels were assessed. Results: The older women showed higher GS and TUG at POST than PRE and POST-TR, besides lower HG at POST-TR than PRE, whereas older men exhibited lower HG at POST and POST-TR than PRE. Both groups presented (1)lower IL-10 and IL-12p70 values in contrast to higher IL-6/IL-10 and IL-8/IL-10 ratios at POST than PRE; (2)higher IL-10 values and lower IL-8/IL-10 ratio at POST-TR than POST; (3)higher IL-12p70/IL-10 ratio at POST-TR than PRE and POST. Particularly older women showed: (4)lower IL-6 values at POST and POST-TR than PRE; (5)lower IL-8 and IL-10 values at POST-TR than POST; (6)and higher TNF-α/IL-10 and IFN-γ/IL-10 ratios at POST than PRE and POST-TR. Significant correlations between the variables were found in both groups. Conclusion: During the COVID-19 pandemic, detraining and resumption of exercise training promoted distinct alterations in physical capacity and inflammaging among older women and older men population.

With the proliferating of video surveillance system deployment and related applications, real-time video analysis is very critical to achieve intelligent monitoring, autonomous driving, etc. It is non-trivial to achieve high accuracy and low latency video stream analysis through the traditional cloud computing. In this paper, we propose a non-orthogonal multiple access (NOMA) based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the QoE of UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimension, which respectively select the offloading and resource allocation action, the resolution scaling action. Experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of transmission and computation, and have better performance in convergence compared to other baseline.

In this review we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.

The free energy principle (FEP) is a mathematical and scientific principle that describes the relationship between the physical behavior of a dynamical system and the interpretation of such behavior as carrying out inference. It blurs the boundary between the ontological mode of the system of interest and the epistemological significance it acquires through its dynamic interaction with the environment. New materialism (or neo-materialism), despite lacking a single definition, refers to a range of emerging perspectives that attempt to dismantle the long-held divisions between ontology, epistemology, and even ethics, with the aim of achieving a comprehensive transformation of naturalized thinking. In this context, new materialism provides powerful tools to reframe some of the conceptual foundations of the FEP. This paper is the first in a series that aims to systematically construct a neo-materialistic perspective for the FEP, exploring its wide-ranging implications. It serves as an introduction to the series, arguing for the justification of deploying new materialism to reframe the FEP and introducing some essential concepts to be utilized in developing a neo-materialistic account of the FEP.

The spread of ideas is a fundamental concern of today’s news ecology. Understanding the dynamics of the spread of information and its co-option by interested parties is of critical importance. Research on this topic has shown that individuals tend to cluster in echo-chambers and are driven by confirmation bias. In this paper, we leverage the active inference framework to provide an in silico model of confirmation bias and its effect on echo-chamber formation. We build a model based on active inference, where agents tend to sample information in order to justify their own view of reality, which eventually leads to them to have a high degree of certainty about their own beliefs. We show that, once agents have reached a certain level of certainty about their beliefs, it becomes very difficult to get them to change their views. This system of self-confirming beliefs is upheld and reinforced by the evolving relationship between agent's beliefs and its observations, which over time will continue to provide evidence for their ingrained ideas about the world. The epistemic communities that are consolidated by these shared beliefs, in turn, tend to produce perceptions of reality that reinforce those shared beliefs. We provide an active inference account of this community formation mechanism. We postulate that agents are driven by the epistemic value that they obtain from sampling or observing the behaviors of other agents. Inspired by digital social networks like Twitter, we build a generative model in which agents generate observable social claims or posts (e.g. `tweets') while reading the socially-observable claims of other agents, that lend support towards one of two mutually-exclusive abstract topics. Agents can choose which other agent they pay attention to at each timestep, and crucially who they attend to and what they choose to read influences their beliefs about the world. Agents also assess their local network’s perspective, influencing which kinds of posts they expect to see other agents making. The model was built and simulated simulated using the freely-available Python package pymdp. The proposed active inference model can reproduce the formation of echo-chambers over social networks, and gives us insight into the cognitive processes that lead to this phenomenon.

In this paper, the single event burnout(SEB) and reinforcement structure of 1200V SiC MOSFET (SG-SBD-MOSFET) with split gate and schottky barrier diode(SBD) embedded were studied. The device structure was established using Sentaurus TCAD, and the transient current changes of single event effect(SEE), SEB threshold voltage, as well as the regularity of electricfield peak distribution transfer were studied when heavy ions were incident from different regions of the device. Based on SEE analysis of new structural device, two reinforcement structure designs for SEB resistance were studied, namely the expansion of P+body contact area and the design of multi-layer N-type interval buffer layer. Firstly, two reinforcement schemes for SEB were analyzed separately, and then comprehensive design and analysis were carried out. The results showed that the SEB threshold voltage of heavy ions incident from the N+source region was increased by 16% when using the P+body contact area extension alone; When the device is reinforced with multi-layer N-type interval buffer layer alone, the SEB threshold voltage increases by 29%; The comprehensive use of P+body contact area expansion and multi-layer N-type interval buffer layer reinforcement increased the SEB threshold voltage by 33%. Overall, the breakdown voltage of the reinforced device decreased from 1632.935V to 1403.135V, which can be seen as reducing the remaining redundant voltage to 19%. The device performance was not significantly affected.

Abstract—Online learning platforms provide diverse course resources, but this often result in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, Existing course recommendation methods face the challenge in accurately recommending the target course that is most relevant to the user, because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the Actor-Critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendation effectively. Furthermore, we propose a policy gradient based on temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and improves the accuracy of the recommendation. We evaluate the proposed model on two real datasets, and the experimental results show that the proposed model is significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

Shape Memory Alloy (SMA) have emerged as a revolutionary material that holds the potential to change the design and construction methods of buildings and other structures. One of the most promising applications of SMAs in civil engineering is their use as reinforcements for steel structures. This paper presents the latest research progress, opportunities and challenges of SMA in the field of steel structural reinforcement, both in terms of basic components and applications. In terms of components, the construction forms and working mechanisms of Fe-SMA strips, SMA/CFRP composite patches and SMA dampers are introduced. On this basis, the application of SMA in steel structures reinforcement is introduced, and its effect is analyzed from three aspects: crack restoration, seismic retrofitting and structural strengthening. Finally, the results of the current research are summarized and the shortcomings are analyzed, hoping to provide a reference for the research of SMA in the field of steel structures reinforcement.

The increasing popularity of carbon-reinforced concrete (CRC) is attributed to its exceptional tensile properties, low density, no corrosion phenomenon, and remarkable flexibility, allowing it to be easily shaped into various forms. This research investigates the feasibility of using a special 2D Netzgitterträger (NetzGT) reinforcement system, featuring a net-shaped fabricated textile made of multiple diagonally offset rovings with overlapping edge strands, as a viable alternative to traditional steel reinforcement in concrete beams. This reinforcement is manufactured from carbon rovings with three different diagonal angles of 50⁰, 60⁰, and 70⁰ respectively. Laboratory experiments were conducted to assess the mechanical behavior of beams reinforced with the 2D NetzGT reinforcement. Bending and shear tests were performed on beams with varying numbers of overlapped edge roving and roving angles to evaluate the tensile capacity and failure characteristics of beams. The increase in the number of overlapped edge rovings led to a noticeable increase in the maximum tensile force. Tensile tests on strands were also performed with the increasing number of overlapped rovings to analyze their tensile strength. Additionally, single yarn pull-out tests were also conducted to examine the influence of the roving angle on the bond strength between the carbon textile roving and the concrete matrix

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.

Fast convergence routing is an important issue for LEO constellation network, due to its dynamical topology changing and time varying transmission requests. Most of existing research focus on the OSPF routing algorithm, which cannot handle the frequently links state changing of network. In this paper, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL-SR) for LEO satellite networks, in which the satellite gets the network links status fast and adjusts its routing strategy. In FRL-SR, each satellite node is regarded as an agent. The agent selects the port for packet forwarding according to its own routing policy. When the satellite network state changes, agent would send ’hello’ packets to the neighbor node to update the neighbor node’s routing policy. Compared with traditional reinforcement learning, FRL-SR can perceive network information faster, and then converge faster. Also, FRL-SR can mask the dynamics of satellite network topology and adaptively adjust the forwarding strategy according to the link state. Various simulation is constructed, the results show that the proposed FRL-SR algorithm out performance the Dijkstra algorithm in performance of average delay, packet arriving ratio, network load balance.

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

We consider the problem of coordinating the charging of an entire fleet of electric vehicles (EV), using a model-free approach, i.e. purely data-driven reinforcement learning (RL). The objective of the RL-based control is to optimize charging actions, while fulfilling all EV charging constraints (e.g. timely completion of the charging). In particular, we focus on batch-mode learning and adopt fitted Q-iteration (FQI). A core component in FQI is approximating the Q-function using a regression technique, from which the policy is derived. Recently, a dueling neural networks architecture was proposed and shown to lead to better policy evaluation in the presence of many similar-valued actions, as applied in a computer game context. The main research contributions of the current paper are that (i)we develop a dueling neural networks approach for the setting of joint coordination of an entire EV fleet, and (ii)we evaluate its performance and compare it to an all-knowing benchmark and an FQI approach using EXTRA trees regression technique, a popular approach currently discussed in EV related works. We present a case study where RL agents are trained with an epsilon-greedy approach for different objectives, (a)cost minimization, and (b)maximization of self-consumption of local renewable energy sources. Our results indicate that RL agents achieve significant cost reductions (70--80%) compared to a business-as-usual scenario without smart charging. Comparing the dueling neural networks regression to EXTRA trees indicates that for our case study's EV fleet parameters and training scenario, the EXTRA trees-based agents achieve higher performance in terms of both lower costs (or higher self-consumption) and stronger robustness, i.e. less variation among trained agents. This suggests that adopting dueling neural networks in this EV setting is not particularly beneficial as opposed to the Atari game context from where this idea originated.

In recent years, wireless networks have undoubtedly become a convenient way to connect to the Internet and provide connection to everyone in any corner of the world. In fact, in this era, people are connected to the internet almost everyday and wireless networks give us this privilege in a seamless manner. A wireless network normally consists of access points and nodes where the access points are responsible to amplify the wireless signals, while the nodes are the gadgets that are receiving these signals. However, with such great convenience provided, many challenges are also faced by the users and stakeholders. With no physical connection to devices, wireless networks are evidently more vulnerable to invisible cyber attacks. In this research paper, it the security issues that cause issues in the wireless networks are discussed. Furthermore, an analytical review of privacy challenges found in these networks is performed; these challenges are segregated into security issues and privacy issues. The paper will then present the methods used in conducting a survey and gathering the research results along with further discussion on the results obtained through this study. Finally, a suitable solution is proposed to prevent and overcome the intrusions faced in terms of security and privacy in wireless network scenarios through detection and response mechanism for mitigation of the problems.

Type two diabetes mellitus (T2DM) represents a chronic condition with increasing prevalence worldwide among the older population. T2DM condition increases the risk of micro and macro-vascular complications as well as the risk of geriatric syndromes as falls, fractures and cognitive impairment. The management of T2DM in the older population represents a challenge for the cli-nician, and a Comprehensive Geriatric Assessment should always be prioritized, in order to tailor the glycate haemoglobin target according to functional and cognitive status comorbidities, life ex-pectancy and type of therapy. According to the most recent guidelines, older adults with T2DM should be cathegorized in three groups: healthy patients with good functional status, patients with complications and reduced functionality and patients at the end of life; for each group the target for the glycemic control is different, also according to the type of treatment drug. The therapeutic ap-proach should always begin with lifestyle changes; after that, several lines of therapies are available, with different mechanism of action and potential effect other than glucose level reduction. Partic-ular interest is growing around sodium-glucose cotransporter-2 inhibitors, due to their effect on the cardiovascular system. In this review we evaluate the therapeutic options available for the treat-ment of older diabetic patients, to ensure a correct treatment approach

Background: Few studies have assessed built environment correlates of active commuting in low-and-middle-income countries, but the different context could yield distinct findings. This cross-sectional study investigated associations between home neighbourhood environment characteristics and active commuting in Chennai, India. Methods: Adults (N = 370, 47.2% female, mean age = 37.9 years) were recruited from 155 wards in the metropolitan area of Chennai in southern India between January and June 2015. Participants self-reported their usual mode of commute to work, with responses recoded into three categories: (1) multi-modal or active commuting (walking and bicycling); (2) public transit; and (3) private transport. Environmental attributes around participants’ homes were assessed using the Neighborhood Environment Walkability Scale-India (NEWS-India). Associations between environmental characteristics and likelihood of active commuting and public transit use were modelled using logistic regression with private transport (driving alone or carpool) as the reference category, adjusting for age, gender, and household car ownership. Results: Consistent with other international studies, participants living in neighbourhoods with a mix of land-uses and a transit stop within a 10-minute walk from home were more likely to use active commuting (both p < 0.01). Land-use mix was significantly associated with the use of public transit compared to private transport (aOR = 5.2, p = 0.002). Contrary to findings in high-income countries, the odds of active commuting were reduced with improved safety from crime (aOR = 0.2, p = 0.003), aesthetics (aOR = 0.2, p = 0.05) and street connectivity (aOR = 0.2, p = 0.003). Conclusions: Different environmental attributes were associated with active commuting, suggesting that these relationships are complex and may distinctly differ from those in high-income countries. Unexpected inverse associations of perceived safety from crime and aesthetics with active commuting emphasize the need for high quality epidemiologic studies with greater context-specificity in the study of physical activity in LMICs. Findings have public health implications for India and suggest that caution should be taken when translating evidence across countries.

This study explores the significance of active suspension systems for vehicles with lower chassis compared to conventional ones, aiming at the development of future automobiles. Conventional linear MR (Magneto-Rheological) dampers were found inadequate in ensuring sufficient vibration control because the vehicle's chassis becomes lowered in the unmanned vehicles or purposed based vehicles. As an alternative, a rotary type of MR damper is proposed in this work. The proposed damper is designed based on pre-specified design parameters through mathematical modeling and magnetic field analyses. Subsequently, a prototype of the rotary MR damper identical to the design is fabricated, and effectiveness is shown through experimental investigations. In configuring the experiments, a proportional-integral (PI) controller is employed for current control to reduce the response time of the damper. The results presented in this work provides useful guidelines to develop a new type of MR damper applicable to various types of future vehicles’ suspension systems with low distance from the tire to the body floor.

(1) Background: Considering highly selected patients with ductal carcinoma in situ (DCIS), active surveillance is a valid alternative to surgery. Our study is aimed at showing the reliability of post-biopsy complete lesion removal, documented by mammogram, as additional criterion to select these patients. (2) Methods: 2173 Vacuum Assisted Breast Biopsies (VABB) documented as DCIS have been reviewed. Surgery has been performed in all cases. We retrospectively collected the reports of post-VABB complete lesion removal and the histological results of the biopsy and surgery. We calculated the rate of upgrade of DCIS identified on VABB upon excision for patients with post-biopsy complete lesion removal and for those showing residual lesion. (3) Results: We observed 2173 cases of DCIS: 408 classified as low grade; 1262 as intermediate grade; 503 as high grade. The overall upgrading rate to invasive carcinoma was 15.2% (330/2173). The upgrade rate was significantly lower (8.2%) when considering patients showing mammographically documented complete removal of the lesion. (4) Conclusion: The absence of mammographically documented residual lesion following VABB is associated to a lower upgrading rate of DCIS to invasive carcinoma on surgical excision and should be considered when deciding the proper management DCIS diagnosis.

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.

Continuous greenhouse gas emissions are causing global warming and impacting the habitats of many animals. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable is challenging. To address the issue, we propose a grid dispatching technique that combines prediction technology, reinforcement learning, and optimization technology. Prediction technology can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the power system, the aggregation of distributed energy resources increases uncertainty, particularly due to the fluctuating generation of renewable energy. This requires the use of advanced predictive control techniques to ensure long-term economic and decarbonization goals. In this paper, we present a real-time dispatching framework that integrates deep learning-based prediction, reinforcement learning-based decision-making, and stochastic optimization techniques. The framework can rapidly adapt to target uncertainty caused by various factors in real-time data distribution and control processes. The proposed framework achieved global Champion in the NeurIPS Challenge 2022 competition and demonstrated its effectiveness in practical scenarios of intelligent building energy management.

Wireless energy harvesting (EH) communication has long been considered a sustainable networking solution. However, it has been limited in efficiency, which has been a major obstacle. Recently, strategies such as energy relaying and borrowing have been explored to overcome these difficulties and provide long-range wireless sensor connectivity. In this article, we examine the reliability of the wireless-powered communication network by maximizing the net bit rate. To accomplish our goal, we focus on enhancing the performance of hybrid access points and information sources by optimizing their transmit power. Additionally, we aim to maximize the use of harvested energy by energy-harvesting relays for both information transmission and energy relaying. However, this optimization problem is complex as it involves non-convex variables and requires combinatorial relay selection indicators optimization for decode and forward (DF) relaying. To simplify this problem, we utilize the Markov decision process and deep reinforcement learning framework based on the deep deterministic policy gradient algorithm. This approach enables us to tackle this non-tractable problem, which conventional convex optimisation techniques would be difficult to solve in complex problem environments. The proposed algorithm significantly improves the end-to-end net bit rate of the smart energy borrowing and relaying EH system by 13.22%,27.57%, and 14.12% compared to the benchmark algorithm based on borrowing energy with an adaptive reward for Quadrature Phase Shift Keying, 8-PSK, and 16-Quadrature amplitude modulation schemes, respectively.

This article focuses on developing a continuous authentication system using behavioral biometrics to recognize users accessing computing devices. The user’s distinct behavioral biometric is captured through keystroke dynamics, and reward-based reinforcement learning (RL) ideas are applied to recognize them throughout the session. The suggested system adds an extra layer of security to traditional authentication methods, forming a robust continuous authentication system that can be added to static authentication systems. The methodology involves training a RL model to detect unusual user typing patterns and flag suspicious activity. Each user has an agent trained on their historical data, which is preprocessed and used to create episodes for the agent to learn from. The environment involves fetching observations and randomly corrupting them to learn out-of-order behavior. The observation vector includes both running features and summary features. The re-ward function is binary and minimalistic. The Principal Component Analysis (PCA) model is used to encode the running features, and the Double Deep Q-Network (DDQN) algorithm with a fully connected neural network is used as the policy net. The evaluation achieved an average training accuracy and EER (equal error rate) of 94.7% and 0.0126 and test accuracy and ERR of 81.06% and 0.0323 for all users when the number of encoder features was increased. Therefore, it is concluded that by continuously learning and adapting to changing behavior patterns, this approach can provide more secure and personalized authentication, lowering the possibility of unauthorized access and cyberattacks. Overall, the use of reinforcement learning and behavioral biometrics for continuous authentication has the potential to significantly enhance security in the digital age and are effective in identifying each user.

Selecting the optimal activation function for training deep neural networks has always been challenging because it significantly impacts the neural network’s performance and training speed. At this point, researchers are more likely to employ RELU than other well-known activation functions. After RELU, researchers have proposed many activation functions to improve RELU. None of them was capable of surpassing RELU as their most significant rival. SWISH outperformed RELU in several challenging experiments like classification, object detection, and tracking. Replacing RELU units with SWISH, which improves performance in many tasks. This paper proposed a new activation function surpassing Google’s brain’s SWISH function, Which we named AIF. Experiments indicate that our proposed activation function outperforms SWISH in various tasks.

Due to the rapid development of nanotechnologies, materials, in particular , solid dispersions (SDs), which are actively introduced into the life of modern man, have been obtained. Special progress in this area is observed in industry and medicine. The use of SDs in agriculture is lagging far behind, despite the growing number of scientific papers on this topic. At the same time, the prospects for the introduction of SDs in the agro-industrial complex are obvious. The review presents the results of research on the development of innovative preparations based on SD to protect plants from diseases and pests of cultivated plants, as well as parasiticides to protect animal health based on modern achievements of nanotechnology. One of these technologies is the methods of mechanochemistry, which improve the properties of poorly soluble biologically active substances by their joint mechanical treatment with water-soluble polymers and auxiliary substances.

Risk-based redispatch optimization is proposed as a methodology to support the Transmission System Operator (TSO) with preventive remedial actions obtained by extending the security constrained unit commitment/economic dispatch with constraints resulting from the risk assessed for the power system. Although being heuristic, the methodology is based on comprehensive dynamic security assessment as time-domain simulations are used, allowing to express the degree of all types of instabilities, e.g., caused by contingencies, in monetary terms. Therefore, the risk is assessed as the expected value of the cost incurred by the TSO. Such approach forms a new pathway to including risk in planning procedures already used by TSOs. Results obtained for the IEEE39 dynamic power system, with costs assigned to load shedding and generator tripping due to single transmission lines short-circuits, are shown as a reference case.

Fluxgate magnetometers require calibration methods appropriate to their application levels and particularities; however the development of fully controlled calibration procedures presents a particular challenge regarding the inevitable influence of the local geomagnetic field and other external interferences when a laboratory with magnetically shielded walls is not available. In that context, we discussed the development of an automated calibration method for fluxgate magnetometers, considering those limitations in time and space, and avoiding some of the problems commonly found in other proposed solutions for the same challenge. For this task, we designed and built a new set of high level procedures, electronic systems and software, which perform active testing and automated calibration of fluxgate magnetometers, considering some resource constraints and employing instruments commonly found in electronic calibration laboratories.

In this study, a comprehensive review on production scheduling based on reinforcement learning (RL) techniques using bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of all pertinent publications contained in the Web of Science database. The study focuses on a wide spectrum of publications spanning the years between 1998 and 2023. The findings and recommendations of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.

This article presents the development of an innovative Maximum Power Point Tracking (MPPT) strategy, utilizing a Particle Swarm Optimization (PSO) algorithm to improve the effectiveness of PV systems and expedite convergence. The new MPPT method incorporated a unique Swarm Self-Reinforcement Mechanism (SSRM) within the PSO algorithm, targeting quick convergence and excellent tracking accuracy. This approach enables the PSO to eliminate the fitness function that has the lowest value and subsequently reinforce it in the next iteration, revolving around the global maximum power point (GMPP). By applying this novel PSO-based method, the MPPT performance of PV systems was significantly improved, facilitating the algorithm to proficiently navigate through the solution space and quickly locate the GMPP, even in rapidly changing environmental conditions. The outcomes derived from this novel approach were contrasted with other algorithmic optimization methods, validating its superior convergence speed and tracking accuracy. Different swarm sizes were examined using SSRM, and the optimal swarm size for the system employing MPPT was identified to achieve the lowest convergence time (CT). The results showcased the impressive performance capabilities of this novel strategy, resulting in a time con-traction of up to 28% compared to the conventional PSO technique, where the optimal swarm size was found to be 5. This achievement marks a significant milestone in the evolution of PSO-based MPPT techniques, and paves the way for future advancements in this exciting field.

We propose a deep reinforcement learning based manipulator path tracking method to solve the computationally difficult and non-unique problem of manipulator path tracking methods based on inverse kinematics. By transforming the path tracking task into a sequence decision problem, our method adopts an end-to-end learning method for closed-loop control and avoids the process of finding the inverse solution. We first explored the feasibility of the deep reinforcement learning method in the path tracking of the manipulator. After verifying the feasibility, the path tracking of the multi-degree-of-freedom(multi-DOF) manipulator was realized by combining the maximum entropy deep reinforcement learning algorithm. The experimental results show that our method has a good effect on the path tracking of the manipulator, which not only avoids the process of finding the inverse kinematics solution, but also requires no dynamic model. Therefore, we believe that our method has great significance in the study of manipulator path tracking.

In tunnel excavation with boring machines, the tunnel face is supported to avoid collapse and minimise settlement. This article proposes the use of reinforcement learning, specifically the Deep Q-Network algorithm, to predict the face support pressure. The approach is tested both analytically and numerically. By using the soil properties ahead of the tunnel face and the overburden depth as the input, the algorithm is capable of predicting the optimal tunnel face support pressure, adapting to changes in geological and geometrical conditions.

Analyzing urban-rural spatial structures and changes are of great significance for under-standing the urban-rural relationship evolution. While previous studies mostly focused on the urban internal spatial structure evolution, less from the regional scale to explore the urban and rural evolution. Here, taking Dezhou city, a rapidly urbanization city in China as a case study, we employed the local contour tree method and nighttime light data to map urban and active rural regions’ range during 2012-2020, and further explored the respective development processes of them. This study found that different from rural regions, internal structures of urban regions are more complex, and there are often multiple hot spots inside them. The urban-rural regions’ area had increased significantly by 39.3% during 2012-2020 (p&lt;0.05). The urban and rural region rankings of the identified counties are basically consistent with the urban and rural population rankings. Particularly, different from perspectives of earlier land use (i.e., built-up land or impervious surface), this study identified urban and active rural regions in view of the scope of active human activities. This study therefore found populations were greatly responsible for affecting changes of urban and active rural region structures. These findings could be helpful for understanding urban-rural dynamics.

A systematic investigation of the dynamic clustering behavior of active particles under confinement, including the effects of both particle density and active driving force is presented based on a hybrid coarse-grained molecular dynamics simulation. First, a series of scaling laws were derived with power relationships for the dynamic clustering time as a function of both particle density and active driving force. Notably, the average number of clusters N ̅ assembled from active particles in the simulation system exhibits a scaling relationship with clustering time t described by N ̅∝t^(-m). Simultaneously, the scaling behavior of the average cluster size S ̅ is characterized by S ̅∝t^m. Our findings reveals the presence of up to four distinct dynamic regions concerning clustering over time, with transitions contingent upon the particle density within the system. Furthermore, as the active driving force increases, the aggregation behavior also accelerates, while an increase in density of active particles induce alterations in the dynamic procession of the system.

Image segmentation and classification still represent an active area of research since no universal solution can be identified. Established segmentation algorithms like thresholding are problem specific, treat well the easy cases and mostly relied on single parameter i.e intensity. Machine learning approaches offer alternatives where predefined features are combined into different classifiers. On the other hand, the outcome of machine learning is only as good as the underlying feature space. Differential geometry can substantially improve the outcome of machine learning since it can enrich the underlying feature space with new geometrical objects, called invariants. In this way, the geometrical features form a high-dimensional feature space for each pixel, where original objects can be resolved. Alternatives based on the geometry of the image scale-invariant interest points have been exploited successfully in the field of computer vision. Here, we integrate geometrical feature extraction based on signal processing, machine learning, and input relying on domain knowledge. The approach is exemplified on the ISBI 2012 image segmentation challenge data set. As a second application, we demonstrate powerful image classification functionality based on the same principles, which was applied to the HeLa and HEp-2 data sets. Obtained results demonstrate that feature space enrichment properly balanced with feature selection functionality can achieve performance comparable to deep learning approaches.

Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.

This work is aiming to motivation on the prospect of evolving new thiazole dyes with respectable application properties, expected pharmacological activities. Curcumin Coupling with diverse diazonium salts of 2-amino thiazole derivatives as 2-aminobenzothiazole, 2- amino-5-phenylthiazole, 2 amino-5-methylthiazole and 2 amino-5-nitrothiazole to produce novel azo dyes. All synthesised dyes were completely confirmed their structures via elemental and spectroscopic techniques. The synthesised thiazole derivatives were examined for their “antimicrobial, anticancer and antioxidant” activities. All of the synthesized dyes were applied on synthetic fabrics as polyester and successively their dyeing properties, “light, washing, perspiration, rubbing and sublimation” fastness were evaluated. Prepared dyestuffs are suitable for dyeing polyester fabrics. It was initiate that all of prepared dyes own extraordinary colour hue, along with respectable fastness properties. Also the synthesised thiazole derivatives display moral pharmacology activity.

With the advancement of Additive Manufacturing and its applications in various industrial segments, it becomes increasingly important to investigate the processability parameters associated with nano-reinforcement. This study examines the influence of infill parameters, such as patterns (hexagonal, triangular and concentric), shape (solid and honeycomb) and concentrations by mass of Carbon Nanotubes (CNTs) at 1.0 wt% and 2.0 wt% on the mechanical compression properties of the 3D printed material. In this sense, nanocomposites based on Polylactic Acid (PLA) matrix nano-reinforced by CNTs were characterized using the Scanning Electron Microscopy -SEM, X-ray Diffraction, Raman Spectroscopy techniques and mechanical analysis. The mechanical properties of both the matrix and the nanocomposites were determined through mechanical compression tests, in accordance with [1]. SEM revealed compacted regions, voids, and detachment in the structures. XRD characterization indicated PLA predominantly amorphous nature, while CNTs exhibited characteristic diffractions of carbon nanotubes. Raman characterization showcased bands and characteristic vibrations of CNTs and PLA, nanocomposites' vibrational modes combine those of CNTs and PLA. Mechanical compression analysis indicates a direct influence of infill pattern, shape, and nano-reinforcement. The triangle pattern outperformed other patterns, exhibiting a 15.03% increase compared to concentric and 8.8% compared to hexagon infill. In nanocomposites, all showed higher compressive strength than the matrix. For PLA/1.0%CNTs, strength was 16.8%, and for PLA/2.0%CNTs, it reached 39.2%. In honeycomb shapes with infill variations, the triangular pattern excelled with a 0.97% increase over concentric and 6.12% over hexagon patterns. Honeycomb results indicate a 59.6% increase for (PLA/1.0%CNTs) and a 0.48% increase for (PLA/2.0%CNTs) compared to the matrix. This study highlights 3D printing parameters, emphasizing the effectiveness of the triangular pattern and the enhanced strength with the addition of CNTs, providing insights to enhance products manufactured through Additive Manufacturing.

Selective Paste Intrusion (SPI) is an additive manufacturing (AM) process in which thin layers of aggregates are selectively bonded by cement paste only where the structure is to be produced. In this way, concrete elements with complex geometries and structures can be created. Reinforcement is required to increase the flexural strength of the concrete elements and, thus, enable their applicability in practice. Integrating the reinforcement is a difficult task, particularly in the case of SPI due to the layer-wise printing method. Especially with respect to possible complex structures, the production of the reinforcement needs to be adapted to SPI, thereby offering a high degree of freedom. One concept for a reinforcement integration is combining the two additive manufacturing processes SPI and Wire and Arc Additive Manufacturing (WAAM). However, since the two processes serve different fields of application, their compatibility is not necessarily given. Ongoing investigations show that the temperatures caused by WAAM adversely affect both, the cement paste rheology required for sufficient paste penetration into the particle bed and the overall concrete strength. This paper provides an overview of ongoing research focusing on different cooling strategies and their effects on the compressive strength of SPI-printed concrete parts.

The robust control of high precision electromechanical systems, such as micropositioners, is challenging in terms of the inherent high nonlinearity, the sensitivity to external interference, and the complexity of accurate identification of the model parameters. To cope with these problems, this work investigates a disturbance observer-based deep reinforcement learning control strategy to realize high robustness and precise tracking performance. Reinforcement learning has shown great potential as optimal control scheme, however, its application in micropositioning systems is still rare. Therefore, embedded with the integral differential compensator (ID), deep deterministic policy gradient (DDPG) is utilized in this work with the ability to not only decrease the state error but also improves the transient response speed. In addition, an adaptive sliding mode disturbance observer (ASMDO) is proposed to further eliminate the collective effect caused by the lumped disturbances. The sterling performance is revealed with intensive tracking simulation experiments and demonstrates the improvement in the accuracy and response time of the controller.

The fifth Generation (5G) mobile networks use millimeter Waves (mmWaves) to offer giga bit data rates. However, unlike microwaves, mmWave links are prone to user and topographic dynamics. They easily get blocked and end up forming irregular cell patterns for 5G. This in turn cause too early, too late, or wrong handoffs (HOs). To mitigate HO challenges, sustain connectivity and avert unnecessary HO, we propose a HO scheme based on Jump Markov Linear System (JMLS) and Deep Reinforcement Learning (DRL). JMLS is widely known to account for abrupt changes in system dynamics. DRL likewise emerges as an artificial intelligence technique for learning highly dimensional and time-varying behaviors. We combine the two techniques to account for time-varying, abrupt, and irregular changes in mmWave link behaviour by predicting likely deterioration patterns of target links. The prediction is optimized by meta training techniques that also reduces training sample size. Thus, the JMLS-DRL platform formulates intelligent and versatile HO policies for 5G. Results show our proposed prediction scheme about target link behavior post HO to be highly reliable. The scheme also averts unnecessary HOs thus ably supports longer dew time.

An increasing use of non-metallic reinforcement is problematic as it has to be detected at the stage of accepting construction works, or later when expert opinions are prepared for the building. In contrast to metallic reinforcement, location of this type of reinforcement is difficult using non-destructive techniques. Small diameters of rebars and their location in a tested element were troublesome. This article describes an attempt to locate non-metallic reinforcement in a concrete element and the masonry. Tests were performed using an ultrasonic tomograph and GPR with a broad range of frequencies.

The objective of this study was to investigate age-related changes in cashmere production and the population of active secondary hair follicles in cashmere goats across different age groups, as well as to explore the association between secondary hair follicle activity and oxidative stress. A total of 104 adult Inner Mongolian ewe goats, aged between 2 and 7 years old, were randomly selected as experimental subjects. Skin samples were collected in August 2020 and cashmere samples were collected in April 2021. The cashmere fiber yield, staple length, and diameter showed age-related variations in cashmere goats aged 2 to 7 years (P < 0.05). Cashmere production was higher in goats aged 2-4 years compared to those aged 5-7 years (P < 0.05). There were no significant differences in the population of primary and secondary hair follicles among goats aged 2 to 7 years. However, the population of active secondary hair follicles varied significantly with age, with the younger group (aged 2-4 years) having a higher population than those aged 5 – 7 years (P < 0.05). A moderate negative correlation was observed between cashmere fiber diameter and the population of active secondary hair follicles (P < 0.05). Age-related variations in skin antioxidant capacity and oxidative damage were observed among cashmere goats aged 2 to 7 years old (P < 0.05). Goats aged 2 to 4 years exhibited higher antioxidant capacity and lower oxidative damage (P < 0.05). Interestingly, the skin antioxidant capacity and oxidative damage exhibited significant positive and negative correlations, respectively, with the population of active secondary hair follicles (P < 0.05). This study presents a novel approach to enhance the activity of secondary hair follicles and improve cashmere production performance through the regulation of oxidative stress.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

Cladding is typically used to protect components from wear and corrosion while also improving the aesthetic value and reliability of the substrate. The cladding process induces significant residual stresses due to the temperature difference between the substrate and the clad layer. However, these residual stresses could be effectively utilized by modifying processes and geometrical parameters. This paper introduces a novel methodology for using the weld-cladding process as a cost-effective alternative to various existing reinforcement techniques. The numerical analyses are performed to maximize the reinforcement of a cylindrical tool. The investigation of how the weld cladding develops compressive stresses on the specimen in response to a change in the weld beads and the welding sequence is presented. For the benchmark shape, experimental verification of the numerical model is performed. The impact of the distance between the weld beads and the effect of the tool diameter is numerically investigated. Furthermore, the variation in compressive stresses due to temperature fluctuations during the extrusion process has been evaluated. The results showed that adequate compressive stresses are generated on the welded parts through the cladding process after cooling. Hence, the targeted reinforcement of the substrate can be achieved by optimizing the welding sequence and process parameters.

In this paper, we study the performance of various deep reinforcement learning algorithms to enhance the energy management system of a microgrid. We propose a novel microgrid model that consists of a wind turbine generator, an energy storage system, a set of thermostatically controlled loads, a set of price-responsive loads, and a connection to the main grid. The proposed energy management system is designed to coordinate among the different flexible sources by defining the priority resources, direct demand control signals, and electricity prices. Seven deep reinforcement learning algorithms were implemented and are empirically compared in this paper. The numerical results show that the deep reinforcement learning algorithms differ widely in their ability to converge to optimal policies. By adding an experience replay and a semi-deterministic training phase to the well-known asynchronous advantage actor-critic algorithm, we achieved the highest model performance as well as convergence to near-optimal policies.

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.

Carbon concrete PAN/lignin-based CF composites are a new promising material class for the building industry. The replacement of the traditional heavy and corroding steel reinforcement by carbon fiber (CF) based reinforcements offers many significant advantages: a higher protection of environmental resources because of lower CO2 consumption during cement production, a longer lifecycle and thus muss less damage in structural components and a higher degree of design freedom because lightweight solutions can be realized. However, due to cost pressure in civil engineering, completely new process chains are required to manufacture CF based reinforcement structures for concrete. The article describes the necessary process steps in order to develop CF reinforcement: (1) the production of cost-effective CF using novel carbon fiber lines, (2) the fabrication of CF rebars with different geometry profiles. It was found that PAN/lignin-based CF is currently the most promising material in order to meet the future market demands. However, significant research needs to be undertaken in order to improve the properties of lignin-based and PAN/lignin-based CF, respectively. The CF can be manufactured to CF-based rebars using different manufacturing technologies which have been developed on prototype level in this study.