Experimental Evaluation of a MIMO Radar Performance for ADAS Application

Abstract

Among the sensors necessary to equip vehicles with an autonomous driving system, there is a tacit agreement that cameras and some type of radar would be essential. The ability of radar to spatially locate objects (pedestrians, other vehicles, trees, street furniture, and traffic signs) makes it the most economical complement to the cameras in the visible spectrum in order to give the correct depth to scenes. From the echoes obtained by the radar, some data fusion algorithms will try to locate each object in its correct place within the space surrounding the vehicle. In any case, the usefulness of the radar will be determined by several performance parameters, such as its average error in distance, the maximum errors, and the number of echoes per second it can provide. In this work, we have tested experimentally the AWR1843 MIMO radar from Texas Instruments to measure those parameters.

1. Introduction

In detection systems for autonomous driving, radar is considered an essential element, along with cameras and, potentially, lidar [1,2,3,4,5,6,7,8]. The role of radar is relevant in providing an estimate of the distance to objects captured by the cameras and also for detecting other vehicles or obstacles in adverse weather situations, such as heavy rain or fog. In these situations, cameras and lidar lose much or all their effectiveness. When it comes to providing depth to images, lidar-type sensors are even more effective than radar, but at the cost of a much higher price, greater size and mechanical complexity, and a significant computational cost. In this work, we have tried to experimentally estimate the value of some parameters of interest related to radar performance, such as the effective false alarm rate, typical distance error, maximum errors, and the number of received echoes that can be expected per unit of time.

We have used a Texas Instruments (TI) AWR1843 radar, which is a Multiple Input Multiple Output (MIMO) Frequency Modulated Continuous Wave (FMCW) radar designed to work in the 77 to 81 GHz band. It consists of three transmitting and four receiving antennas. This model is one of the most complete, reliable, and updated developed by this company. The processing of the signals, after mixing with in-phase and quadrature signals carried out by the device itself, has been carried out off-line on a standard computer after transferring the raw signals using a DCA1000EVM card, also from TI. We have used a single transmitting antenna for the measurements, so in this case it has been operated as SIMO, single input, multiple output. This mode of operation allows position information to be obtained in the horizontal plane; nevertheless, target height is not available. The signal processing and visualization program was developed in MATLAB (R2023b). The radar has been operated with parameters optimized for short range, up to 25 m in our measurements. The scenarios include pedestrians and motorcycles, and static targets have been excluded from the analysis. Some other works may be found in the literature that use the same model or others commercial MIMO radars [9,10,11,12,13,14,15,16,17,18], always for short distances, although they do not provide enough measurements to have a complete idea of the performance of those models. In Figure 1, the scenario where the measurements have been carried out is shown.

In Section 2, we present the radar configuration, which allows quite a bit of flexibility, as well as the types of experiments carried out. In Section 3, these experiments and the corresponding echo maps obtained are shown. The first of these experiments was to ensure that the precision of the distance measurement was sufficient for our purposes, which was performed over a grid marked in the ground with the aid of a laser distance meter. Some conclusions for the effective probability of a false alarm, the mean error in distance, and the maximum error obtained are presented. We prove that all these parameters are highly-dependent on the level chosen for the adaptive threshold, which is set by means of the well-known CA-CFAR algorithm. In Section 4 and Section 5, a discussion of the results and their significance is realized.

2. Materials and Methods

In each scenario, the targets that carry a certain speed (radial with respect to the radar) have been analyzed separately to improve target separation and minimize background noise. That is, the speed estimation, obtained thanks to the Doppler shift of the signals, is carried out with the sole purpose of selecting particular targets, pedestrians [19,20] or motorcycles in our case. This way of proceeding is the natural one after performing a two-dimensional FFT of the received signals. It is the standard method of analysis, as shown in the bibliography provided by TI and in other references [21,22,23,24].

The emitted signal is made up of FMCW signals, where the frequency increases linearly over time in each chirp. The slope selected for the temporal frequency variation was $S=20 \mathrm{M}\mathrm{H}\mathrm{z}/\mathsf{\mu }\mathrm{s}$. The sampling frequency in all cases has been $f_s=12 \mathrm{M}\mathrm{H}\mathrm{z}$ and a number of samples were taken per chirp $N_{sc}=256$. Signals are processed in blocks, called frames, with $N_{cf}=128$ or $N_{cf}=256$ chirps per frame in this work. The chirps are separated by a certain interval, which can be modified [25].

When the $N_{sc}$ samples of the $N_{cf}$ chirps are received, that is, a frame, the samples are ordered by rows in a two-dimensional matrix $N_{sc}\times N_{cf}$. A 2D-FFT is performed, and the range-doppler map (or range-velocity map) is obtained. From it, you can select the columns of interest (targets with a certain speed) and the different targets present with that velocity. In our case, we have analyzed the targets detected at all present speeds except zero speed; that is, static targets with respect to the radar are ignored. This was performed because the measurements were carried out in a parking lot with parked vehicles, which were of no interest to the study.

The selection of received echoes, both in speed and distance, has been carried out by applying a CA-CFAR (cell averaging constant false alarm rate) algorithm [26,27,28,29,30]. The CFAR algorithm creates an adaptive threshold to discriminate the peaks in the FFT function in order to select the more likely echoes in the scene. In the selection of speeds, an undemanding value has been set, with a probability of false alarm $PF{A}_{vel}=0.2$, or $0.3$ in order not to lose echoes, while in distance, the results obtained for different speed values have been evaluated, with $PF{A}_{range} \in (0.1–0.001)$. A small value of PFA produces a more selective threshold.

The parameters of interest were the number of false targets in each scenario, which gives us a measurement of the final effective false alarm probability ($PF{A}_{eff}$), the error in estimating the distance of the targets, the maximum errors in distance, and the number of useful echoes obtained per second.

The average error in distance, as well as the maximum errors, have been evaluated by measuring the error in each echo considered good by the algorithm. To do this, it is not enough to evaluate the error of the echoes with respect to the final trajectory traced by the target; it is necessary to track the position of each echo and compare it with the estimated instantaneous position of the object. In our case, we did not have a more accurate system capable of providing the instantaneous trajectory of the targets, so an estimate of the likely instantaneous trajectories has been built from the radar echoes themselves. The moment in which the echo appearing on the screen is assigned to a particular target is sometimes critical because, in reality, you cannot be certain that this echo really comes from that target. This fact adds a point of uncertainty to the process, which, in any case, is inevitable. The recorded videos of the different scenes are used to visually validate the movement of the targets and estimate the temporal duration of the measurements.

3. Results

3.1. Calibration of the Radar

The first measurements carried out were intended to ensure the accuracy of the radar in determining the position of the targets. We were interested in doing that, not in an anechoic chamber (where, moreover, the maximum distance is quite limited), but rather in the same scenario chosen for the other measurements. To this end, we draw a grid on the parking floor with the help of a laser rangefinder, with points located every 5 m in one direction and every 3 m in the perpendicular direction. The global precision in the position of the marked points was of $\pm 0.1 \mathrm{m}$. Several measurements were taken with a turning retroreflector sited over those grid points. The set of results can be seen in Figure 2.

In that figure, the points on the ground are marked as red circles. Squared blue points are the set of echoes recovered during the measurements. Although some measurements were very clean and accurate, with all the echoes recovered on the reference red point, others were not so clear. In the case of the closer point to the radar, in coordinates (0, 3.8), some false targets were generated; actually, all the isolated light blue squares are present in the figure near the $x=0 line$(about eight false targets). The other, less accurate set of points corresponds to the reflector in the coordinates (−0.2, 13.5). In this particular case, the problem could be a metallic pillar situated very close to that position, a little close in the direction towards the radar. We think there were double reflections from that pillar, provoking an error in the determination of the reflector position. With the exception of these two circumstances, the mean radar error is 0.2 m, and the maximum error is, in one case, 0.6 m.

3.2. Effective Probability of False Alarm

In the first scenario with real targets, there are three pedestrians moving in different directions and at different distances in front of the radar. All of them describe straight trajectories at approximately constant speeds. The captured scenes were also taken on video. To evaluate the error made in the processing of the different echoes, a most probable instantaneous trajectory is estimated with a constant speed, or constant in sections, for the target. The echoes are inspected one by one, selecting those that most likely come from the target under study. In this type of analysis, there is always an intrinsic uncertainty since it is not possible to know if an echo comes from the target that is being specifically tracked, from a nearby target, or if it is a false target generated by noise. Faced with this, the only practical procedure is to track with an appreciable number of echoes along the probable instantaneous trajectory, as we have tried to do in this study.

Figure 3 shows the trajectory of the three targets in this first scenario, with all the echoes selected by the analysis program. As stated, the selection of speeds was carried out with a value $PF{A}_{vel}=0.2$, while the result in the figure is the one corresponding to the value $PF{A}_{range}=0.05$. The radar is located at the origin of the coordinates.

The figure shows the echoes that most likely belong to one of the targets, located along the path of each pedestrian. There are also false targets, some of them generated by noise peaks or coming from echoes affected by noise, so that their position has been incorrectly estimated. Some of the echoes would fall into the doubtful category, as those framed with a solid blue line, since the observer or a subsequent echo evaluation system could consider them valid, assigning them to a particular pedestrian. It is worth noting that to perform the task of classifying an echo as valid, doubtful, or clearly erroneous, it is not enough to simply look at the final echo map shown in the figure. To make this classification, it is necessary to track it in time during the course of the experiment so that the position of the received echoes can be compared with the real or estimated instantaneous position of each real target.

The second scenario was composed by two pedestrians and a motorbike, describing straight paths and also the return of all three to their initial positions. As in the first case, the raw data obtained from the radar were analyzed again using $PF{A}_{vel}=0.2$, for a first selection of targets, in velocity, and with several values, increasingly small, for the parameter $PF{A}_{range}$. Figure 4 shows the global set of echoes received from the radar during this experiment.

In the particular map of Figure 4, a larger number of echoes, valid and false, can be seen in comparison with the first scene. This is because a very low-demand PFA has been used, but also because the duration of the scene is longer.

Finally, the third scene is a motorbike describing rounds in front of the radar with an approximate constant speed. The set of echoes received is shown in Figure 5.

We observed a concentration of false targets near the radar in a position sixty degrees to the left, as is marked with a dashed contour in the figure. This phenomenon of concentration of echoes where there is no real object did not happen in the other two experiments, and we do not know why they appear. It could be related to some kind of double reflection of the signal or, perhaps, with the reflection in the wheels, which can introduce an overlap with their own Doppler shift.

The first analysis carried out in the experiments was to evaluate the number of false targets detected based on the PFA value used in the detection of targets for each of the selected speeds. That result is shown in Figure 6. It has been defined as follows:

$$PF{A}_{eff}=\frac{nº of false echoes}{nº of received echoes}.$$

(1)

In this calculation, the class of doubtful echoes has been omitted, assigning them to the set of valid echoes or false echoes in each case.

The choice of a higher or lower value for the PFA parameter depends on the necessity of having enough echoes. So, in situations where the signal-to-noise ratio is low, even admitting an important number of false echoes, the choice could be $PF{A}_{range}~0.1$. The alternative is minimizing that number of false targets when the signal-to-noise ratio is not bad. In this situation, a value $PF{A}_{range}~0.01$ would be more appropriate. In our case, having enough echoes received per second and avoiding, as much as possible, the number of false echoes seems to be the best strategy. The behavior of the echoes in the three scenarios is slightly different. In scenarios 1 and 2, the number of false targets is lower than in the other, and with a value of $PF{A}_{range}=0.01$ or $PF{A}_{range}=0.005$, the detection of targets is quite clean. In scenario 3, however, there are a significantly larger number of false targets, and a value as low as $PF{A}_{range}={10}^{-3}$ should be used to sufficiently filter out false alarms. It must be considered, however, that an unnecessarily small value of this parameter would also cause a significant loss of true echoes. On the other hand, we have fixed most of the time the other variable threshold in the velocity axis, with $PF{A}_{vel}=0.2$, but it must be said that this value could also be relaxed.

Figure 6. Effective probability of false alarm in the three scenes, obtained as a function of the PFA parameter chosen for the detection of targets at a distance. For the selection of velocities, the value $PF{A}_{vel}=0.2$ was used in all cases.

Figure 6. Effective probability of false alarm in the three scenes, obtained as a function of the PFA parameter chosen for the detection of targets at a distance. For the selection of velocities, the value $PF{A}_{vel}=0.2$ was used in all cases.

3.3. Mean Error in Range

Each of the pedestrians in the first scene was also followed separately, calculating the average error of the position of the echoes that, with high probability, are coming from the target under study. As explained, the echoes that come from the target (or pedestrian) under test have to be selected manually, and the position of those echoes must be compared with the corresponding points on the estimated instantaneous trajectory of each target.

In Figure 6, the obtained mean error in range is shown for the three pedestrians. The basic limit of the system’s spatial resolution is given by the known expression

$$∆r=\frac{c}{2S{T}_c},$$

(2)

S is the slope of the frequency-temporal variation, c is the speed of light, and $T_c$ is the duration of a chirp. In our case, we have $S=20 \mathrm{M}\mathrm{H}\mathrm{z}/\mathsf{\mu }\mathrm{s}$ and $T_c=N_{pc}/f_s=21.33 \mathsf{\mu }\mathrm{s}.$ And it results $∆r=0.35 \mathrm{m}$.

As it can be seen in Figure 7, the mean error in the localization of the targets does not depend critically on the selected value for the $PF{A}_{range}$, although it is clearly better for low values. This behavior comes from the fact that the targets have been detected correctly most of the time. However, on some occasions, echoes affected by noise or false targets have been considered good. This can be seen in Figure 8, where the error in the localization of one of the pedestrians has been represented as a function of the distance to the radar in two cases.

There are two appreciable differences between the two cases shown in the figure. For the higher value of $PF{A}_{range}$ a quite large number of echoes have been accepted by the system/observer as coming from the target, including some of them with an important error (~2.5 m). For the low value of that parameter, however, the number of valid echoes is much lower, as is the maximum error (<1 m). It must be remembered, however, that the selection of the ‘correct’ echoes has been carried out visually, considering what the temporal processing of the signals was showing on the screen. This adds a point of arbitrariness that, on the other hand, cannot be avoided. Nor if the task is performed by an algorithm or an artificial intelligence application, because the system does not know previously the number, nature, trajectory, or speed of the targets.

The evaluation of errors has also been made for the two paths of the motorbike in scenario 2, as shown in Figure 9. As the motorbike has an important horizontal dimension, the position mean error is higher than in the case of pedestrians, although, on the other hand, in this particular experiment, the radar never has a complete vision of that horizontal dimension of the vehicle due to the angle of the paths with respect to the radar position (see Figure 4).

To end this section, in Figure 10, the range error measured during the evolution of the motorbike in scenario 3 is shown. Actually, the trajectory with which the position of the echoes is being compared has been deduced from the echo map itself, averaging the position of the echoes as they appear in time in order to obtain a smooth and continuous path. It could be seen that important errors are reported in some specific positions of the motorbike, which correspond to the moments when the motorcycle is traveling perpendicular to the line of sight of the radar (see Figure 5). As the length of the motorcycle is somewhat more than 2 m, the echoes can come from different parts of the vehicle, while the errors are merely calculated with respect to the position of the hypothetical center of the vehicle. Of course, then, the errors shown in the figure are only a coarse approximation, and most of the echoes are certainly from one or another part of the target.

3.4. Number of Echoes per Unit of Time

Finally, we have performed a count of the useful echoes available from some of the targets. It has to be said, anyway, that we have not optimized the timing of the radar, but we were only interested in the variation of the number of echoes with the PFA parameters. The most significant point is that the number of valid echoes diminishes strongly with a low value of the parameter $PF{A}_{vel}$, as can be seen comparing the two plots in Figure 11. If the requirements of the system are about 25 echoes per second (from each target), this parameter, $PF{A}_{vel}$, will have to be relaxed accordingly. As it has been said before, a too low value of PFAs provokes not only the decrease of false echoes but also of the good ones.

As it can be noted in Figure 11, the number of good echoes from different targets follows a different behavior. That is due to the different trajectories of the pedestrians. The distance of pedestrian M with respect to the radar has a low variation along time, so that the number of received echoes from her suffers a variation only for the lowest value of PFA. Pedestrian A is progressively approaching the radar, and most of the echoes produced have a very good signal-to-noise ratio. As a consequence, the decrease in the PFA value has little effect. On the contrary, pedestrian F is in the worst situation to be detected: at the beginning of his path, the distance to the radar is larger than for the other two targets, and, at the final state, he is nearing going out of the field of view of the radar. So, the echoes produced by him are, generally speaking, of lower amplitude. Furthermore, the echoes of the farthest pedestrian are occasionally intercepted by the other two when crossing the line of sight of the radar.

Also worth noting is the fact that an appreciable number of frames sent by the radar have no response at all or fewer responses than the number of real targets. This is an experimental verification of the three scenarios shown in this work.

In Figure 12, the time intervals to take into account in the configuration of the TI device are shown [25].

As it was explained above, each chirp was sampled with a frequency $f_s=12 \mathrm{M}\mathrm{H}\mathrm{z}$, up to 256 samples per chirp. So, we have the effective duration of the chirp $T_{acq}=256/12 \mathrm{M}\mathrm{H}\mathrm{z}=21.33 \mathsf{\mu }\mathrm{s}.$ However, the real period of the signal is not related to that value, as seen in the figure. The total period will be $T_c=T_{idle}+T_{ramp}$. In our experiments, we took $T_{idle}=100 \mathsf{\mu }\mathrm{s}$ and $T_{ramp}=60 \mathsf{\mu }\mathrm{s}$. (On the other hand, we set $T_{ADCst}=6 \mathsf{\mu }\mathrm{s}$ and $T_{TXst}=0,$ which were the default values in the software pack mmWave Studio (https://www.ti.com/tool/MMWAVE-STUDIO), provided by TI). With these values, the expected number of frames per second is about 49 frames/s with 128 chirps per frame and about 24 frames/s with 256 chirps per frame. These numbers include frames without response and frames provoking false echoes. It has to be said, however, that we were not worried about the time the radar takes between frames, which is also an adjustable value. Actually, we tried to avoid the radar sending a very high number of echoes per second in the presented measurements. Adjusting the time between frames in other experiments, we observe that the radar can provide much more echoes per second than the one presented here.

4. Discussion

The main idea of the work was to evaluate the characteristics of the TI AWR1843 radar in terms of the error committed in distance, the number of false echoes, and the number of valid echoes per unit of time. Experiments were performed with pedestrians and motorcycles, with the idea that they are the targets that could eventually go unnoticed by the radar.

In the analysis, a double adaptive threshold has been used, one on the speed axis and another on the distance axis, for each speed selected by the first, and both with the CA-CFAR method. The results show that it is a good strategy and that it provides a level of intuitive control over the number of false echoes.

Initially, the experiments were carried out by setting the PFA for determining the threshold at speeds with a loose value of 0.2. It was later found that this value actually eliminates too many potentially valid echoes, and a higher value, $PF{A}_{vel}=0.3$, has been found to be more suitable for obtaining a higher echo rate. On the other hand, in our experiments, there is no appreciable difference by using $Ncf=256 \mathrm{o}\mathrm{r} 128$, so $Ncf=128$ is a more economic value.

The maximum error of the radar in estimating the position of a target turns out to be a difficult parameter to quantify since it depends critically on what criterion is used to assign an echo to a specific target.

The average error has turned out to be very acceptable, taking into account that interpolation techniques can still be included and, in this work, with the exception of zero-padding in the angle-FFT, have not been used.

However, it must be said that making meaningful statistics from realistic experiments with FMCW MIMO radar seems like a not possible task, at least at this initial level. There are two factors that make this task very difficult, or simply impossible. The first one is, as it was pointed out, that there can be an important number of false or poorly positioned echoes, and the final statistic will depend strongly on the criterion adopted in the selection of valid echoes and, also, on the PFA values selected in the CFAR thresholds. The second reason is that in large targets, the echoes can come from different parts of the targets, making it necessary to have another sensor, e.g., a camera, to inform the size of the vehicle and validate the accuracy of the selected echoes.

Finally, we did not set out to estimate the maximum distance at which our radar can capture targets, although, in other measurements not presented in this work, pedestrians have been detected at about 30–35 m and, eventually, some cars were perfectly detected at 80 m.

5. Conclusions

We have carried out several experiments with a MIMO radar AWR1863 from Texas Instrument in realistic scenarios with pedestrians and motorcycles in order to evaluate some critical points of its performance, such as the number of false targets, the accuracy in the localization of the targets, and the number of valid echoes received per second. These parameters are important for the usefulness of the radar itself and also for the use of the obtained data to make a fusion with the images captured by the cameras in a more complete ADAS system. We use a double CFAR filter, one for separating velocities and the other for separating targets by distance. We have used large values in the first of them, with $PF{A}_{velocity}=0.2–0.3$, and more restricted values for the second, with $PF{A}_{range}$ between 0.1 and 0.001. The results show that there is a compromise in selecting the value for the two parameters because a too small number diminishes not only the false targets but also the good ones.

The accuracy in range is generally good, although a small value for $PF{A}_{range}$ helps to avoid disturbance in the measurements due to false or poorly estimated targets. The mean error in distance along the experiments is about 0.25 m for pedestrians and 0.5 m for the motorbike. We maintain some doubt about the usefulness of this radar model due to the fact that it is difficult to detect a pedestrian at a distance of more than 30 or 35 m. This maximum distance seems too short in a real traffic situation. The maximum error in the localization of targets cannot be estimated properly because it is not possible to discriminate between false targets and positions erroneously calculated due to the present noise. Moreover, assigning a singular echo to a particular target depends on some chosen criterion, probably at a further stage in the complete detection algorithm, but always with a certain degree of arbitrariness. We have noticed that the number of valid echoes per unit time falls significantly in more demanding measurements, i.e., with a lower PFA value chosen. In other measurements, however, it has been determined that this model can provide easily 50 frames per second and more. Finally, real-world statistics for the characterization of the mean error in distance and its variance could only be made in a more advanced state of development, when a camera was available that allowed grouping echoes in large targets, or, at least, in a later stage, using a neural network, performing the task of identifying targets, grouping echoes from them, and giving the position of these targets along their temporal trajectory. So, as a final conclusion, the performance of some neuronal network deducing consistently targets from the echoes or a fusion data algorithm using the images of a camera would be crucial to determining the validity of the system in a real ADAS system. This radar could be a candidate for such a device, with the only objection that it should be able to detect small targets (pedestrians or middle-size objects) at a greater range.