Assessing and improving ADAS effectivess

As Advanced Driver Assistance Systems (ADAS) such as lane assist become common in passenger vehicles, evaluating the real-world effectiveness of ADAS aims to objectively assess system performance and driver perception of these safety technologies.

ADAS play an important role in improving road safety by assisting drivers during routine and safety-critical driving situations. The iMOVE project Evaluating the real-world effectiveness of ADAS aimed to assess their real-world effectiveness through the development of a comprehensive experimental framework for evaluating production ADAS features.

The project was a partnership between Insurance Australia Limited (IAG), the University of Sydney and iMOVE. IAG currently has a program for evaluating ADAS effectiveness, but it relies on subjective measures and lacks the objective data required to rigorously evaluate and compare the performance of systems installed in different vehicle makes and models. This project addresses this need by creating a proof-of-concept system to provide empirical evidence of ADAS’ safety benefits.

Background

While manufacturers report the capabilities of their ADAS technologies, there remains a need for independent and repeatable methodologies that objectively evaluate system performance while also capturing the driver’s perception of these technologies.

This project addressed this need through the development of a comprehensive experimental framework for evaluating production ADAS features under representative driving conditions.

This includes developing a proof-of-concept sensor platform to collect naturalistic driving data for systems such as Lane Departure Warning (LDW) and Lane Keep Assist (LKA), replacing subjective assessments with empirical metrics.

A key focus is to analyse driver interaction with ADAS technologies. Despite their safety benefits, the adoption and effective use of these systems remain limited. The research investigates barriers to driver participation and provides recommendations to improve user acceptance and maximise safety benefits.

Data collection system design

The primary objective was to design, implement and validate a complete ADAS evaluation platform capable of simultaneously collecting objective vehicle performance data and subjective user feedback.

The platform captured comprehensive vehicle, driver and environmental data essential for generating objective ADAS performance metrics. It integrated multiple synchronised sensing modalities, including GNSS, inertial measurements and side-view cameras.

Custom survey and interview instruments collected driver experiences and interactions with ADAS systems. Data collected during the user study was analysed by the USyd team to validate metrics developed for ADAS performance assessment.

This included correlating sensor data with user feedback to identify performance patterns and barriers to driver engagement. The analysis informed recommendations for improving the design of ADAS technology and performance assessment frameworks, as well as strategies to improve driver’s use and understanding of these safety systems.

Pilot study

The pilot study focused on covering drive trials with two participants and a follow-up interview with one of them.

The work involved one IAG researcher running the formal study using a detailed protocol sheet; one Australian Centre for Robotics researcher monitoring the Data Collection System toolbox and three USyd design researchers, who developed the protocol and observed the session from the back seat.

The pilot study findings confirmed the feasibility of the overall procedure and the effectiveness of the planned data collection methods. Refinements were made to the study procedure and several technical fixes were implemented. Questionnaires and interview questions to be used in the formal user study were also revised to improve clarity and data collection quality.

User study

The user study involved 12 participants driving under various real-world conditions to evaluate the performance of the LDW and LKA systems using the developed test protocol. The study generated quantitative data from the sensor platform and qualitative data from participant feedback to assess ADAS effectiveness and usability.

During the 20.7 km drive, participants were asked to drive naturally. Both test vehicles had LDW set to default on. For LKA, the study facilitator in the front passenger seat prompted participants to enable it for the purpose of the test, but they were free to turn it off at any time.

Participants were encouraged to comment on their in-situ experience, while the study facilitator also asked questions based on the situation. Participants wore a wireless microphone clip to capture their comments. Each participant completed the same route in both vehicles, with the order of vehicle exposure counterbalanced to control for any potential order effects.

After each test drive, participants were provided with a set of surveys that included five questionnaires investigating their trust, task load, sense of control, user experience and situational awareness. They were also asked a few short questions about their general opinion and overall experience after completing the questionnaires.

Objective evaluation

Door-mounted cameras capture the wheels and road surface to provide continuous visual information about the lateral position of the vehicle within the lane.

The process begins with a camera calibration step, establishing the relationship between real-world distances and pixel measurements in the image frame. This calibration allows the conversion of pixel-based observations into physical distances.

A comprehensive set of algorithms, capable of interpreting multiple aspects of the driving scenario, were developed to obtain objective ADAS measurements and performance metrics. These algorithms enable a more holistic and reliable assessment of how LDW and LKA perform under various real-world conditions.

Conclusions

This project successfully developed and validated a comprehensive framework for evaluating production ADAS through the integration of objective vehicle performance measurements and subjective driver assessment. Both approaches demonstrated consistent findings, which will uplift ADAS insurance evaluations.

A key achievement was the design and implementation of a modular data collection platform capable of synchronising information from multiple sensing modalities. This prototype allows for objective testing of ADAS systems, and development will continue to add more functions.

The results also demonstrate that production ADAS exhibit measurable differences in performance and user perception, despite offering similar functionality. The proposed evaluation framework successfully captures these differences in a structured and repeatable manner.

Future work should consider larger and more diverse participant cohorts to improve statistical significance and better represent the wider driving population. Expanding the evaluation to include additional vehicle manufacturers, different ADAS implementations, varying road geometries, adverse weather conditions and nighttime driving would further strengthen the generalisability of the findings.

Expected roject impacts

This project demonstrates the value of collaboration in advancing vehicle safety and creating evidence-based approaches to emerging technologies.

Assessing ADAS effectiveness is important to ensure these systems deliver meaningful safety benefits and to better understand where further improvements may be needed.

Shawn Ticehurst, Head of the IAG Automotive Research Centre

Final report

A final report has been produced and is being used internally.