Trusted simulation models in AI systems engineering

Initial situation

Knowtion GmbH is a successful data science company with expertise in research, development and application of data processing algorithms, e.g. for sensor or machine data. In the context of automated driving, complex virtual development and test environments are already used today, for example to generate vehicle or traffic scenario data, but also sensor data virtually. The goal such an approach is to train or test AI-based algorithms for environment perception at an early stage in virtual driving tests with corresponding sensor models. 

Problem statement

In line with the definition of a model, i.e. the abstraction of reality and the focus on essential aspects, a large number of models and simulation environments are established on the market. Each simulation environment focuses on different aspects or abstractions. Even with different product versions it is not guaranteed that the phenomenological replication remains the same. When using such models in the context of an AI engineering process, e.g. for the validation of neural networks, the question arises how the quality and/or suitability of simulation models can be quantified and persisted in an sustainable way?   

Solution

For the reliable use of simulation models, e.g. in the context of AI engineering, it is necessary to assess the quality of the simulation models independently of the system to be developed, but with respect to the expected phenomena. The complexity and the high degree of interaction between different models make a formal proof of correctness almost impossible, which is why a test case driven approach and integration into a development process-compliant framework was suggested.

QuickCheck results

As part of the quick check, an exemplary virtual development and test environment for automated driving functions was examined and essential models such as sensor, environment and vehicle models were identified. Relevant applications regarding AI system design were identified and concepts for quantifying model quality were outlined.