Frequently asked questions about the PAISE® process model for AI Systems Engineering

• The project manager has a basic knowledge of project management and uses appropriate tools (e.g., for scheduling, budgeting, etc.) as needed.
• There exists a product idea that addresses a business case.
• The project manager has a basic understanding of ML, particularly the following aspects:
  • Dependence of ML on data sets and their quality
  • Dependence of performance metrics of ML models on the quantity of dat
  • Validation methods of state-of-the-art ML models
• The team includes both Data Scientist(s) and ML experts. Experts required from other disciplines will depend on the specific use case.

It is important that you know what you want to use PAISE for: In particular, PAISE addresses the challenge of developing AI-based system solutions in a systematic and unified way.

Internal company requirements and existing processes in product development should already be known to the project manager. The basic concepts of PAISE ought to be understood and a plan should exist how these can be mapped to internal company processes. PAISE trainings can help (see question: Are there trainings). Basically PAISE connects to different process models and culture (see question: Scrum).

Project management needs to have a basic understanding of data-driven development and validation of ML subsystems and be able to assess risks. The competence for the development and especially the validation of ML models needs to be present at different points in the project. Risks are strongly linked to data quality, in particular in AI engineering, as this has a decisive influence on the functionality of the ML components. A basic ML course is therefore a prerequisite for project management and can also be useful for other non-ML experts involved in ensuring a basic understanding.

The ML experts involved in the project should be fluent in the ML procedures in question and have already gained experience in the context of the relevant data.

Of course! Provided that “from scratch” means that there is a first product idea, but no other preliminary work.

In summary, PAISE is a process model for the systematic development of AI-based technical systems with the aim of putting a product integrating an AI solution into production. Hence, at least an initial product idea must be available, and a basic applicability of the AI problem may, but does not have to be, investigated in advance.

Normally, you can directly start with the first phase of PAISE: The description of the goals and the problem understanding. Goals and problem understanding can still be formulated rather vaguely since both will be specified more precisely in the further phases

You might also find Question 1 (Who is PAISE for?), Question 8 (How does PoC/pre-development fit into PAISE?), and Question 4 (What do I need to know before I start PAISE?) interesting!

PAISE can still be used in this case. If no ML component is needed, the procedure of ML component development is omitted. If, however, there are components whose design depends heavily on data, then it can be decided on a case-by-case basis whether the procedure of data provisioning for the development of data sets is nevertheless useful. In addition, it cannot be ruled out that an ML component will be integrated in the future. Then you are already well prepared for it with PAISE.

In particular, PAISE addresses the challenge of developing AI-based technical systems in a systematic and standardized manner. Thus, a clear target definition, i.e., the understanding of which processes are to be structured with PAISE, is important.

Internal company requirements and existing processes in product development projects must be known to the project manager. The basic concepts of PAISE should be understood and a plan should exist how to map them to internal processes. PAISE trainings can help in this matter (see question 5 (Are there trainings?)). PAISE can dock to different process models and cultures (see question 9 (Can I combine PAISE with SCRUM?)).

The project management must have a basic understanding of data-driven development and validation of ML subsystems and be able to assess risks. The competence for the development and especially the validation of ML models should be present at different points in the project. Risks are strongly linked to data properties and data quality, especially in AI engineering, as this has a decisive influence on the functionality of the ML component. A basic ML course is therefore a minimum requirement for project management and can also be useful for other non-ML experts involved to achieve a basic understanding.

The ML experts involved in the project should be proficient in the ML procedures in question and have already gained initial experience in the context of the relevant data.

Question 1 (Who is PAISE for?) might also be interesting for you!

Training courses on PAISE® will be offered in the future via the Competence Center for AI Engineering in Karlsruhe (CC-KING). There will be classical on-site trainings as well as virtual seminars. Training dates can be found on the CC-KING web page as well as on social media such as LinkedIn and Twitter.  If you as a company or association are interested in an event or workshop on AI engineering or PAISE®, you can also contact the CC-KING project management directly.

PAISE defines a process model for the systematic and standardized development and operation of AI-based system solutions. The process model defines detailed processes both at the system and component levels. In order to take the explorative nature of AI solution development into account, a flexible arrangement and a cyclical execution of the different development phases of PAISE is possible. The sequence of phases is primarily controlled by the Refinement and Checkpoint/Evaluation phases.

PAISE is suitable for getting started with machine learning in that the typical development steps are outlined in the procedure model. These are distributed into the process models for ML Subsystem Development and Data Provisioning, in order to allow the necessary documentation but also flexibility, especially in the case of multiple development teams. In the following diagram, the ML-specific steps from these phases are extracted and shown sequentially. This gives an overview of the typical steps and their order. These steps are essential for starting the development of AI-based solutions and are included in PAISE. However, repeated executions or refinements, of the individual steps, which are typically necessary, are neglected. For a detailed description of the steps, please refer to the corresponding phases in PAISE.

The Problem Solution Specification is performed for each ML subsystem in the Refinement phase. The typical Data Curation steps are performed in the phase Data Provisioning as well as the Data Labeling step, in the case of supervised learning. Feature Engineering takes place in the step Integration of Data Sources in the process model for ML Subsystem Development.  The subsequent steps Model Creation, Model V&V as well as Deployment, are also performed in the phases of ML Subsystem Development.

Currently, PAISE focuses on the technical and content-related aspects of AI Systems Engineering. Of course, the definition of goals in PAISE Phase 1 also includes describing the time and budget restrictions. PAISE breaks down the development process into sub-steps. Times can be estimated and monitored individually for these sub-steps. However, the monitoring of the schedule and the budget plan is not explicitly considered in the current PAISE version.

However, the checkpoint concept defined in the development cycle is a suitable tool that can be used to meet time and budget requirements. The concept provides for a comparison between defined requirements and the development status to be carried out and documented at well-defined design points. Feature-based, maturity-based and time-based checkpoint strategies are presented in the white paper.

PAISE addresses the development of an AI model as part of systems for long-term operations. In our view, the development of a proof of concept (PoC) is essential for this. The result of the PoC is the experience and the necessary certainty for the solution approach of the AI procedures to invest greater effort in the development of a system for long-term operation.

The PoC may have already taken place before the start of the PAISE procedure model. Otherwise, it is strongly recommended to anchor the PoC in the milestones for the development cycle.

SCRUM is a framework for agile product and project management. The product is developed incrementally, whereby the long-term goal of the project is approached. Throughout the individual phases, especially with the iterative component development, PAISE has a large overlap with SCRUM while focusing on AI-based solutions. These overlaps make SCRUM ideally suited as tool for project management in combination with PAISE.

As a concrete example for the use of SCRUM in PAISE, the planning and execution of the checkpoint/assessment phase can be mentioned. In PAISE, within this phase, the project management technique to be used is intentionally left open. For example, the typical SCRUM product backlog is very suitable as a basis for planning. The refinement phase can be broken down for SCRUM sprint planning and for the creation of a sprint backlog. In the Checkpoint/Evaluation phase, the fulfillment of a Sprint (SCRUM Sprint Review) can be evaluated, and the next sprint can be prepared in the subsequent refinement phase. Hence, PAISE and SCRUM complement each other very well.

Both PAISE and Crisp-DM propose an approach to metrics-based development of an ML application. However, PAISE stretches the scope further than Crisp-DM. Crisp-DM focuses on creating a business case based on data and models built on top of it. However, Crisp-DM does not offer a standardized procedure for the integration of the developed ML models into complex systems and their mutual influence. Nor does it address the quality criteria that must be applied to the data in order to ensure that the requirements for the models are met. Those points are realized in PAISE.

In the context of AI systems engineering, Crisp-DM is well suited to evaluate the potential provided by exploiting some given data with ML techniques, i.e., to make a proof of concept (PoC). However, in order to develop a product that is finally delivered to customers, Crisp-DM lacks fundamental points, such as the validation of the ML models in interaction with other components and the maintenance of the ML models over the product lifecycle. Those are addressed in PAISE by the Checkpoint concept and the ML model monitoring during operation.

Machine learning is a special form of Artificial Intelligence. Learning models from data has many advantages and is thus the dominant form of AI today. In fact, however, there are many procedures which work on models known in advance and make intelligent decisions without having learned from data. Some development steps are omitted in this case compared to ML-based methods, such as data collection and preparation. However, the classical challenges of systems engineering still remain, which are addressed in PAISE as well. Therefore, the application of PAISE is justified even if AI is used without ML.

First, it should be noted that PAISE was not designed for high-risk applications containing regulative requirements that are of particular interest within the AI Act!

The AI act is the main source of regulative requirements related to the development of AI systems in the European Union (“Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) […]”) and is expected to become law in 2024 or 2025. It is part of the New Legislative Framework of the EU and sets up a wide range of requirements for the development and operation of AI systems, especially but not exclusively systems in “high-risk” applications. These comprise transparency obligations during development, but also requirements on datasets such as fairness and absence of errors. For an application of PAISE, such exterior requirements must be considered in several phases. During Goals & Problem Specification it must be determined which regulative requirements are derived from the project goals. In the phase Requirements & Solution Approaches specific approaches to addressing these requirements must be specified and concrete metrics must be defined. Checkpoints are then used to evaluate these metrics and determine the level of compliance with respect to the regulative requirements.

In this context it must be noted that the implementation of the AI Act intends that those regulations for product development (the typical use case for PAISE) should continue to be drawn from sectoral certification and approval processes, which are to be updated to integrate requirements from the AI Act. Hence, developers must especially consider regulative requirements concerning the certification of the intended product in question and verify whether this product certification process is compatible with an application of PAISE.

Beyond the AI Act, especially the “regulation 2019/881 of the European Parliament and of the Council […] on information and communications technology cybersecurity certification […]”, the Cybersecurity Act can be relevant for the development of AI systems, which is in effect since 2019. Here, PAISE can contribute to identify cybersecurity misuse risks in AI systems earlier and more systematically, if these requirements are identified in the phase Goals & Problem Specification (see above) and are evaluated regularly during checkpoints.

Yes, PAISE can also be used for the further development of existing systems. One of the guiding questions in Phase 1 is specifically designed to query the state of existing systems: "What is the initial state?".

It is irrelevant whether or not the existing system already contains ML components. Similarly, it is irrelevant whether further development includes ML components or not (see question "What if I find out during the PAISE procedure that I don't need ML at all?").

New components are inserted into the overall system by extending the functional decomposition, developed in the development cycle, and integrated and tested within the already existing overall system.

PAISE® supports the introduction of heterogeneous requirements, which are to be identified initially during the phase Goals & Problem Specification and to be specified concretely during Requirements & Solution Approaches. During checkpoints in the development cycle, these requirements are to be evaluated regularly. Due to the wide range of applications and the complexity of AI systems, those requirements often and especially involve human-centric and ethical system design. To this end, a methodology based on the IEEE 7000 “IEEE Standard Model Process for Addressing Ethical Concerns during System Design” can be applied. A harmonization between this approach and PAISE® is currently explored and will be detailed in future versions.

PAISE® was developed by the CC-KING partners in particular on the basis of experience and requirements in the use of AI processes in the application domains of industrial production and mobility systems. In these domains, application scenarios were also carried out via so-called QuickChecks and PAISE® was thus validated. This does not exclude that PAISE® can also be used for other technical application domains, such as healthcare, medicine, circular economy or logistics. Ultimately, PAISE is potentially suitable for all domains with a need for systematic development and reliable deployment of AI methods.

PAISE is primarily suitable for product development projects in which several components are to be developed in parallel in different teams or even across organizations. Here, PAISE structures collaboration in a consistent and targeted manner.

For small teams and less complex systems, e.g. if components are developed by single persons, PAISE can provide orientation, but causes a considerable additional organizational effort if the procedure is strictly followed.

Yes, PAISE is very well suited for pre-development or a proof of concept. For this purpose, all phases of PAISE are run through and are documented, except for the Operation & Maintenance phase, which is omitted in this case. In the Handover phase, the results are delivered internally within the organization. After pre-development, the actual product development can be realized by going through PAISE again. The results of pre-development must then be included in all phases of PAISE.