PAISE®: Process model for AI Systems Engineering

The challenges arise from the characteristics of AI-based methods: The performance of technical systems that use machine learning (ML) methods can often only be poorly estimated in advance. This makes it difficult to make reliable statements about safety and reliability. This is offset by a large potential benefit: Used successfully, data-driven methods can often make decisions faster and better than would be possible with classically developed methods. In this way, they support humans, relieve them and complement them. In industrial production, ML processes lead to higher-quality and thus longer-lasting products, increase resource efficiency or enable predictive maintenance. In the field of mobility, ML processes can increase driving safety, e.g. by emergency braking in dangerous situations, and thus save lives.

To integrate AI-based components effectively and efficiently into existing or new applications, a systematic approach is essential. Established systems engineering process models are intended for complex technical systems. However, the use of AI and ML brings new challenges that a dedicated process model should explicitly address.

Systematically Developing and Operating AI Solutions with AI Systems Engineering

PAISE® [1], the Process Model for AI Systems Engineering, is specifically designed for the development and operation of AI-based systems. It combines approaches from computer science and data-driven modeling with those of classical engineering disciplines to overcome the challenges. AI Systems Engineering, translated as AI engineering, is what the scientists* call the interdisciplinary approach they have been working on since mid-2020. "With AI Systems Engineering, we want to systematize the development and operation of AI-based solutions. Only if AI methods can be used reliably from an engineering perspective will there be an opportunity to leverage the high value creation potential," says Prof. Dr.-Ing. habil. Jürgen Beyerer, head of Fraunhofer IOSB and the scientific board of directors in CC-KING. "With PAISE®, we have created a set of tools that also provides small and medium-sized companies in particular with a practical guide for achieving this goal."

In development, it can be difficult to estimate the performance of an overall cyber-physical system with AI components in advance. "This means that changes to the high-level design of the overall system may still be necessary at a late stage," says Constanze Hasterok, a scientist at Fraunhofer IOSB and editor of the PAISE® model. "Among other things, this effect occurs when the final ML models are trained with data from real operations. For new developments, however, high-quality data from operation is typically not available until late." For operations, he said, monitoring and, ideally, automatically adjusting ML models is necessary when systems and their environmental conditions can change over time.

In addition, there are difficulties related to staff: As a rule, companies - especially medium-sized ones - do not have their own AI experts. At the same time, managers need to know which AI expertise should be available in the long term for the operation of AI-based systems and how the development process and its interim results are to be evaluated.

Adaptable development through checkpoints

PAISE® divides the development process into seven phases. Project teams in companies must first create a common understanding of the problem, define goals and requirements, and collect solution approaches. The product is then divided into subsystems based on the requirements. This so-called functional decomposition is not final; this is where the agile approach of the model begins. The development of the individual components proceeds cyclically, step by step the subsystems are refined and checked for compatibility. Each run increases the maturity of the overall system.

Checkpoints play an important role in this process, as Hasterok explains: "The checkpoint-based concept of PAISE® enables a flexible development process. ML methods often require an exploratory approach: one develops an ML component on a test basis and empirically checks whether it is suitable for the desired purpose. Other subsystems require a targeted approach, for example according to established systems engineering methods for electronic components. In PAISE®, the individual systems are developed in parallel, each according to a domain-specific appropriate procedure." The checkpoints synchronize the development status of the subsystems early in the project and evaluate their interaction as an overall system. "In contrast to classic milestones, the targets are not firmly defined for all checkpoints at the beginning of the project," she continues. "For example, if it turns out that an ML-based method is not the appropriate tool after all, statistical methods can be used, and their suitability is evaluated in the following checkpoint."

Four end-to-end artifacts create framework

The organization of heterogeneous teams also benefits: stakeholders with different competencies meet regularly and can discuss cross-cutting issues such as safety, cost, or ethical issues. The PAISE® role distribution defines phase-specific functions and responsibilities.

In addition to the distribution of roles, there are three other end-to-end results documentation (artifacts) in PAISE®: the system model describes dependencies of the individual components; the documentation for external audits covers aspects required for review by third parties such as authorities; and the data documentation captures metadata of the data used, such as its source, quality, preprocessing steps, and framework conditions of data extraction.

"By providing systematic methods, we want to encourage companies and developers* to tackle AI projects. PAISE® is a big step forward in this regard. It maps the entire process from conception and data acquisition to operation and maintenance, and addresses all the difficulties that can arise from a technical perspective when implementing an AI project," explains Dr.-Ing. Thomas Usländer, head of department at Fraunhofer IOSB and project manager of CC-KING.

The whitepaper is available for download at www.ki-engineering.eu/paise-en. A workshop on AI Systems Engineering and PAISE® will be held as part of the opening of the Karlsruhe Research Factory® at the end of March 2022. Those interested in attending the training can register in advance by emailing to kompetenzzentrum@ki-engineering.eu.