AI governance in the company: The practical guide
AI is already in use in many teams – often without uniform rules. This is exactly where AI governance comes into play: who is allowed to do what, what data is permitted, how do we document decisions? Without this, it gets expensive: incorrect results, compliance risks, loss of trust. The article provides you with a concrete roadmap with roles (CEO/CAIO/IT/Legal/HR), the most important documents and a quick start package.
Key Takeaways:
- AI governance = operating system for AI: roles, processes and evidence make AI in the company secure, legally compliant and measurable.
- Start immediately with 5 modules: AI register, RACI, model/data cards, monitoring & incident handling, quarterly audits.
- Evidence beats gut feeling: release SOPs, audit trail and KPI review (e.g. training rate, MTTR, drift alerts) accelerate go-lives and audits.
- AIMS/ISO 42001 as a structure: Introduce lightweight, scale later; principle → process → data set.
- Training is a compliance lever: Role-based training increases AI literacy, reduces errors and creates acceptance in the teams.
What is AI governance?
Brief definition:
AI governance is the company-wide framework of rules, roles and processes with which AI tools and systems are developed, operated and monitored in a secure, legally compliant and traceable manner. It includes ethical guidelines, risk and effectiveness controls, documentation and regular audits – to protect customers and employees.
What is it for?
It makes decisions transparent, reduces the risk of errors and bias and ensures that the use of AI remains compatible with corporate goals and compliance.
This is what it includes in practice:
- Rules & roles: clear responsibilities for development, operation, testing and release.
- Processes: Monitoring, risk assessment, deviation/incident handling, periodic audits.
- Transparency: comprehensible documentation of data sources, assumptions and model changes.
- Ethics & fairness: continuous checks for bias and unintended effects.
Differentiation from IT governance:
Traditional IT governance focuses on the availability and security of systems. AI governance also controls the characteristics of learning models (e.g. data quality, bias, model drift, explainable decisions) and requires suitable control instruments over the entire life cycle.
Why is AI governance important?
More AI means more responsibility: professionally, legally and towards stakeholders. AI governance ensures traceability, reduces risks and makes value contributions measurable.
What is at stake for companies?
- Law & liability:companies are liable for incorrect or discriminatory decisions made by AI. Governance createsdocumentation, audit trailandclear responsibilities. A basic prerequisite for compliance (including AI Act requirements).
- Operational risks:Unclearrules lead to shadow IT, IP/data leaks, bias, model drift and uncontrolled changes. Governance establishes release and change processes.
- Data & transparency:Without data governance (sources, quality, rights), there is a lack of traceability. Governance ensures model/data cards, logging and audit reports.
- Stakeholder trust:The Management Board, customers, employees and regulators expect fair, explainable decisions. Governance creates transparency and AI competence (see AI literacy).
For companies that are just starting out with AI governance, three key figures are particularly helpful: How complete is the internal AI register – i.e. are all productive use cases recorded? How quickly are incidents detected and resolved (MTTR)? And how many of the systems used are documented in a verifiable manner – for example through model cards or release notes? These three values are often enough to make progress visible – and to identify risks at an early stage.
Principles and standards of responsible AI governance
What it’s really about: principles only work if they are implemented in everyday life. It is of little use if they gather dust as a PDF on the intranet. It is crucial that they become part of concrete decisions, processes and evidence. Governance thrives on routines – not rules. The goal is clear roles, stable processes and comprehensible documentation that work in practice – especially in SMEs. The following sections therefore focus on three things: Why the principle is important, how to implement it in everyday life – and how to make it verifiable.
Empathy: thinking about social impact
AI is not just a technical topic. Good governance always asks: Who does this decision actually affect – and how? A simple tool for this is a brief impact check: what is the aim of the application, who is affected, what side effects are possible? In some cases, a feedback round with stakeholders, such as the works council or the specialist department, also helps. This allows teams to recognize early on whether they should set different thresholds or avoid more sensitive data. This is documented in design notes, review logs or short explanatory texts.
Bias control - ensuring fairness
Machine learning can reinforce existing inequalities if it runs unchecked. That’s why governance involves taking a firm look at biases: Is the data representative? Are there gaps or shifts (drift)? Are certain groups disadvantaged?
Answers are provided by regular health checks – for example with test cases for sensitive characteristics such as age, gender or origin. If something is noticed, a correction process takes effect: adjust data, change thresholds, retrain model.
It is crucial that these steps are documented – for example in bias reports, action lists or change logs.
Transparency - making decisions comprehensible
If you want to answer questions later, you have to document them properly today. This is especially true for AI, where decisions are often difficult to explain.
Transparency is created through a well-maintained AI register (with purpose, data, contact person), supplementary model and data cards (versions, assumptions, limits) and a clean logging of changes.
This means that systems can still be checked months later – for audits or complaints, for example.
Accountability - clear responsibilities
Who is responsible? Who is authorized to release? Who intervenes if something goes wrong? These questions should be answered before every AI rollout – preferably in a RACI matrix.
Typical: the CEO bears overall responsibility, the CAIO coordinates, IT operates, legal/compliance checks, HR provides training. A defined approval process is required before going live, followed by structured incident handling with clear escalation levels.
It is important that this process is also documented – for example through SOPs, release and incident reports.
Get started with our AI workshop
AI governance standards: orientation for implementation & audit
Anyone who takes governance seriously needs a clear framework – not as an end in itself, but as a practical guide in everyday life. These four standards help companies to put principles such as fairness, transparency and control into practice and make them tangible for internal and external audits.
- EU-AI Act provides the legal framework: the higher the risk, the stricter the requirements. If you set up documentation, responsibilities and logs at an early stage, you will save time during audits later on.
- OECD principles offer soft but clear guidelines – for example on fairness, transparency and accountability. They are well suited for ethics guidelines and internal communication.
- ISO standards such as ISO/IEC 42001 help to translate principles into repeatable management processes. A lightweight AIMS (“AI Management System”) is ideal for SMEs – with roles, risk checks and continuous improvement.
- NIST AI RMF (Risk Management Framework) is particularly practical: There is a common thread along the questions: What are the risks? How do we assess them? What do we do about them? How do we monitor progress?
When companies translate these principles into tangible processes – and make them documented, verifiable and scalable – AI governance becomesa real competitive advantage. Not as a brake, but as a structure on which trust, security and progress can be built.
Interim solution for AI needed?
Practical example: Deutsche Telekom anchors Responsible AI in the development process
Many companies wait for laws, some prefer to build ahead. Deutsche Telekom is one of the pioneers: As early as 2018, it relied on its own AI guidelines to make ethical and safe development the norm. Not as a reaction to regulations, but as a conscious decision to firmly build trust and quality into products and processes. This foresight is having an effect: when the AI Act came along, Deutsche Telekom was prepared because governance had long been part of everyday life.
Governance before regulation:
Deutsche Telekom published binding AI guidelines back in 2018 (e.g. people at the center, transparency, safety “kill switch”). The aim was to anchor principles early on in the product and service lifecycle – not just when legal obligations take effect. When the EU Parliament passed the AI Act in 2024, the Group considered itself “prepared” because the guidelines had been incorporated into the development processes for years.
Solution (from principle to routine):
The guidelines were translated into concrete SOPs: mandatory ethics/principles checks in early phases, human-in-the-loop for critical decisions, transparency artifacts (e.g. documentation of data origin and model assumptions) and supplier self-commitments to the AI guidelines. Governance is therefore part of the development and test cycle, not a downstream audit. This increases audit readiness and speeds up approvals
Effect (scaling & maturity):
Public statements by the CEO indicate ~400 AI projects in the company(as of 2024) – an indicator that governance can grow without slowing down innovation paths. At the same time, Telekom 2025/26 is driving forward AI infrastructure initiatives (including with Nvidia), which further increases the need for robust governance processes. The lesson for SMEs is that guidelines alone are not enough – SOPs, artifacts and roles must be integrated into the development cycle.
Levels and structures of AI governance in the company
Not every company needs a complete management system right away. It is much more important that AI governance matches the size, maturity and risk – and can grow with it.
Maturity levels: How AI governance suits the company
Informal:
In many companies, AI governance starts with common sense. Principles are known and decisions are discussed within the team. This works quickly and without much effort, but there is often a lack of evidence and know-how is lost when there are personnel changes. Switching to structured approaches is worthwhile from three ongoing use cases at the latest or if external data and partners are involved.
Ad-hoc:
The first guidelines are already in place here – project-based, with simple logs and clear approvals. The advantages: Less uncontrolled growth, more transparency, a better sense of risk in the team. The weakness: There is still no connecting line between projects. The next level of maturity makes sense when the first models go live or external audits are due.
Formal:
Companies with a higher level of maturity rely on clear roles, defined processes and technical evidence, based on ISO/IEC 42001, for example. This status is scalable, auditable and particularly necessary for critical applications or in regulated industries. The structures are more complex, but help to keep risks and approvals under control.
Organizational: roles, timing and responsibilities
AI governance doesn’t need bureaucracy, but clarity: Who decides? Who checks? And what happens if something goes wrong?
Role model:
The CEO sets the priority and bears external responsibility. Operational control is usually assumed by a CAIO or a designated AI manager. This person coordinates the roadmap, risk reviews and approvals. IT and data engineering take care of operations, platform and security. Legal and Compliance check data rights and governance evidence. HR ensures that all roles involved can build up and demonstrate the necessary expertise.
Committees & routines:
A monthly AI steering committee provides an overview: Status, risks, model changes, special releases. A change board takes care of new model versions and an audit team checks the KPIs, deviations and lessons learned on a quarterly basis. Escalation paths are clearly described in advance – with contact persons, deadlines and a communication plan.
An example: The Mini-RACI – Who is responsible for what?
The AI register is maintained by IT, managed by the CAIO, advised by Legal – and the CEO remains informed. The CAIO is responsible for approvals before going live, legally secured by Legal and IT. In the event of incidents, IT keeps the log, CAIO evaluates and HR pays attention to learnings. Training? HR does that – but the CAIO is responsible for ensuring that they take place across the board.
Technical: Platform, control points and evidence
A few technical basics are needed to ensure that decisions remain comprehensible and secure, even in SMEs.
Core functions:
A model register records which models are in use, who has released them and in which version. Data lineage describes the origin and purpose of the data. Monitoring tools record performance, distortions and drift. An audit trail shows who changed what and when, including notes and tests. Access controls secure operations: only those who really need to can change systems – ideally with the dual control principle.
Controls in the life cycle:
Fixed audit trails apply to every model change: tests, checks and a new release. In case of doubt, critical decisions are backed up by people (“human-in-the-loop“), and a rollback plan allows for quick reversals in the event of problems.
Documents & evidence:
Model and data cards describe training data, assumptions and limits. Test protocols show whether a model is robust, fair and safe. Release notes and a monthly review provide an overview, also for subsequent audits or internal reviews.
Human: competence, behavior and culture
Technology alone is not enough. AI governance lives through people, their decisions and their behavior.
Training & formats:
Managers need an overview, for example in 90-minute sessions on risk and control. Specialist departments learn how to document use cases correctly in half-day workshops. Tech teams receive 1-2 day MLOps training, including test strategies, logs and model cards. Everything is geared towards practice: How do I fill out a form? What do I need to log? How do I deal with deviations?
Measurability of AI literacy:
Target rate: At least 90% of the relevant roles are trained, with refresher training every year. Short tests measure understanding, the evidence ends up in the HR system. This creates a real overview: Who can do what and where does it need sharpening?
Culture lever:
An important rule of thumb: “Default to document“, i.e. document decisions briefly, even in everyday life. After incidents, there are blameless reviews that look for solutions, not blame. And good examples are made visible: a successful register, a comprehensible model card, a clean approval process. In this way, governance becomes part of the culture, not an external obligation.
Who monitors responsible AI governance?
AI governance is not an Excel job for interns, but a matter for the boss. Without clear responsibilities, principles get bogged down in day-to-day project work. This is why management, specialist departments, compliance and audit need to work together. Those who establish this early on create clarity internally and withstand external audits.
Overall responsibility (tone from the top)
The CEO bears responsibility, prioritizes resources and resolves conflicts of objectives. Top management (C-level) ensures that governance is measurable (KPIs, audits) and does not bypass projects. The CAIO (or a clearly designated role) is responsible for operational implementation: roadmap, approvals, risk reviews and reporting.
AI Governance Board (first line of defense)
An interdisciplinary board (CAIO, IT/data, specialist department, legal/compliance, HR) defines standards, threshold values and approval processes. It decides on critical use cases, exception approvals and stop criteria and requires artifacts (AI register, model/data cards, test protocols). Protocols, decisions and owners must be documented in writing.
Risk/Compliance & Internal Audit (second/third line)
Risk/Compliance checks data rights, documentation, bias/drift controls and incident handling, independently of the project team. Internal Audit tests effectiveness and traceability on a sample basis (audit trail, dual control principle, change control) and reports directly to the management/supervisory body – including deviations and deadlines for rectification.
External supervision & evidence
Depending on the industry, there may also be regulatory requirements and certifications (e.g. product/IT tests). Verifiable evidence is expected: current AI register, model/data cards, release notes, monitoring reports, incident reports (incl. MTTR) and proof of training (AI literacy).
Indirect support from outside
External help can simplify and acceleratemany things, especially in the initial phase. Whether it’s setting up clear roles and processes, practical templates for guidelines and decision-making processes or training courses that make teams capable of taking action: An experienced partner brings structure, saves time and helps to avoid typical stumbling blocks. And if things need to move quickly, an interim CAIO ensures that the first projects get off to a safe and clean start without having to wait for the internal setup.
What regulations require AI governance?
AI governance is not an end in itself, but arises from obligations. It is driven by three sources: EU-AI Act, data protection & product liability and industry-specific supervision.
1) EU-AI Act (risk-based framework)
What is required: Traceability, documentation, monitoring, clear responsibilities; the requirements increase depending on the risk classification (e.g. data/model quality, human supervision, conformity assessments).
What that means in practical terms:
- Artifacts: AI register (use cases, purpose, data sources), model/data cards, test/release logs, audit trail for changes, monitoring reports (performance, drift, bias).
- Processes: Risk assessment before going live, human-in-the-loop for critical decisions, change control for retraining/updates, regular audits.
- Roles: CEO responsible, CAIO manages, IT/data operates, legal/compliance checks, HR ensures AI literacy.
2) GDPR & product liability (data & security)
What is required: Lawful data processing (purpose, legal basis, minimization, data subject rights), technical/organizational measures and secure, “fit for purpose” products.
What this means in practice
- Artifacts: Data directory & data flow (source, law, purpose), consent/legal basis documentation, security/test protocols, incident reports.
- Processes: Privacy-by-design, access/least-privilege, deletion/retention concepts, security testing prior to deployment, incident reporting channels.
- Roles: Advise/check legal/data protection & IT security; CAIO ensures implementation in the models.
3) Industry standards & supervision (e.g. finance, health)
What is required: In addition to the obligations mentioned above, there are also sector-specific rules (e.g. testing and reporting cycles, proof of quality/safety, depth of documentation).
What that means in practical terms:
- Artifacts: Sector-specific test reports (e.g. fairness/stress tests), quality-assured SOPs, approval paths with dual control principle, proof of training/competence.
- Processes: Fixed audit dates in the Board calendar, exception/override rules with justification, post-market monitoring (ongoing effectiveness review).
- Roles: Department & Compliance together; Internal Audit checks effectiveness, reports to management/supervision.
Conclusion
Many companies start with use cases before structures are in place. This is understandable, but risky. Governance must grow with the company: with clear roles, comprehensible decisions and reliable evidence. Those who set up a lightweight set early on, such as registers, approval processes and training, create an overview, reduce shadow processes and are better prepared for external audits.
The short conclusion: AI governance pays off – professionally, legally and economically.
FAQs on AI governance
An AI consultancy for SMEs can help to identify the potential applications of AI, develop an AI strategy and support the implementation. This enables companies to work more efficiently, reduce costs, discover new business opportunities and improve customer interaction.
AI can be used in a variety of areas in the midmarket, such as customer service, process automation, and data analytics.
Challenges often include a lack of expertise, high implementation costs, and privacy and security concerns.
AI can be used in a variety of areas in the midmarket, including marketing, sales, human resources, and supply chain management.
AI has the potential to support SMEs even more in the future by providing innovative solutions, increasing competitiveness and opening up new business opportunities.
There are various technologies and solutions in the field of AI for SMEs, such as Machine Learning, Natural Language Processing and Robotics Process Automation.
Yes, there are some best practices and tips that can help SMBs adopt AI, such as employee engagement and ongoing training.
You might also be interested in
From hotline congestion to the AI chatbot How to noticeably relieve your customer service Chatbots are now everywhere: on websites, in customer portals, in apps. Almost all of us have seen a small chat window asking: “How can I help you?” Nevertheless, many companies still have an uneasy feeling. Is
