Module 4 — Building a Responsible AI Testing Strategy

Exercise

Consider 3 AI systems:
A. AI recommends movies to streaming customers.
B. AI determines whether insurance claims should be flagged for investigation.
C. AI assistant helps technicians diagnose problems with industrial equipment.

Identify the top three failures you would worry about for each.

Then determine:
Impact → Likelihood/uncertainty → Exposure → Testing response

Exercise

ShopSmart is an online retailer selling clothing, electronics, and household products.
The company is introducing an AI-powered customer service assistant to reduce the workload of its support staff.
The assistant will:

• Answer questions about products, shipping, and returns.
• Retrieve information from ShopSmart's product catalog and return policies.
• Look up customer order status after verifying the customer's identity.
• Recommend products based on customer preferences.
• Refer customers to a human representative when it cannot resolve an issue.

The assistant uses a large language model (LLM) and retrieval-augmented generation (RAG) to access company information.

Important Business Rules

• Most products can be returned within 30 days.
• Electronics must be returned within 15 days.
• The assistant cannot approve exceptions or issue refunds.
• Customer information must not be disclosed to unauthorized users.
• The assistant must not invent product specifications or company policies.

ShopSmart plans to launch the assistant in four weeks.

Develop a basic AI test strategy for ShopSmart.

Address these six areas:
1. Key Risks - Identify the three most important risks associated with the assistant.
2. Quality Characteristics - Which AI quality characteristics matter most?
3. Acceptance Criteria - How will you determine whether the assistant performs acceptably?
4. Test Approach - What types of testing will you perform?
5. Test Data - What scenarios and data will you need?
6. Monitoring - What should ShopSmart monitor after deployment?