Author avatar

Arsh Rai

Technical Writer | Content Designer

  • Home
  • Blog
  • Work
  • Music
  • About
  • Contact
LinkedIn GitHub Medium Instagram youtube Spotify

Work

Access Required

This section contains confidential work samples.
Please enter the password to continue.

Incorrect password. Please try again.

I design how AI systems behave

What I Design

I design how AI systems:

  • interpret ambiguous input
  • communicate intent and output
  • handle uncertainty and failure
  • integrate into real human workflows

My work focuses on making AI behavior:

  • clear (understandable outputs)
  • predictable (consistent patterns)
  • trustworthy (controlled, explainable, human-validated)

Design Principles

  • Clarity over cleverness — AI outputs must be immediately understandable
  • Constraint over freedom — systems perform better with defined boundaries
  • Explainability over opacity — users should understand why something was generated
  • Human authority over automation — final control always remains with people
3
Case Studies
2
AI Systems Built
1
UX Research Project
Jump to Case Study
  1. Claude Code “Drift-Check” Agent — Designing Trust & Verification in Autonomous Systems
  2. Release Notes “AI Bridge” — Multi-Agent Language Orchestration at Scale
  3. Documentation Architecture Optimization — Solving Navigation Paradoxes via AI Prototyping

Case Study 1: Claude Code “Drift-Check” Agent

Project Lead AI / Agentic Workflows Claude Code Technical Writing Content Operations Prompt Engineering

Project Title: Designing Trust & Verification in Autonomous Workflows
Role: Project Lead
Collaborators: Other writers to run QA tests
Core Objective: Use Claude Code to detect when product UI changes cause documentation drift and automate the drafting of fixes.

The Problem

In complex product ecosystems, UI changes often outpace documentation, creating “content drift” that erodes user trust.

The Principal Objective

Design a verification-first AI system that detects documentation drift and proposes fixes while making its reasoning visible, auditable, and controllable by humans.

System Behavior & Logic

I designed a custom Claude Code skill (/drift-check) that functions as a specialized agent. Rather than treating the system as a black box, I designed it to behave as a transparent assistant:

  • Surfaces confidence levels for each proposed change
  • Maps UI strings to documentation nodes with traceable logic
  • Explains why a change is suggested, not just what changed

AI Behavior Design

  • Defined how the system behaves under low-confidence scenarios, including:
  • requesting clarification instead of guessing
  • deferring action when mappings are unclear
  • Designed confidence signaling to help users assess reliability of suggestions
  • Established PR-based interaction model as the primary interface, ensuring all AI actions are reviewable and reversible

Strategic Impact

  • Trustworthy AI: Established a standard where AI behavior is predictable and auditable via Pull Requests.

  • Systems Thinking: Solved for “discovery inconsistency” by partnering with Engineering to standardize UI string paths across repositories.

Nothing publishes automatically. The agent opens a Pull Request (PR) for a writer to review, edit, and approve.

Workflow: UI Change to Writer-Approved Update
flowchart LR A[UI Change Detected] --> B[drift-check Skill] B --> C[Claude Code Orchestrator] C --> D[Find Stale Doc Files] D --> E[Draft Content Fix] E --> F[Open Pull Request] F --> G[Writer Reviews and Approves] style A fill:#e8f0fb,stroke:#0066cc,color:#333 style G fill:#e8f8e8,stroke:#2e7d32,color:#333

The Roadmap: Crawl, Walk, Run

Phase 1
Pilot
In Progress

Focus on a single product. A 10-week cycle covering discovery, building the skill, and a live pilot during an active release cycle.

Phase 2
Scale
Planned

Expansion to all product repos. An orchestrator agent loops through a product config file, routes PRs to specific writers, and automates via GitHub Actions.

Lessons Learned & Risks

  • Standardization: The agent depends on standard UI string file paths — discovery revealed inconsistency across engineering teams.
  • Prompt Engineering: Careful iteration was required to minimize false positives and keep Claude’s edits targeted and minimal.
  • Scope: Strictly limited to UI label drift. Does not replace full content audits or human decision-making.

Key Outcomes

  • Removes manual, reactive doc checks from the writer’s workflow
  • Notifies writers of UI changes in real time via Pull Request
  • Establishes a repeatable, auditable AI-assisted content operations process

Case Study 2: The Release Notes “AI Bridge”

Multi-Agent Systems Gemini API Style Guide Enforcement Jira Integration Content Operations

Project Title: Multi-Agent Orchestration & Scalable Language Frameworks
Role: Lead AI Content Designer (System & Interaction Design)
Collaborators: Editors, Release Notes Lead
Core Objective: Convert raw technical updates into clear, user-benefit-focused release notes using a multi-agent Gemini system.

The Problem

Manually editing release note blurbs to meet style standards was a significant time sink. It led to inconsistency, writer fatigue, and risk of non-compliant content reaching customers. While standards existed, there was a persistent gap between the written rules and their execution.

The Principal Objective

Design a scalable AI language system that translates ambiguous technical inputs into consistent, user-centered release notes by encoding editorial standards into structured prompt and agent architectures.

Designing for Feedback Loops

Designed a feedback loop where the AI not only edits content but explains which rules were applied and why. This shifts the system from a passive generator to an active learning interface, improving both output quality and writer skill over time.

AI Behavior & Interaction Design

  • Defined how the system:
  • interprets incomplete or inconsistent Jira data
  • prioritizes input fields
  • requests clarification when data is insufficient
  • Designed strict output constraints to ensure:
  • clarity
  • consistency
  • adherence to style rules
  • Established human-in-the-loop approval as a core interaction model
  • Balanced rule enforcement vs flexibility in prompt architecture

Explainability & Trust Design

  • Encoded 30+ style rules into machine-readable constraints
  • Designed outputs to be:
  • predictable
  • auditable
  • consistent across use cases
  • Reduced ambiguity by forcing the system to operate within clearly defined boundaries
System Architecture: Main Agent and Sub-agents
flowchart TD W[Writer inputs Jira ticket or draft blurb] --> M[Main Agent - Release Notes Editor] M --> S1[Edit Blurb Sub-agent - 30+ style rules enforced] M --> S2[Fetch from Jira Sub-agent - Pulls ticket data and generates draft] M --> S3[Batch Processing Sub-agent - Iterates ticket lists and updates Jira] S1 --> O[Compliant Draft for Writer Review] S2 --> O S3 --> N[Writer Notification via Gmail] style W fill:#e8f0fb,stroke:#0066cc,color:#333 style O fill:#e8f8e8,stroke:#2e7d32,color:#333 style N fill:#e8f8e8,stroke:#2e7d32,color:#333

Impact & The “Crawl, Walk, Run” Path

Stage Status Description
Crawl Achieved AI enforces complex style rules. Writers submit drafts and receive standard-compliant versions back for review.
Walk In Progress Writers provide a Jira ticket number; Gemini pulls the data and returns a complete draft automatically.
Run Planned System automatically processes weekly ticket lists, updates Jira fields, and notifies writers for final approval.

Challenges & Roadblocks

  • Ambiguity: Exposed “gray areas” in the existing style guide, requiring the team to codify subjective rules into explicit, measurable criteria.
  • Input Quality: Blurb quality is directly tied to the quality of data entered in Jira tickets, requiring a parallel focus on mandatory field standards.
  • Prompt Complexity: Extensive iteration was required to build prompts that are strictly rule-compliant yet flexible enough for varying technical inputs.

Key Outcomes

  • Reduced manual editing effort across release cycles by standardizing AI-generated outputs
  • Codified 30+ stylistic rules into a reproducible, testable AI system
  • Created a scalable pipeline that grows from single-blurb edits to full weekly automation

Case Study 3: Documentation Architecture Optimization

UX Research Lead Heuristic Evaluation Information Architecture Journey Mapping UX Prototyping AI-Accelerated Design

Project: Designing Clarity in Complex Systems through Research, IA, and AI-Accelerated Prototyping
Role: Project Lead / UX Specialist
Core Objective: Resolve navigation ambiguity and improve discoverability between current and legacy documentation.

Relevance to AI Experience Design

While not an AI system, this project reflects the same core principles required for AI experience design:

  • reducing ambiguity
  • improving system clarity
  • designing for user confidence and control

These principles directly inform how I design AI interactions and explanation patterns.

The Challenge: The Navigation Paradox

In complex enterprise software, users often don’t know “where” they are in the product lifecycle. Data from Jira, customer support tickets, and direct feedback identified a significant usability gap: users consistently struggled to differentiate between current and legacy content. I led a research initiative to solve this navigation paradox.

Impact of the Problem

  • Users unknowingly followed incorrect steps for their engine, causing configuration errors and increased support volume
  • No visual indicator on search results to identify which engine a doc belongs to
  • No mechanism to switch between corresponding OIE and OCE pages — users had to start a new search entirely

The Process: Heuristic Evaluation & Journey Mapping

A structured research and evaluation process was conducted in collaboration with the UXR team.

Research Process: Discovery to Stakeholder Sign-off
flowchart LR A[Data Analysis - Jira and Support Tickets] --> B[Journey Mapping - Lucidchart screen-by-screen] B --> C[Heuristic Evaluation - Nielsen Norman 4 parameters] C --> D[Findings Consolidation - 3 critical areas] D --> E[Prototype Development - v0 AI high-fidelity] E --> F[Stakeholder Sign-off] style A fill:#e8f0fb,stroke:#0066cc,color:#333 style F fill:#e8f8e8,stroke:#2e7d32,color:#333

Step 1 — User Journey Mapping: Mapped the full documentation journey screen-by-screen in Lucidchart, allowing the team to visualize exactly where users felt “trapped” or “lost” during engine transitions.

Step 2 — Expert Heuristic Evaluation: Led a team of writers through an independent evaluation based on Nielsen Norman principles, measured against four specific parameters:

Heuristic What Was Evaluated
Discoverability Can users find the correct engine-specific label?
Navigation Can users move between OIE and OCE without feeling trapped?
Recognition over Recall Does the UI provide enough context cues about which engine they’re in?
Consistency Are visual treatments uniform across the documentation site?

Step 3 — Consolidation of Findings: The evaluation surfaced three critical areas for improvement:

  1. Engine Identification: Clear, persistent labeling of which engine a document applies to.
  2. Visual Differentiation: Stronger visual cues to distinguish the two doc sets.
  3. The Switcher: A functional toggle to move between corresponding pages.

Design & AI Prototyping: “Ship it and Measure”

  • Low-Fidelity Wireframes: Led development of three layout variations to explore solutions.
  • High-Fidelity Mockups: Collaborated with the UX Design team to finalize the visual aesthetic.
  • AI-Powered Prototyping: Used v0 to build a functional prototype in a fraction of the time, demonstrating two critical features to stakeholders:
    • Seamless toggle between OIE and OCE
    • Contextual Error Handling: A designed empty-state message for cases where a corresponding page does not exist in the alternate engine

Leadership & Influence

I utilized these high-fidelity AI prototypes to influence cross-functional stakeholders (Product and Engineering), moving the project from “backlog” to “shipped” in a single quarter.

Strategic Impact

Key Outcomes

  • Immediate Friction Reduction: Users can now verify their engine and switch pages instantly, without restarting a search. New designs to be released in Q3 2026.
  • Data-Backed Design: Every UI change was grounded in documented heuristic violations and stakeholder-reviewed research findings
  • AI-Accelerated Delivery: Used v0 to go from concept to functional prototype faster than traditional manual development
  • Repeatable Methodology: The “Ship it first, measure after” approach is now an established pattern for future documentation UX improvements

© arshabhirai.com. All rights reserved. Site created with Stackbit ❤ Kick-ass hosting provided by Netlify ❤

Back to top