Labs built on the cluster

Everything built and proven on the opp-ai cluster (RHOAI 3.4) — what each lab does, the tools it uses, and the concrete outcome. living doc

These labs were built to cover Red Hat OpenShift AI end to end — from the core ML lifecycle to MLOps, scale, responsible-AI, and infrastructure. Each has a folder in the dslab repo (manifests + a README) and, where it records a decision, an ADR. Two GPUs back the cluster: an NVIDIA A2 (16 GB) and an RTX 4000 Ada (20 GB).

At a glance

Lab	Focus	Key tools	Outcome	Status
RAG + Capstone	ML lifecycle	LoRA · MLflow · KServe/vLLM · PGVector	fine-tune → serve → RAG proven	✅ stopped
Agentic — OppyOps	agents	Llama Stack · MCP · vLLM	agent calls live tools; guardrail blocks	✅ stopped
MLflow server	experiment tracking	MLflow · sqlite/PVC	dashboard tab works; runs logged	✅ running
Feast feature store	features	Feast · sqlite online store	register → materialize → online lookup	✅ running
Model Registry	governance	ModelRegistry · MariaDB (MLMD)	models + versions + artifacts registered	✅ running
Data engineering	data pipelines	ODF S3 · KFP · pandas/pyarrow	bronze→silver→gold data lake + quality gates	✅ done
Data Science Pipeline	MLOps	KFP v2 · Argo · MariaDB+Minio	prep→train→eval→register auto-run	✅ running
Distributed training	scale	Trainer v2 · Kueue · PyTorch DDP	2-node DDP across A2+RTX	✅ done
Model evaluation	evaluation	LMEval · lm-eval-harness	Qwen2.5-0.5B arc_easy acc 0.65	✅ done
TrustyAI	responsible AI	TrustyAIService · SPD/DIR	bias workflow + API (metric quirk)	⚠️ partial
Guardrails Orchestrator	safety	FMS Guardrails · regex/PII detector	flags email/SSN/IP PII	✅ running
GPU time-slicing	infra	NVIDIA GPU Operator	A2 → 4 shareable GPUs	✅ active

The ML & agent lifecycle

RAG + Capstone

Proves the whole ML lifecycle in one thread: fine-tune → track → serve → RAG.

What it does: carries a single model through the full lifecycle — LoRA fine-tune a small model ("Oppy"), track the run, serve it on a GPU, and answer questions with retrieval augmentation over a vector store.

Tools: PEFT/LoRA · MLflow (file-store) · KServe + vLLM · PGVector · LangChain · JupyterLab workbench.

Outcome: the same fine-tuned model hallucinates alone but answers correctly through RAG — training, GPU serving, retrieval, and generation all working together.

Namespace rag-lab · capstone/ · ADR-0007 · stopped (GPUs freed)

Agentic — “OppyOps”

Proves agentic + MCP: an LLM that reasons in a loop and calls live cluster tools.

What it does: an agent on Llama Stack (Responses API) drives a tool-calling LLM that invokes an MCP server exposing live, read-only cluster tools, fronted by an input guardrail.

Tools: Llama Stack (odh-llama-stack-core) · MCP (FastMCP/SSE) · Qwen2.5-7B on vLLM (tool-calling) · LLM-judge guardrail.

Outcome: the agent autonomously answers with live data (“1 free GPU of 2 … 3 worker nodes … — Oppy”); the guardrail blocks “delete all pods.”

Namespace agent-lab · ADR-0008 · stopped (GPU freed)

Platform services (dashboard tabs)

MLflow tracking server

Turns the dashboard’s “MLflow” tab from unavailable into a real server.

What it does: deploys a self-contained MLflow tracking + model-registry server, so experiments/params/metrics are recorded and visible in the dashboard.

Tools: MLflow 3.10 · sqlite + PVC (no external DB/S3) · Data Science Gateway · Kubernetes-auth (per-project “workspaces”).

Outcome: the MLflow tab resolves; a test run logs to a DS-project workspace and persists on the PVC.

Namespace redhat-ods-applications · no GPU · running

Feast feature store

Proves the “Feature Store” tab — define once, serve online for real-time inference.

What it does: registers features, materializes them from an offline source into a low-latency online store, and serves them by entity key.

Tools: Feast · file registry + sqlite online store on PVCs · the shipped driver_stats demo.

Outcome: full loop verified — register → materialize → online lookup returns live features per driver (e.g. driver 1001 conv_rate 0.0069).

Namespace feast-demo · no GPU · running

Model Registry

Model governance — named models, versions, storage URIs, promotion.

What it does: stands up a Model Registry (the hand-off from “trained” to “released”) and registers models with versions, metrics, and artifact locations.

Tools: ModelRegistry CR · MariaDB (MLMD metadata) · REST API · OpenShift Route.

Outcome: registered the fine-tuned “Oppy” model and (from the pipeline lab) an iris-classifier, each with a version + s3:// artifact.

Namespace rhoai-model-registries · ADR-0009 · no GPU · running

Data engineering

Medallion data lake (bronze → silver → gold)

The plumbing upstream of ML: turn messy multi-source data into trustworthy, curated, queryable layers — with quality gates.

What it does: ingests two dirty raw feeds (orders + customers) into a real S3 data lake, profiles them, cleans & conforms into a partitioned silver layer, enforces data-quality gates (retention + referential integrity), then builds gold outputs — BI analytics and a Feast-ready ML feature table — plus a lineage manifest.

Tools: ODF Ceph RGW object storage (ObjectBucketClaim) · Data Science Pipelines (KFP v2) · pandas / pyarrow · partitioned parquet · kfp-kubernetes for S3 creds.

Outcome: SUCCEEDED — bronze 5,150 orders → silver 3,150 (30 date partitions, 100% referential integrity, gate PASSED) → gold ($1.38M revenue across 5 countries; 784×12 customer features). Everything persisted as parquet in S3 with a lineage manifest.

Namespace data-eng-lab · ADR-0015 · no GPU · feeds the feature store · done

MLOps & scale

Data Science Pipeline (KFP)

Automates the lifecycle as a repeatable pipeline — and wires it to governance.

What it does: a KFP v2 pipeline runs prep → train → evaluate → register; if the model passes an accuracy gate, the final step registers it into the Model Registry.

Tools: DataSciencePipelinesApplication (KFP v2 on Argo Workflows) · operator-managed MariaDB + Minio · kfp SDK · scikit-learn.

Outcome: a run SUCCEEDED (accuracy 1.0), gated, and auto-registered iris-classifier into the Model Registry — orchestration driving governance.

Namespace pipeline-lab · ADR-0010 · low GPU · running

Distributed training (Kueue + Trainer v2)

Exercises the batch-queue + training stack across both GPUs.

What it does: submits a 2-node PyTorch DDP job through the Kueue queue; Kubeflow Trainer v2 builds the worker pods and torchrun coordinates them over NCCL.

Tools: Kubeflow Trainer v2 (TrainJob) · Kueue (quota-managed) · PyTorch DDP / NCCL · torchrun.

Outcome: rank 0 landed on the RTX, rank 1 on the A2; the all_reduce (0+1=1) confirmed cross-GPU collectives; loss fell 0.75 → 0.41; Complete=True.

Namespace fine-tuning · ADR-0013 · 2 GPUs (released on finish) · done

Evaluation, responsible AI & safety

Model evaluation (LMEval)

Quantitative benchmarking — the objective counterpart to eyeballing outputs.

What it does: runs a standard academic benchmark against a model on GPU via TrustyAI’s LMEvalJob (a wrapper around lm-evaluation-harness).

Tools: LMEval (LMEvalJob) · EleutherAI lm-evaluation-harness · vLLM/torch on GPU.

Outcome: evaluated Qwen2.5-0.5B-Instruct on arc_easy — acc 0.65 / acc_norm 0.55 on the RTX in ~36 s.

Namespace eval-lab · ADR-0012 · 1 GPU (released on finish) · done

TrustyAI — bias & fairness

Measures group fairness on a model’s decisions (SPD, DIR).

What it does: uploads inference data with a protected attribute (gender) and an outcome (approved), then requests fairness metrics.

Tools: TrustyAIService (PVC-backed) · SPD (statistical parity difference) / DIR (disparate impact ratio) API.

Outcome: service + data upload + schema/name-mapping + the SPD/DIR API all work on a deliberately biased dataset (F 43% vs M 73% approval). Open item: the on-demand metric doesn’t consume uploaded data in this build — the intended path is live KServe payload logging.

Namespace trustyai-lab · low GPU · partial

Guardrails Orchestrator

Native content safety — the production answer to the agent lab’s LLM-judge.

What it does: runs the FMS Guardrails Orchestrator with a built-in regex/PII detector and screens text for sensitive content (in front of an LLM, or standalone detection).

Tools: FMS Guardrails Orchestrator · built-in regex/PII detector (email, SSN, IPv4, …).

Outcome: the detection API flags email_address, social_security_number, and ipv4 as PII (score 1.0) and returns [] for clean text — no false positives.

Namespace guardrails-lab · ADR-0014 · no GPU · running

Infrastructure

GPU time-slicing

Fit more workloads onto limited GPUs by sharing one card across pods.

What it does: configures the NVIDIA GPU Operator so the A2 advertises 4 schedulable nvidia.com/gpu, letting several pods share it (compute time-shared; VRAM shared, not isolated). The RTX is left 1:1 for whole-GPU jobs.

Tools: NVIDIA GPU Operator · ClusterPolicy · time-slicing ConfigMap · node label.

Outcome: the A2 node now shows nvidia.com/gpu: 4; two pods ran concurrently on the same physical A2 (identical GPU UUID) — impossible under 1:1 allocation.

Cluster-wide (A2 / wk-1) · ADR-0011 · active (persistent)

Add a lab by copying a <div class="persona">…</div> block with an <h3 id="…"> — the sidebar updates automatically. Full manifests + READMEs live in the dslab repo (ADRs 0007–0014).