Client-Side AI with Angular & Hugging Face

Machine learning is no longer confined to Python or server-side APIs. With Transformers.js, you can now run state-of-the-art NLP and vision models entirely in the browser using JavaScript. In this post, you'll learn what Transformers.js is, why it's useful, and how to integrate with angular.

🤖 What is Transformers.js?

Transformers.js is a JavaScript library developed by Xenova. Transformer.js is a groundbreaking JavaScript library that brings the power of Hugging Face's pre-trained AI models directly into your browser. Hugging Face has established itself as the premier open-source platform for machine learning, making advanced AI tools available to developers everywhere.

This means you can:

• Run a wide range of AI tasks locally: Perform text classification, translation, object detection, image classification, and much more, all without needing a backend server.
• Work completely offline: Once the model is loaded, your application can perform AI inferences even without an internet connection.
• Experience surprising speed: Leveraging WebAssembly and WebGPU, Transformers.js executes complex AI operations with remarkable efficiency, often faster than you'd expect from client-side execution.

💻 Why use Transformers.js?

1. No backend or server setup.
2. Privacy-preserving (runs locally).
3. Great for frontend ML apps.
4. Offline support.
5. Faster than expected thanks to ONNX (open neural network exchange) and WebAssembly.

🔧 How Transformers.js Works with Angular

 1. Client-Side Execution (Browser-Native AI):
 

• Transformers.js enables direct, in-browser execution of Hugging Face's pre-trained AI models. This means the AI runs directly on the user's device, not on a server somewhere else.
• This eliminates server round-trips for predictions, reducing latency and potentially infrastructure costs.
   

 2. No Direct Angular Integration (Framework Agnostic):
 

• Transformers.js is a pure JavaScript library. It's not an Angular-specific library (like @angular/forms or @angular/router).
• It doesn't rely on Angular's change detection, zones, or specific lifecycle hooks directly. Angular simply uses it like any other third-party JavaScript library.
   

 3. Core Mechanism: The pipeline Function:
 

• The central API of Transformers.js is the pipeline() function. This function abstracts away the complexity of loading models (model weights, tokenizers, preprocessors) and performing inference.
• You specify the AI "task" (e.g., 'object-detection', 'sentiment-analysis') and the Hugging Face model ID (e.g., 'Xenova/detr-resnet-50').
   

 4. Model Management & Caching (Offline Support):
 

• Transformers.js automatically handles the download and caching of model files (ONNX binaries, configuration files, etc.) from the Hugging Face Hub CDN.
• These models are stored in the browser's IndexedDB.
• After the initial download, the models are loaded from IndexedDB, enabling full offline functionality for subsequent inferences.
• The env object (@xenova/transformers/env) allows configuration of caching behavior, such as env.allowRemoteModels, env.useCache, and env.localModelPath (for IndexedDB subdirectories).
   

 5. Performance with WebAssembly (Wasm) & WebGPU:
 

• Transformers.js leverages WebAssembly (Wasm) for high-performance CPU execution of ONNX models. This allows near-native speed for computations.
• For supported hardware and browsers, it can also utilize WebGPU for even faster GPU-accelerated inference.
• Models are often available in quantized (e.g., q8) versions, which are smaller in size and faster to load and execute in the browser.
   

  6. Asynchronous Operations:
 

• Model loading (pipeline() call) and inference operations are asynchronous, returning Promises. This is crucial for non-blocking execution.
   

  7. Web Workers for UI Responsiveness (Crucial for Heavy Tasks):
 

• For computationally intensive tasks like object detection (where models can be large and inferences take time), it's highly recommended to run Transformers.js inside a Web Worker. The ng generate web-worker worker-name command to generate workers.
• This offloads the heavy processing to a separate thread, preventing the main UI thread from freezing and ensuring a smooth, responsive Angular application.
• Communication between your Angular components (main thread) and the Web Worker happens via message passing (postMessage() and onmessage).
   

  8. Angular Service for Encapsulation & Reusability:
 

• The Web Worker creation, communication logic, and Transformers.js API calls are best encapsulated within an Angular service.
• This makes the logic reusable across multiple components, improves testability, and keeps components lean and focused on UI concerns.
• Angular services can expose RxJS Observables (Subject, ReplaySubject) to push model loading states, errors, and inference results back to subscribed components.

⚙️ How to Use Transformer.js with Angular

This guide will show you how to harness the power of Transformers.js, a groundbreaking JavaScript library, within your Angular projects. With a simple installation of @huggingface/transformers via npm, you're just moments away from bringing the vast capabilities of Hugging Face's pre-trained AI models right into your browser. Today, we'll dive hands-on into building a real-world object detection application, demonstrating just how seamlessly Transformers.js empowers Angular developers to unlock on-device machine learning.

1. Install node package for transformers.js support

npm i @huggingface/transformers

2. Create UI to upload image for detect object
Create a simple file upload in your Angular component template

<div class="main-container">
 @if(error) {
   <div class="error">
	{{error}}
   </div>
 }
 <div class="detection-container">
 	@if (selectedFile) {
	  <div class="image-wrapper">
	  	<img [src]="selectedFile" (load)="onImageLoad($event)" />
	  	@if (isLoading) {
			<div class="loader"></div>
	  	}
	  	@for (obj of detections; track $index) {
		 	<div class="box" [ngStyle]="obj.box">
		  		<span class="label">{{ obj.label }} ({{ obj.score }}%)</span>
		 	</div>
	  	}
  	  </div>
    }
 </div>

 <div class="footer-container">
   @if (!isLoading) {
  	 <div>
   	   <label class="upload-label" for="file-upload">Upload Image</label>
   	   <input type="file" id="file-upload" accept="image/*" (change)="fileChange($event)" hidden>
  	 </div>
     @if (selectedFile) {
       <div>
	     <label class="upload-label" (click)="detect()">Detect</label>
	   </div>
     }
   }
 </div>
</div>

3. Create worker file to alive detection in background
Generate a worker file (e.g., detection-worker.worker.ts) in your app folder with use of command: ng generate web-worker detection-worker

detection-worker.worker.ts

/// <reference lib="webworker" />

import {env, ObjectDetectionPipeline, pipeline} from '@huggingface/transformers';

env.allowRemoteModels = true;

env.useFSCache = true;

env.localModelPath = 'models/';

let detector: ObjectDetectionPipeline | null = null;

addEventListener('message', async ({ data }) => {
  const { type, payload } = data;
  switch (type) {
   	case 'loadModel':
  		if (!detector) {
  			try {
   				detector = await pipeline('object-detection', 'Xenova/detr-resnet-50', { dtype: 'fp32' });
 				postMessage({ type: 'modelLoaded' });
			} catch (error: any) {
 				postMessage({ type: 'modelError', error: error.message });
 			}
    	} else {
      	   postMessage({ type: 'modelAlreadyLoaded' });
    	}
    break;

  	case 'detectObjects':
		if (!detector) {
 			postMessage({type: 'detectionError', error: 'Model not loaded in worker.'});
			console.error('Model not loaded in worker.');
			return;
		}

		try {
 			const outputs = await detector(payload, { threshold: 0.9 });
 			postMessage({ type: 'detectionResult', detections: outputs });
		} catch (error: any) {
 			postMessage({ type: 'detectionError', error: error.message });
 			console.error('Error during detection in worker:', error);
		}
  	 break;
  }
});

4. Generate worker service to communicate between component and worker
Generate a worker service file (e.g., detection-worker.service.ts) in your service folder with use of command: ng generate service detection-worker

detection-worker.service.ts

import { Injectable } from '@angular/core';
import { Subject } from 'rxjs';

@Injectable({
  providedIn: 'root'
})

export class DetectionWorkerService {
	private worker: Worker | undefined;
	private workerSubscriber$ = new Subject();

	constructor() {
 		this.workerInit();
 		if (this.worker) {
  			this.sendMessage('loadModel');
 		}
	}

	setWorkerSubscriber$(data: any) {
 		this.workerSubscriber$.next(data);
	}

	getWorkerSubscriber$() {
 		return this.workerSubscriber$.asObservable();
	}

	workerInit() {
 		this.worker = new Worker(new URL('../detection-worker.worker', import.meta.url));
 		this.worker.onmessage = (res) => {
  			this.setWorkerSubscriber$(res);
 		};
	}

	sendMessage(type: 'loadModel' | 'detectObjects', payload?: File) {
 		this.worker?.postMessage({type, payload});
	}
}

5. Use worker service to your component

import { Component, inject, OnInit } from '@angular/core';
import { SharedModule } from '../../shared/shared.module';
import { DetectionWorkerService } from '../../services/detection-worker.service';
import heic2any from 'heic2any';
import { takeUntilDestroyed } from '@angular/core/rxjs-interop';

interface DetectionResult {
  box: {
    xmax: number;
    xmin: number;
    ymax: number;
    ymin: number;
  };
  label: string;
  score: number;
}

interface ModifiedResult {
  label: string;
  score: number;
  box: {
    left: string;
    top: string;
    width: string;
    height: string;
  };
}

@Component({
 selector: 'app-home',
 imports: […],
 templateUrl: './home.component.html',
 styleUrl: './home.component.scss',
})

export class HomeComponent implements OnInit {
private detectionWorkerService = inject(DetectionWorkerService);
private destroyRef = inject(DestroyRef);

private fileObject!: File;
private imageWidth = 0;
private imageHeight = 0;

selectedFile = '';
detections: ModifiedResult[] = [];
isLoading: boolean = false;
error: string = '';

async ngOnInit() {
 this.detectionWorkerService.getWorkerSubscriber$().pipe(takeUntilDestroyed(this.destroyRef)).subscribe({
  next: (res: any) => {
    if (res.data.type == 'detectionResult') {
      if (!res.data.detections?.length) {
        this.error = 'There is nothing to detect';
      } else {
        this.error = "";
      }
      this.modifyDetectionObject(res.data.detections);
    }
 	if (res.data.type == "detectionError") {
   	  this.error = res.data.error;
 	}
  	this.isLoading = false;
  },
 });
}

async fileChange(event: any) {
	let file = event.target.files[0];
	if (String(file.name).toLowerCase().includes('.heic') || String(file.name).toLowerCase().includes('.heif')) {
 		const jpegFile: File = (await this.convertHeicToJpeg(file)) as File;
 		const dataTransfer = new DataTransfer();
 		dataTransfer.items.add(jpegFile);
 		file = dataTransfer.files[0];
	}
 	this.fileObject = file;
 	this.selectedFile = URL.createObjectURL(file);
}

async detect() {
 if (this.isLoading) {
  return;
 }
 this.detections = [];
 this.isLoading = true;
 await this.detectOffline(this.fileObject);
}

private modifyDetectionObject(data: DetectionResult[]) {
 const detections: ModifiedResult[] = [...data]
   .filter((obj: DetectionResult) => obj.score > 0.6)
   .sort((a: DetectionResult, b: DetectionResult) => b.score - a.score)
   .map((obj: DetectionResult) => {
     const { xmin, ymin, xmax, ymax } = obj.box;
     return {
       label: obj.label,
       score: Math.round(obj.score * 100),
       box: {
       	// To create box over the objects of image.
        left: (xmin / this.imageWidth) * 100 + "%",
        top: (ymin / this.imageHeight) * 100 + "%",
        width: ((xmax - xmin) / this.imageWidth) * 100 + "%",
        height: ((ymax - ymin) / this.imageHeight) * 100 + "%",
       },
      };
   });
   detections.forEach((ele) => {
      const find = this.detections.find((x) => x.label === ele.label);
      if (!find) {
        this.detections.push(ele);
      }
   });
}

private async detectOffline(file: File) {
  this.detectionWorkerService.sendMessage('detectObjects', file);
}

onImageLoad(event: Event) {
 this.detections = [];
 const image = event.target as HTMLImageElement;
 this.imageWidth = image.naturalWidth;
 this.imageHeight = image.naturalHeight;
}

/**
 * To convert HEIC format to JPEG format.
 * This is required because most of the browsers do not support HEIC format.
 * @param {File} heicFile 
 * @returns {File | null}
 */
private async convertHeicToJpeg(heicFile: File): Promise<File | null> {
	try {
 		const result = await heic2any({blob: heicFile, toType: 'image/jpeg', quality: 0.5});
 		const file = Array.isArray(result) ? result[0] : result;
 		let metadata = {type: `image/jpeg`};
 		return new File([file], `${+new Date()}.jpeg`, metadata);
 	} catch (error) {
  		console.error('Error converting HEIC to JPEG:', error);
	}
 	return null;
 }
}

6. Output

7. GitHub Repo
https://github.com/Rorix-Internal-Apps/transformerJs-angular

The era of server-dependent AI is evolving. With Transformers.js and Angular, you now have the tools to build smarter, faster, and more private web applications than ever before. Dive in, experiment with the vast Hugging Face ecosystem, and start transforming your Angular projects into intelligent powerhouses. What will you build next?