Identify Web-UI interaction from Screen Recordings

Abstract

Digital tools become more complexed and varied in application through the implementation of A.I and transformers in different fields and sectors. What about application for customer services?

• Tradditionally:
  • Static slides
  • Linear video walkthrough
  • Not able to teach agents how to handle in actual real-life situations where customers are typically more cranky and less patience that what has been introduced my the materials
• Need for the simulation that can capture that nuances customer agents can only get from constantly being exposed to various situations

I. Project Overview

The final goal for this project enables the creation of realistic, simulation-based training experiences that closely mirror real-life interactions between customer and support agent. These simulations can be enriched with visual annotations and voiceovers to represent an full interactive simulations between agent and a "AI-generated" customer.

Due to the time, this is the current scope of this project to created a system that analyzes screen recordings to detect significant UI changes and the UI components.

By identifying key frames where meaningful actions occur—such as button clicks, form entries, or dropdown selections—and mapping them to the corresponding interface elements, the system can reconstruct a step-by-step simulation for the end users .

II. System Overview:

The system processes screen recordings of customer service agents interacting with UI workflows and performs the following steps:

• Key frame extraction: Filters out redundant video frames to isolate moments of significant screen change, such as submitting a form or selecting a menu item.

• UI element mapping: Identifies which interface elements (e.g., buttons, dropdowns, text fields) the user is interacting with, enabling the system to predict what triggered the screen change.

1. Initial Frame Extractor:

To reduce computational load and speed up downstream processing, we first filter out redundant or low-value frames. This step ensures we only keep frames that reflect meaningful changes

Part 1 SSIM (Structural Similarity Index) Analysis

Purpose: Helps detect fine-grained UI updates like typing or clicking that don’t involve full-screen changes.

• Divides frames into grids to capture local differences.
• Compares structural similarity between consecutive frames.
• Uses adaptive thresholds to detect subtle variations and reduce false positives.

Part 2: Color Histogram Analysis

Purpose: Identifies major screen transitions and removes unimportant frames, like when the screen is just one color during loading or when the recording has stopped.

• Analyzes changes in color distribution between frames.
• Captures scene-level shifts such as popups or navigation changes.
• Complements SSIM by highlighting broader visual changes.

Part 3: Optical Flow Detection

Reduces frame redundancy from transitions like vertical and horizontal mouse scrolling or hover effects.

• Measures pixel movement to detect directional motion.
• Identifies scrolling events to avoid over-sampling similar frames.
• Helps filter out frames where no significant user-triggered action occurs.

2. Main Frame Extractor

Traditional methods like SSIM or color histograms often miss subtle, context-sensitive changes. Vision Transformer (ViT) or Seeclick captures semantic shifts in the interface, detecting meaningful actions—like toggling a switch or submitting a form—even when visual changes are minimal.

A. Google BERT VIT

After the initial filtering, we apply Google’s Vision Transformer (google/vit-base-patch16-224) to further refine frame selection by capturing subtle, semantically meaningful UI changes that traditional methods may overlook.

Key Steps

• Feature extraction: extracts both global (whole-frame) and local (patch-level) features using a pretrained ViT model to capture semantic content beyond pixel differences.
• Frame-to-frame comparison: calculates Mean Squared Error (MSE) between feature vectors of consecutive frames to quantify semantic change.
• Peak detection: identifies key transition points using statistical thresholds and peak detection on change scores.
• Final frame selection: outputs frames that mark changes in transitions or UI state.

B. Seeclick

SeeClick is a visual GUI agent trained to detect UI changes directly from screenshots. Unlike ViT, it's specifically trained on diverse GUI images, allowing it to capture subtle yet meaningful interface changes more precisely. This improves the accuracy of key frame selection for realistic training simulations.

Key Steps

• Feature extraction Uses a Vision Transformer to extract perceptual-level embeddings from GUI screenshots.
• Embedding comparison: Computes pairwise distances between frame embeddings using MSE loss to measure how much the screen differs from the previous or average state.
• Adaptive frame clustering: Selects key frames based on significant shifts from a running mean using an adaptive threshold.
• Final frame selection: Outputs frames that mark transitions or UI state changes.

SeeClick captures subtle UI state changes that traditional or Google ViT methods might miss, improving the realism and precision of simulation training.

3. Identify UI Components with Bounding Boxes:

• Why? This will serve as a component to solve the problem: what area, or where would users click given this frame/images.

This method is applied from the paper "An Accurate GUI element detection approach based on old-fashioned computer vision algorithm"

Key Steps

• Input handling: accepts selected frames from ViT or SeeClick.
• Detection methods: merges results from the following techniques:
  • Text detection with OCR: identifies bounding boxes for text components using OCR (e.g., Tesseract).
  • Graphic component detection: identifies bounding boxes for non-text UI components using traditional computer vision methods such as binarization, grayscale conversion, and median blur.

Why this method and not just basic computer vision algorithm ?

• I am comparing against methodology of finding the contour and lines of a given image

III. Demo

IV. Assessment & Evaluation

Results:

• SeeClick takes approximately 10 minutes to process a 20-second video (including GUI detection), while ViT completes the same task in about 3 minutes.
  • In this project, I have to resize images for Seeclick because the processing time when loading the full image size would take triple the amount
• Across four videos of varying lengths, both models produced similar outputs in terms of the number of extracted frames and their corresponding timestamps.

Limitations

1. ViT/ Seeclick:

• Both models capture flashing UI elements (e.g., blinking recording buttons), which may result in redundant frames where no interaction occurs.
• Frames with sensitive customer input (e.g., typing into forms) are also captured, raising privacy concerns for simulation use.

• Frame that has loading icons are sometimes retained

• There is currently no metric to evaluate the relevance or accuracy of selected frames—this remains an open area for improvement.
• In order to mapped the steps and flow of action sequence, it is a lot more difficult if the clients doesn't provide us with videos that have mouses.

2. GUI element Detection:

• Lack of semantic understanding for each region: The system detects UI elements but doesn’t recognize their function (e.g., "Submit" vs. "Cancel"). this called for a different solutions to train for training UI semantic classifiers.
• Bounding box reliability: Detected regions aren’t validated for interactivity—non-clickable or decorative elements may be falsely identified.
• Mouse occlusion: Elements hidden by the cursor (e.g., during hover) may be missed or only partially detected in some frames.

This image illustrates that when the mouse hovers on top of a button, the algorithm won’t recognize that as a clickable area.

V. Critical Analysis:

This project presents a foundational pipeline for extracting meaningful frames from screen recordings, detecting UI elements, and create new methods on how to prepare training data for further finetuning. However, several areas remain for improvement and expansion.

Limitations and Future Work

• Implementing cursor tracking: to locate and track the position of the mouse cursor in key frames to infer user's interaction intent.
• Semantic interpretation of UI components: Future work can include buttons semantic classification using labeled datasets of UI interactions.
• Benchmarking and comparing with models like ScreenAI and OS-Atlas will help assess performance and generalizability.

---

VI. Model & Data Card

Model Card: Google ViT Base Patch16 224

• Model Name: ViT Base Patch16 224
• Version: Base
• Architecture: Vision Transformer (ViT)
• Source: Hugging Face Model Hub

Intended Use

• Primary Use Case: Image classification tasks across diverse datasets.
• Domains: General computer vision applications.
• Usage: Suitable for tasks requiring image understanding, such as object recognition and scene classification.

License

• Type: Apache License 2.0
• Details: Permits use, reproduction, and distribution with minimal restrictions. Full license available here.

Ethical Considerations

• Bias: The model's performance is influenced by the diversity and balance of its training data. Users should evaluate and mitigate potential biases when applying the model to specific datasets.
• Fairness: Ensure that the model's deployment does not lead to unfair outcomes, especially in sensitive applications.
• Transparency: Users should document and communicate the model's limitations and potential biases in their applications.

---

Model Card: SeeClick

• Model Name: SeeClick
• Version: Latest
• Architecture: Specialized visual GUI agent
• Source: Hugging Face Model Hub

Intended Use

• Primary Use Case: Automating complex tasks on digital devices by interpreting and interacting with graphical user interfaces (GUIs) using visual inputs.
• Domains: GUI automation, user interface testing, and related applications.
• Usage: Designed to operate solely on screenshots for task automation, eliminating the need for underlying code or structural data.

License

• Type: MIT License
• Details: Allows for flexible reuse, modification, and distribution. Users must include the original license and copyright notice in any substantial portions of the software. Full license details are available in the model repository.

Ethical Considerations

• Bias: The model's effectiveness may vary across different GUI designs and layouts. Users should assess performance across diverse interfaces to ensure reliability.
• Fairness: Care should be taken to avoid unintended consequences when automating tasks, ensuring that the model's actions align with user intentions and ethical guidelines.

---

VII. Documentation & Resource Links

The following models and research efforts offer valuable insights and tools that align closely with the goals of this project—particularly around UI understanding, GUI grounding, and training interface-aware agents:

1. ScreenAI

• Paper: ScreenAI: A Vision-Language Model for UI and Infographics Understanding
• Authors: Gilles Baechler, Srinivas Sunkara, Maria Wang, et al.
• Why it's relevant:
  ScreenAI is trained specifically to understand screen-based interfaces and infographics using a vision-language approach. Its focus on UI element localization and summarization is closely aligned with our goal of detecting interaction regions and creating screen simulations.

2. OS-Atlas

• Paper: OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
• Authors: Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, et al.
• Why it's relevant:
  OS-Atlas is a cross-platform foundation model trained on GUI screenshots and grounding tasks. Its emphasis on out-of-distribution generalization and open-source training data makes it a compelling reference for building robust, UI-aware models like ours that can scale across different software environments.

VIII. References

1. Google ViT Base Patch16-224
   Hugging Face Model Card: https://huggingface.co/google/vit-base-patch16-224

2. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
   Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu
   arXiv preprint, 2024.
   https://arxiv.org/abs/2401.10935

3. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
   Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li
   arXiv preprint, 2020.
   https://arxiv.org/abs/2008.05132