Instructions for Extracting Images from AndroidControl TFRecord

Follow these steps to download and extract images from the AndroidControl TFRecord dataset.

Step 1: Download the Original AndroidControl TFRecord

1. Navigate to the AndroidControl TFRecord repository where the AndroidControl TFRecord is hosted.
2. Download the TFRecord file to your local machine or server.

Step 2: Run the Python Script to Extract Images

1. Ensure that extract_images.py is in the same directory as your downloaded TFRecord or update the script to point to the location of your TFRecord.
2. Open a terminal or command prompt.
3. Execute the script by running:

   python extract_images.py
   

Step 3: Enjoy the parsed json data

Here’s the information about the VH annotation:

Each screenshot corresponds to a .pkl file with the same name, containing a list of dictionaries. Each dictionary includes the following keys:

• "bounds_in_screen": Defines the screen boundaries of the window, with values:

  {
      "left": windows.bounds_in_screen.left,
      "right": windows.bounds_in_screen.right,
      "bottom": windows.bounds_in_screen.bottom,
      "top": windows.bounds_in_screen.top
  }
  

• "window_id": The unique identifier of the window.
• "window_layer": The layer or z-index of the window.
• "window_type": The type of the window.
• "tree": A list containing dictionaries representing each element node in the window.

Each node in the "tree" list includes:

• "bounds_in_screen": The screen boundaries of the element node, with values:

  {
      "left": node.bounds_in_screen.left,
      "right": node.bounds_in_screen.right,
      "bottom": node.bounds_in_screen.bottom,
      "top": node.bounds_in_screen.top
  }
  

• "class_name": The class name of the element.
• "content_description": The content description of the element.
• "package_name": The package name the element belongs to.
• "text_selection_start": The start position of any text selection.
• "text_selection_end": The end position of any text selection.
• "view_id_resource_name": The resource name of the view ID.
• "window_id": The window ID associated with the element.
• "is_enabled": Boolean indicating if the element is enabled.
• "is_visible_to_user": Boolean indicating if the element is visible to the user.
• "actions": A list of action IDs available for the element, derived as [action.id for action in node.actions].
• "child_ids": A list of IDs for child elements, derived as [child_id for child_id in node.child_ids].
• "depth": The depth of the element within the hierarchy.
• "drawing_order": The order in which the element is drawn within the window.