QuPath extension SAM

Latest release v0.9: Support for SAM3!

The extension now supports SAM3 models for enhanced segmentation and tracking capabilities.

https://github.com/user-attachments/assets/801d9465-e3d3-47c1-9999-e556b0b4a5d0

The command history is also now available from Automate > Show workflow command history and Automate > Create command history script in the menu bar.

Example scripts

var foregroundObjects = [
    PathObjects.createAnnotationObject(
        ROIs.createRectangleROI(
            433.000000, 194.000000, 21.000000, 37.000000,
            new ImagePlane(-1, 0, 0)
        ),
        null
    ),
    PathObjects.createAnnotationObject(
        ROIs.createRectangleROI(
            416.000000, 174.000000, 23.000000, 32.000000,
            new ImagePlane(-1, 0, 0)
        ),
        null
    ),
]
var backgroundObjects = []
var task = org.elephant.sam.tasks.SAMDetectionTask.builder(getCurrentViewer())
    .server(org.elephant.sam.Utils.createRenderedServer(getCurrentViewer()))
    .regionRequest(RegionRequest.createInstance(getCurrentServer().getPath(), 0.934228, 0, 0, 696, 520, 0, 0))
    .serverURL("http://localhost:8000/sam/")
    .verifySSL(false)
    .model(org.elephant.sam.entities.SAMType.VIT_T)
    .outputType(org.elephant.sam.entities.SAMOutput.MULTI_SMALLEST)
    .setName(true)
    .setRandomColor(true)
    .checkpointUrl("https://github.com/ChaoningZhang/MobileSAM/raw/master/weights/mobile_sam.pt")
    .addForegroundPrompts(foregroundObjects)
    .addBackgroundPrompts(backgroundObjects)
    .build()
task.setOnSucceeded(event -> {
    List<PathObject> detected = task.getValue()
    if (detected != null) {
        if (!detected.isEmpty()) {
            Platform.runLater(() -> {
                PathObjectHierarchy hierarchy = getCurrentHierarchy()
                hierarchy.addObjects(detected)
                hierarchy.getSelectionModel().clearSelection()
                hierarchy.fireHierarchyChangedEvent(this)
            });
        } else {
            print("No objects detected")
        }
    }
});
Platform.runLater(task)

var clearCurrentObjects = true
var task = org.elephant.sam.tasks.SAMAutoMaskTask.builder(getCurrentViewer())
    .server(org.elephant.sam.Utils.createRenderedServer(getCurrentViewer()))
    .regionRequest(RegionRequest.createInstance(getCurrentServer().getPath(), 0.934228, 0, 0, 696, 520, 0, 0))
    .serverURL("http://localhost:8000/sam/")
    .verifySSL(false)
    .model(org.elephant.sam.entities.SAMType.VIT_T)
    .outputType(org.elephant.sam.entities.SAMOutput.MULTI_SMALLEST)
    .setName(true)
    .clearCurrentObjects(clearCurrentObjects)
    .setRandomColor(true)
    .pointsPerSide(64)
    .pointsPerBatch(8)
    .predIoUThresh(0.880000)
    .stabilityScoreThresh(0.950000)
    .stabilityScoreOffset(1.000000)
    .boxNmsThresh(0.200000)
    .cropNLayers(0)
    .cropNmsThresh(0.700000)
    .cropOverlapRatio(0.340000)
    .cropNPointsDownscaleFactor(1)
    .minMaskRegionArea(0)
    .includeImageEdge(false)
    .checkpointUrl("https://github.com/ChaoningZhang/MobileSAM/raw/master/weights/mobile_sam.pt")
    .build()
task.setOnSucceeded(event -> {
    List<PathObject> detected = task.getValue()
    if (detected != null) {
        if (!detected.isEmpty()) {
            Platform.runLater(() -> {
                PathObjectHierarchy hierarchy = getCurrentHierarchy()
                if (clearCurrentObjects)
                    hierarchy.clearAll()
                hierarchy.addObjects(detected)
                hierarchy.getSelectionModel().clearSelection()
                hierarchy.fireHierarchyChangedEvent(this)
            });
        } else {
            print("No objects detected")
        }
    }
});
Platform.runLater(task)

def fromIndex = 20
def toIndex = 40
var objs = [
    11: [org.elephant.sam.parameters.SAM2VideoPromptObject.builder(0).bbox(352, 142, 463, 287).build(),]
]
var indexToPathClass = [
    0: PathClass.getInstance("SAM0"),
]
var regionRequests = (fromIndex..toIndex).collect {RegionRequest.createInstance(getCurrentServer().getPath(), 0.687248, 0, 0, 512, 443, 0, it)}}
var task = org.elephant.sam.tasks.SAMSequenceTask.builder(getCurrentViewer())
    .server(org.elephant.sam.Utils.createRenderedServer(getCurrentViewer()))
    .regionRequests(regionRequests)
    .serverURL("http://localhost:8000/sam/")
    .verifySSL(false)
    .model(org.elephant.sam.entities.SAMType.SAM2_S)
    .promptMode(org.elephant.sam.entities.SAMPromptMode.XYT)
    .objs(objs)
    .checkpointUrl("https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_small.pt")
    .indexOffset(fromIndex)
    .indexToPathClass(indexToPathClass)
    .planePosition(0)
    .build()
task.setOnSucceeded(event -> {
    List<PathObject> detected = task.getValue()
    if (detected != null) {
        if (!detected.isEmpty()) {
            Platform.runLater(() -> {
                PathObjectHierarchy hierarchy = getCurrentHierarchy()
                indexToPathClass.values().stream()
                        .filter(pathClass -> !getQuPath().getAvailablePathClasses().contains(pathClass))
                        .sorted(Comparator.comparing(PathClass::getName, new org.elephant.sam.comparators.NaturalOrderComparator()))
                        .forEachOrdered(pathClass -> getQuPath().getAvailablePathClasses().add(pathClass))
                hierarchy.addObjects(detected)
                hierarchy.getSelectionModel().clearSelection()
                hierarchy.fireHierarchyChangedEvent(this)
            });
        } else {
            print("No objects detected")
        }
    }
});
Platform.runLater(task)

New release v0.7: SAM2-based 2D+T tracking and 3D segmentation are supported now!

This is a QuPath extension for Segment Anything Model (SAM).

This is a part of the following paper. Please cite it when you use this project. You will also cite the original SAM paper and the MobileSAM paper.

• Sugawara, K. Training deep learning models for cell image segmentation with sparse annotations. bioRxiv 2023. doi:10.1101/2023.06.13.544786

Install

Drag and drop the extension file to QuPath and restart it.

Since QuPath v0.5.0, you can install the extension from the extension manager dialog by specifying Owner and Repository as shown below.

If you are using QuPath v0.4.x, you need to install the extension file for QuPath v0.4.x, which is now deprecated.

Please note that you need to set up the server following the instructions in the link below.

https://github.com/ksugar/samapi

Update

To update the qupath-extension-sam, follow the following instructions.

1. Open extensions directory from Extensions > Installed extensions > Open extensions directory
2. Replace qupath-extension-sam-x.y.z.jar with the latest version of the extension file. If you are using QuPath v0.4.x, you need to install the extension file for QuPath v0.4.x, which is now deprecated.
3. Restart QuPath application.

Please note that you need to also update the samapi server.
To keep updated with the latest samapi server, follow the instructions here.

Starting from QuPath v0.6, you can receive notifications about new releases of the extension by adding the following settings.

1. Open QuPath
2. Go to Extensions → Manage extensions
3. Click Manage extension catalogs
4. Enter the catalog URL: https://github.com/ksugar/qupath-catalog-ksugar
5. Browse and install the extensions you need

Usage

SAM prompt command

Rectangle (BBox) prompt

1. Select Extensions > SAM from the menu bar.
2. Select the Prompt tab in the Segment Anyghing Model dialog.
3. Select a rectangle tool by clicking the icon.
4. Add rectangles.
5. Select rectangles to process. (Alt + Ctrl + A: Select all annotation objects Ctrl or ⌘ + left click: Select multiple objects)
6. Press the Run for selected button.
7. If you activate Live mode, SAM predicts a mask every time you add a rectangle.

Point prompt

1. Select Extensions > SAM from the menu bar.
2. Select the Prompt tab in the Segment Anything Model dialog.
3. Select a point tool by clicking the icon.
4. Add foreground points.
5. (Optional) add background points.
6. Press the Run for selected button.
7. If you activate Live mode, SAM predicts a mask every time you add a foreground point.

Parameters

SamAutomaticMaskGenerator

1. Select Extensions > SAM from the menu bar.
2. Select the Auto mask tab in the Segment Anything Model dialog.
3. Set parameters.
4. Press the Run button.

Parameters

Register SAM weights from URL

1. Select Extensions > SAM from the menu bar.
2. Press the Register button in the Segment Anything Model dialog.

The weights file is downloaded from the URL and registered on the server. After the registration, you can select the weights from the SAM weights dropdown menu.

Parameters

SAM weights catalog

Here is a list of SAM weights that you can register from the URL.

Tips

If you select a class in Auto set in the Annotations tab, it is used for a new annotation generated by SAM.

Known issues

• SAM3 video predictor does not work with negative bbox prompts. See https://github.com/facebookresearch/sam3/issues/335.

Updates

v0.9.1

• Enable the confidence threshold setting only for SAM3 models.
• Accept SAM3 requests without any text or bounding box prompts. This enables the use of SAM3 with a pre-existing inference state. The samapi server version must be v0.7.1 or above.

v0.9.0

• Support SAM3 (you need to use the samapi server v0.7.0 or above).
• Accept server URL without a trailing slash.

v0.8.0

• Support flexible Groovy scripts to run SAM tasks. See the example scripts.

v0.7.0

• Support 2D+T tracking and 3D segmentation with SAM2 models, available with the samapi server v0.6.0 and above.

v0.6.0

• Support SAM2 models, available with the samapi server v0.5.0 and above.
• Use the current view for the encoder input in the rectangle mode.

v0.5.0

• QuPath v0.5 support by @Rylern and @ksugar in ksugar/samapi#12

v0.4.1

• Properly send the checkpoint URL parameter
  • The checkpoint URL was not sent to the server.
• Add a catalog of SAM weights to README
• Add example scripts under src/main/resources/scripts

v0.4.0

• Support for registering SAM weights from URL. ksugar/qupath-extension-sam#8 ksugar/samapi#11 suggested and tested by @constantinpape
• Combine the Prompt and Auto mask dialogs into a single Segment Anything Model dialog.

v0.3.0

• Support for both point and rectangle foreground prompts by @petebankhead

  • Ensure each new point is a distinct object while SAM is running (i.e. turn of 'Multipoint' mode)
  • Support line ROIs as a way of adding multiple points in a single object
  

• Support point background prompts by @petebankhead

  • Points with 'ignored*' classifications are passed to the model as background prompts (Sidenote: it seems a large number of background points harm the prediction... or I've done something wrong)
  

• Implement 'Live mode' and 'Run for selected' by @petebankhead

  • 'Live mode' toggle button to turn live detection on or off
  • Alternative 'Run for selected' button to use only the selected foreground and background objects
    • This makes it possible to annotate first, then run SAM across multiple objects - as required on the forum
  

• Support SamAutomaticMaskGenerator

  

• Menu items simplified to a single command to launch a dialog to control annotation with SAM by @petebankhead

  • Provide persistent preferences for key choices (e.g. server, model)
  • Run prediction in a background thread with (indeterminate) progress indicator
  • Help the user with tooltips (and prompts shown at the bottom of the dialog)

• Handle changing the current image while the command is running by @petebankhead

  • Send entire field of view for point prediction This is useful for one-click annotation of visible structures

• Include the 'quality' metric as a measurement for objects that are created by @petebankhead

• Support z-stacks/time series (by using the image plane; there's no support for 3D objects) by @petebankhead and @rharkes

• Optionally assign names & random colors to identify the generated objects by @petebankhead

• Optionally return multiple (3) detections instead of 1 by @petebankhead

• Select which detection to retain based upon size or quality, or keep all of them by @petebankhead

• Optionally keep the prompt objects, instead of immediately deleting them by @petebankhead

v0.2.0

• Support any number of channels

  

Citation

Please cite my paper on bioRxiv.

@article {Sugawara2023.06.13.544786,
	author = {Ko Sugawara},
	title = {Training deep learning models for cell image segmentation with sparse annotations},
	elocation-id = {2023.06.13.544786},
	year = {2023},
	doi = {10.1101/2023.06.13.544786},
	publisher = {Cold Spring Harbor Laboratory},
	abstract = {Deep learning is becoming more prominent in cell image analysis. However, collecting the annotated data required to train efficient deep-learning models remains a major obstacle. I demonstrate that functional performance can be achieved even with sparsely annotated data. Furthermore, I show that the selection of sparse cell annotations significantly impacts performance. I modified Cellpose and StarDist to enable training with sparsely annotated data and evaluated them in conjunction with ELEPHANT, a cell tracking algorithm that internally uses U-Net based cell segmentation. These results illustrate that sparse annotation is a generally effective strategy in deep learning-based cell image segmentation. Finally, I demonstrate that with the help of the Segment Anything Model (SAM), it is feasible to build an effective deep learning model of cell image segmentation from scratch just in a few minutes.Competing Interest StatementKS is employed part-time by LPIXEL Inc.},
	URL = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786},
	eprint = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786.full.pdf},
	journal = {bioRxiv}
}

Acknowledgements

• QuPath
• segment-anything
• MobileSAM