Brain Builder » Segmentation » Segmentation Documentation » Polygon Annotation on Video

In This Section:

On This Page:

AI Video Annotator

Neurala's Brain Builder features a number of tools to quickly and efficiently tag videos without having to tag each frame from scratch. After reviewing this documentation, be sure to read the Best Practices: Video Annotation page for more tips on how to successfully use Brain Builder to tag videos.

The AI Video Annotator is a time-saving tool that learns tags from a single video frame and intelligently applies them to subsequent frames. For example, if you have ten seconds of video at 30 fps you will end up with 300 images annotated in just a few minutes.

Information The AI Video Annotator is an AI-assisted tagging tool that may not be available to all users. If you would like access to this tool, please speak to your Neurala account manager.

Video Tagging

  1. Select the video frame that you want to begin tagging, and use the Pen tool to outline the edges of any objects you want to tag.

    This should be done in the same manner as if you were tagging a single image, making the most of the available tagging tools in the Workspace. Tighter outlines will result in more precise bounding boxes.

    After a polygon has been created, the Class Selector appears.

  2. Assign a class to the polygon by choosing an existing class or creating a new one.


    When you have more classes than can be displayed at once in the Class Selector, typing a partial name will filter the selector to only show the matching classes.

    The Workspace also features a Class Manager where you can create, review, and edit your classes. See the Class Manager section below.

  3. After the first frame has been tagged to your satisfaction, click the Annotator button in the bottom-right of the Workspace. The annotator pop-up appears.

    Use the Video Clip slider to define the start and end points within the video where you want to apply the annotator. The start point should be the frame you just tagged and the endpoint can be as much or as little as you wish. However, we recommend testing performance on a few seconds of video before doing longer clips.

    Adjust the Quality slider depending on whether you want the video annotator to work more quickly or more accurately. Accurate annotations take longer, while results from faster annotations may not be as precise.

    Set a minimum size for the polygons the annotator generates. In some situations, the video annotator can generate numerous small false positive results, so setting this value can help reduce those.

  4. Click Start to run the annotator. A progress bar appears.


    There is no need to stay on the page while the annotator runs. You can continue working on other data and return shortly after the annotator finishes.

    If you would like to preview the results while the annotator runs, click Preview Results and the Workspace will show you the frames as they are annotated. When the annotator is complete, you can view its predictions by playing the video or clicking to view specific frames.

Annotating Multiple Clips within a Video

There are a number of reasons you may want to create multiple annotation clips within the same video:

  • There are "cuts" in the video or significant changes in perspective
  • The objects you are annotating change their appearance (example: a person turning from front to back)
  • There are changes in the objects you wish to annotate in different parts of the video

The annotator works best when the objects you want to tag and the perspective are reasonably consistent throughout the clip you are annotating.

To annotate multiple clips within a single video, start by tagging the first frame of the clip you want to annotate, and then use the Video Clip slider in the Annotator window to define the start and end point of the clip you wish to annotate.

Click the Timeline button to see a timeline with all of the annotation clips you have created.

If you have overlapping clips, you can click the Eye buttons on the left side to show/hide each clip individually. The Garbage Can button can be used to delete video annotation clips that you don't want to keep. The more recent clip (higher in the timeline view) will take priority over earlier ones for the frames where they overlap.

Refining Video Annotations

The polygons created by the video annotator may not be perfect, but Brain Builder's tagging tools can be used to correct any details in each frame that need to be fixed.

The Addition and Subtraction tools are typically the most useful for refining the video annotator's polygons to make sure they are tightly bound around the objects of interest.

Information If you find yourself having to correct many frames in a row, it might be more efficient to re-run the video annotator on that portion of the video as a separate clip.

Class Manager

Within the workspace, you can create, delete, and manage the Classes of objects you are annotating.

Open the Class Manager by clicking the Classes button at the top of the left-side toolbar. This opens a window where the current classes are listed with a count of the number of polygons that have been labeled with each class in the current Dataset. Within this window, you can:

  • Rename an existing class—Click the Class name and edit it as you wish.
  • Change the Class color—Click the color field next to the Class name and select a new color from the color selection window.
  • Delete a class—This will delete all associate polygons!
  • Add a new class—Type the name into the field at the top of the window.

Next: Exporting Segmentation Data >>