Radar-video fusion autotracking (edge-to-edge)

The fusion of radar and video data makes it possible to discover, classify, and track moving objects. With edge-to-edge-based radar-video fusion autotracking, you combine the capabilities of one radar and one PTZ camera on edge. The radar discovers and classifies a moving object and directs the camera to it. The camera immediately zooms in on and starts tracking the object. For added accuracy, the camera validates the radar’s object classification and triggers the alarm.

Example of how to use radar-video fusion autotracking

In this example, we want to track objects that move inside a fenced area. A PTZ camera has been mounted with the radar to validate alarms and provide accurate classification thanks to radar-video-fusion technology.

1. Intruders are walking outside the fence, not triggering an alarm.

2. Intruders break in through the fence, the radar discovers them and triggers an alarm.

3. The radar directs the PTZ camera towards the intruders, and lets the camera validate the alarm with video analytics.

To set up radar-video fusion autotracking:

1. Mount the devices. Mount the PTZ camera and the radar together according to instructions in the radar’s installation guide.

2. Pair the devices. Pair the camera with the radar from the camera’s web interface. For instructions, see Pair the camera with a radar.

3. Set the radar’s mounting height. Set the mounting height in the camera’s web interface. For instructions, see Set the mounting height

4. Align the devices. For instructions, see Align the camera and the radar.

5. Align the scene with the radar view. To understand where objects move and where there are static objects in the scene, you can use object trails as well as add a map as a background to the radar view. For instructions, see Relate the radar stream to the real world.

6. Set up radar scenarios. Create one or more radar scenarios to detect moving objects. For instructions, see Create radar scenarios to detect objects.

7. Validate the installation and configuration. We recommend that you validate the installation and the radar scenarios. For instructions, see .

8. Set up autotracking. Set up autotracking by creating one or more tracking profiles. For instructions, see Create autotracking profiles to track objects.

9. Create a rule. Create a rule to record the streams when autotracking is activated. For instructions, see Create rule to record when autotracking is activated.

Radar pairing is a one-way setup where you pair a camera with a radar and use the camera to configure and maintain both devices. The camera has an allocated channel for the radar stream, and when you have paired the devices, the radar stream is automatically assigned to this channel.

Before you start:

• Make sure the camera and radar are directed toward the same area of interest.

• Make sure the camera and radar are synced to the same time source. To check the time sync status, go to Installation > Time sync status in each device.

1. Pair the camera with the radar:

2. In the camera’s web interface, go to System > Edge-to-edge > Pairing.

3. Click Add.

4. In the list of pairing types, select Radar.

5. Enter the host name, username, and password for the radar.

6. Click Connect to pair the devices.

   When the connection is established, the radar settings become available in the camera’s web interface.

Set the radar’s mounting height in the camera’s web interface. The correct mounting height is very important for autotracking to work.

Measure the height from the ground up to the radar as accurately as possible. If the ground is uneven, measure from the average ground elevation instead of from a single point.

1. Go to Radar > Settings > General.

2. Set the height under Mounting height.

For the camera to be able to track objects detected by the radar, you must align the camera’s pan zero with the radar’s pan zero.

1. Go to Status > Camera and radar alignment and click Align devices.

2. Follow the step-by-step instructions. In step 2, select the camera image that best corresponds with the center of the radar’s field of view.

If you later want to realign the pan-offset, you can do it from here by clicking Realign devices.

When you look at the radar stream, it’s hard to understand what it corresponds to in reality or in relation to the camera view. To help you understand where buildings, trees, or bushes are located and where people or cars move, you can:

• Show a map as a background to the radar stream. For instructions, see Use a map to understand the radar view.

• Map out the scene using object trails. For instructions, see Use trails to understand the radar view.

To make it easier to understand where in the scene there are static objects like buildings and where objects move, you can show a map as a background to the radar stream. You can use a ground plan or an aerial photo that shows the area covered by the radar. Adjust and calibrate the map so the radar view fits the position, direction, and scale of the map, and zoom in on the map if you're interested in a specific part of the scene.

1. Go to Radar > Map calibration > Map.

2. Select the image you want to upload, or drag and drop it in the designated area.

   To reuse a map image with its current pan and zoom settings, click Download map.

3. Under Rotate map, use the slider to rotate the map into position.

4. Go to Scale and distance on a map and click at two predetermined points on the map.

5. Under Distance, add the actual distance between the two points you have added to the map.

6. Go to Pan and zoom map and use the buttons to pan the map image, or zoom in and out on the map image.

7. Note

   • The zoom function doesn’t alter the radar’s view. Even if parts of the view aren’t visible after zooming, the radar still detects moving objects in the entire view. The only way to exclude detected movement is to add exclusion zones.
   • You can adjust the pan and zoom at any time from the Map calibration, Exclusion zones, or Scenarios pages by clicking .

8. Go to Radar position and use the buttons to move or rotate the position of the radar on the map.

1. Open the camera’s web interface in two browser windows and place them next to each other.

2. In the first window, go to Video > Stream.

3. In the second window, go to Radar > Settings > Object visualization and set Trail lifetime to one hour.

   Moving objects such as humans, vehicles, bushes, and flags leave trails that are visible for one hour.

4. Pan the camera to the area of interest. The green lines in the radar stream indicate what’s currently covered by the camera’s field of view.

5. Make a colleague walk along the border of the area of interest and around static objects such as buildings and containers.

The trails from the walk show the boundaries of the area of interest and static objects. Use them as a basis for shaping and placing zones and lines in your radar scenarios. Bushes, flags, and other objects that move in the wind also leave trails that you can use to create exclusion zones to minimize false alarms.

With radar scenarios, the radar can discover and classify objects that move in the scene. Radar scenarios are required to set up autotracking, as each tracking profile is based on a radar scenario.

You can create several radar scenarios to detect different behaviors, object types, or for different parts of the scene. You create corresponding autotracking profiles for each radar scenario.

There are two types of radar scenarios:

Movement in area

Detects objects that move inside an area that you define.

Line crossing

Detects objects that cross one or two lines that you define.

Example: Movement in area scenario - Humans in sensitive area

In this example, we want to detect humans that move in a part of the scene where they aren’t supposed to be. We are only interested in objects that stay in the area for at least 5 seconds. In a small section of the area we don’t want to start autotracking.

1. Click Add scenario.

2. Name the scenario Humans in sensitive area.

3. In Triggering conditions, select Movement in area.

4. Click Next.

5. Select the shape preset for your zone.

   Use the mouse to move and adjust the zone to cover the desired part of the scene.

6. Click Next.

7. To only track objects that stay in the area for at least 5 seconds, set Seconds until trigger to 5.

8. In Trigger on object type, select humans.

9. Click Next.

10. Click Save.

11. Go to Exclude zones and click Add exclude zone.

12. Click to expand the new exclude zone.

13. Use the mouse to move and adjust the exclude zone to cover the part of the scenario zone where you don’t want objects to trigger the scenario.

14. Turn on Track passing objects to continue tracking objects that just pass through the exclude zone.

Example: Line-crossing scenario - Cars passing gate

In this example, we want to discover and classify vehicles that pass through a gate. To minimize false alarms, the vehicles must cross two lines before they trigger autotracking to start.

1. Click Add scenario.

2. Name the scenario Cars passing gate.

3. In Triggering conditions, select Line crossing.

4. Click Next.

5. Turn on Require crossing of two lines.

6. Use the mouse to position the lines. Allow some distance between the gate and the lines.

7. If needed, change the direction that vehicles are required to move.

8. Click Next.

9. In Max time between crossings, set the time limit between crossing the first and the second line.

10. In Trigger on object type, clear humans and select vehicles.

11. Click Next.

12. Click Save.

Before you continue to create autotracking profiles, we recommend that you validate the installation. The validation can help you identify problems with the installation or manage static objects such as trees or reflective surfaces in the scene.

Check that there are no false detections

1. Check that the recognition zone is clear from human activity.

2. Wait for a few minutes to make sure the radar doesn’t detect any static objects in the recognition zone.

3. If there are unwanted detections, you can filter out certain types of movement or objects, adjust the radar scenario zones, or adjust the detection sensitivity. For instructions, see Minimize false alarms in the radar’s user manual at help.axis.com.

Check for the correct symbol, direction of travel, and position on map

1. Go to Radar > Stream and start a recording.

2. Start walking just outside the recognition zone, and walk directly toward the radar.

3. Check that a human classification symbol is shown when the person enters the recognition zone.

4. Check that the correct direction of travel is shown.

6. Check that the person’s actual position matches the position on the map.

Create a table similar to the one below to help you record the data from your validation.

Complete the radar validation

Once you have successfully completed the first part of the validation, perform the following tests to complete the validation process.

1. Make sure you have configured your devices according to the instructions.

2. Make sure you have added and calibrated a reference map (if there is a map available).

3. Set the radar scenario to trigger when a human is detected. By default, Seconds until trigger is set to two seconds but you can change this if needed.

4. Go to Radar > Settings > Object visualization and set the Trail lifetime to one hour so that it will safely exceed the time it takes to walk around the area of surveillance. The trail lifetime keeps the track in the radar's live view for the set time and, once you have finished the validation, you can disable it.

5. Walk along the border of the recognition zone and make sure that the trailing on the system matches the route that you walked.

6. If you are not satisfied with the results of your validation, re-calibrate the reference map and repeat the validation.

Validate the height alignment

1. Go to Analytics > Autotracking > Settings.

2. In Visual confirmation, turn on Video objects and Radar objects.

   When you look at objects in Analytics > Autotracking > Tracking profiles, a green bounding box around an object means it’s confirmed by video analytics. When an object starts to move, a white bounding box indicates that it’s confirmed by radar analytics.

3. Go to Tracking profiles and play the video stream.

4. Verify that the white bounding boxes surround the moving objects. If the boxes are above or below the object, you need to adjust the radar’s mounting height in Radar > Settings > General.

Each autotracking profile must be connected to one radar scenario. When a radar scenario is triggered, the corresponding autotracking profile is activated. The PTZ camera is directed to the object and the camera validates the classification. The camera then starts to track the object.

Example: Tracking profile - Humans in sensitive area

In this example, we want to track the humans detected in the Humans in sensitive area radar scenario. We want to continue tracking them as long as they move inside the inclusion area of the radar scenario and fulfills the scenario’s triggering conditions. We only want to track objects that the camera has classified as humans. In case there are moving objects that fulfill the criteria both for this and other radar scenarios with connected tracking profiles, we want to prioritize humans in the sensitive area, so we set a higher priority on this profile than others.

1. Go to Analytics > Autotracking > Tracking profiles.

2. Click + Create.

3. Select the radar scenario Humans in sensitive area.

4. In Tracking profile name, type AT: Humans in sensitive area.

5. In Tracking criteria, select Object triggers radar scenario.

6. Turn on Object type verification to only track objects that both the radar and the camera has classified as humans.

7. Clear all object types except Human to only track objects that the camera has classified as a human.

8. Set Priority to 1 Highest.

9. Click Save.

Example: Tracking profile - Cars passing gate

In this example, we want to track the objects detected in the Cars passing gate radar scenario. We want to continue tracking the object until neither the radar nor the camera can detect the object any longer. We only want to track objects that the camera has classified as cars. In case there are moving objects that fulfill the criteria both for this and other radar scenarios with connected tracking profiles, we want to prioritize the other ones, so we set a lower priority on this profile than the others.

1. Go to Analytics > Autotracking > Tracking profiles.

2. Click + Create.

3. Select the radar scenario Cars passing gate.

4. In Tracking profile name, type AT: Cars passing gate.

5. In Tracking criteria, select Object detected by radar or camera.

6. Turn on Object type verification to only track objects that both the radar and the camera has classified.

7. Clear all object types except Car to only track objects that the camera has classified as a car.

8. Set Priority to 5 Lowest.

9. Click Save.

You can record both the video stream and the radar stream when autotracking is activated. The radar stream recording shows you where the object came from, and the video stream recording shows you a zoomed-in view of the tracked object.

Example: Create rule - Record video stream

In this example we record the video stream to the camera’s SD card when autotracking is active.

1. Go to System > Events and add a rule.

2. Give the rule a name, for example Record video when autotracking is activated.

3. In the list of conditions, select PTZ Autotracking: Is tracking.

4. In the list of actions, under Recordings, select Record video while the rule is active.

5. In Storage, select the SD card.

6. In Camera, select Camera 1.

   To record the radar stream instead, select Radar 1.

7. Set the Prebuffer to 5 seconds to make sure you record from the start.

8. Click Save.