Optimizing the operation of neural analytics on GPU in Linux OS

General information

It can take several minutes to launch neural analytics algorithms on Nvidia GPU after server restart. Meanwhile, the neural models are optimized for the current GPU type.

You can use the caching function to ensure that this operation is performed only once. Caching saves the optimization results on the hard drive and uses them for the subsequent analytics runs. 

Starting with DetectorPack 3.9, a utility was added to the Neuro Pack add-ons (see Installing DetectorPack add-ons), which allows you to create GPU neural network caches without using Axxon One. The presence of the cache speeds up the initialization and optimizes video memory consumption.

Optimizing the operation of neural analytics on GPU

To optimize the operation of the neural analytics on GPU, do the following:

1. Stop the server (see Starting and stopping the Axxon One Server in Linux OS).
   

   Attention!

   If the system has the software running on GPU, you must stop its operation.

2. Login as ngp superuser:

   1. In the command prompt, run the command:

      sudo su ngp

   2. Enter the password for the superuser.

3. Create a folder with a custom name to store the cache. For example:
   

   mkdir /opt/AxxonSoft/AxxonOne/gpucache

4. Change folder permissions:
   

   chmod -R 777 /opt/AxxonSoft/AxxonOne/gpucache

5. Go to the /opt/AxxonSoft/AxxonOne folder:
   

   cd /opt/AxxonSoft/AxxonOne

6. Open the server configuration file for editing:
   

   nano instance.conf

   Attention!

   When you use the server in failover mode, you should:

   1. Open the /etc/AxxonSoft folder:
      

      cd /etc/AxxonSoft

   2. Open the server configuration file for editing:
      

      nano axxon-one.conf

      Attention!

      Before you make changes to the axxon-one.conf file, we recommend reading the instructions on the page Creating system variables for the Axxon One server in the failover mode in Linux OS.

   3. Add the GPU_CACHE_DIR system variable in the configuration file, where the value will specify the path to the cache location folder:
      

      export GPU_CACHE_DIR="/opt/AxxonSoft/AxxonOne/gpucache"

   4. Save the changes in the server configuration file.
   5. Add the GPU_CACHE_DIR system variable in the /etc/profile file:
      

      export GPU_CACHE_DIR="/opt/AxxonSoft/AxxonOne/gpucache"

   6. Run the command:
      

      source /etc/profile

7. Add the GPU_CACHE_DIR system variable, where the value will specify the path to the cache location folder. For example:

   export GPU_CACHE_DIR="/opt/AxxonSoft/AxxonOne/gpucache"

8. Save the server configuration file using the Ctrl+O keyboard shortcut.

9. Exit file editing mode using the Ctrl+X keyboard shortcut.
   

   Attention!

   When you change the server configuration (see Changing the configuration of the Axxon One Server in Linux OS) or update to the new Axxon One version, the system variables previously added to the instance.conf configuration file will be deleted (see Creating system variables for the Axxon One server in Linux OS).

10. In the command prompt, run the command that was used to add the system variable with the path to the cache location folder. For example:
    

    export GPU_CACHE_DIR="/opt/AxxonSoft/AxxonOne/gpucache"

11. Go to the /opt/AxxonSoft/DetectorPack/ folder:

    cd /opt/AxxonSoft/DetectorPack

12. Run the following command:

    ./NeuroPackGpuCacheGenerator

    Attention!

    If more than one Nvidia GPU is available, you will be able to select one. To do this, specify a number from 0 to 3, which corresponds to the required device in the list.

Optimizing the operation of the neural analytics on GPU is complete. The utility will create the caches of four neural networks included in the Neuro Pack add-ons:

• GeneralNMHuman_v1.0GPU_onnx.ann—person;
• smokeScanned_v1_onnx.ann (or bestSmoke_v1.ann starting with Detector Pack 3.14)—smoke detection;
• fireScanned_v1_onnx.ann (or bestFire_v1.ann starting with Detector Pack 3.14)—fire detection;
• reid_15_0_256__osnetfpn_segmentation_noise_20_common_29_onnx.ann—search for the similar in the Neural tracker (see Similitude search).
  

Creating GPU neural network caches using parameters

1. -p is a parameter to create a cache for a particular neural network.
   Command example:
   

   ./NeuroPackGpuCacheGenerator -p /opt/AxxonSoft/DetectorPack/NeuroSDK/GeneralNMHumanAndVehicle_Nano_v1.0_GPU_onnx.ann

2. -v is a parameter to output the procedure log to the console during cache generation.
   Command example to automatically create caches of four neural networks included in the Neuro Pack add-ons with log output:
   

   ./NeuroPackGpuCacheGenerator -v

3. --int8=1 is a parameter to create a quantized version of the cache for those neural networks for which quantization is available. By default, the --int8=0 parameter is disabled.
   Command example:
   

   ./NeuroPackGpuCacheGenerator -p /opt/AxxonSoft/DetectorPack/NeuroSDK/GeneralNMHumanAndVehicle_Nano_v1.0_GPU_onnx.ann --int8=1

   Attention!

   The neural networks for which the quantization mode is available are included in the Neuro Pack add-ons together with the *.info file.

The neural networks for which the quantization mode is available (see Neural tracker, Stopped object detector, Neural counter):

• GeneralNMCar_v1.0GPU_onnx.ann—Vehicle.
• GeneralNMHuman_v1.0GPU_onnx.ann—Person.
• GeneralNMHumanTopView_v0.8GPU_onnx.ann—Person (top-down view).

Starting with DetectorPack 3.11, the following neural networks were added:

• GeneralNMHumanAndVehicle_Nano_v1.0_GPU_onnx.ann—Person and vehicle (Nano).
• GeneralNMHumanAndVehicle_Medium_v1.0_GPU_onnx.ann—Person and vehicle (Medium).
• GeneralNMHumanAndVehicle_Large_v1.0_GPU_onnx.ann—Person and vehicle (Large).

Starting with DetectorPack 3.12, the following neural networks were added:

• GeneralNMHumanTopView_Nano_v1.0_GPU_onnx.ann—Person (top-down view Nano).
• GeneralNMHumanTopView_Medium_v1.0_GPU_onnx.ann—Person (top-down view Medium).
• GeneralNMHumanTopView_Large_v1.0_GPU_onnx.ann—Person (top-down view Large).