OAK-D S2の続き。
概要
前回のサンプルから、検出した手を3Dで表示するサンプルがあったので、それをやってみる。
使ったサンプルはこれ。
depthai_hand_trackerのリポジトリの中の、examplesディレクトリ内にあるもの。
前回、前々回に使ったものの中に含まれている。
なお、今回の記事は、Jupyterで実行した結果を、一部編集して掲載している。
環境構築
Open3Dのインストールが必要。
(Jupyterセル上で実行のため、!
を付けている)
!pip install open3d
クリックでログ開く/閉じる
Collecting open3d
Downloading open3d-0.18.0-cp38-cp38-win_amd64.whl.metadata (4.1 kB)
Requirement already satisfied: numpy>=1.18.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from open3d) (1.24.4)
Collecting dash>=2.6.0 (from open3d)
Downloading dash-2.16.1-py3-none-any.whl.metadata (10 kB)
Collecting werkzeug>=2.2.3 (from open3d)
Downloading werkzeug-3.0.2-py3-none-any.whl.metadata (4.1 kB)
Collecting nbformat>=5.7.0 (from open3d)
Downloading nbformat-5.10.4-py3-none-any.whl.metadata (3.6 kB)
Collecting configargparse (from open3d)
Downloading ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB)
Collecting ipywidgets>=8.0.4 (from open3d)
Downloading ipywidgets-8.1.2-py3-none-any.whl.metadata (2.4 kB)
Collecting Flask<3.1,>=1.0.4 (from dash>=2.6.0->open3d)
Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Collecting plotly>=5.0.0 (from dash>=2.6.0->open3d)
Downloading plotly-5.20.0-py3-none-any.whl.metadata (7.0 kB)
Collecting dash-html-components==2.0.0 (from dash>=2.6.0->open3d)
Downloading dash_html_components-2.0.0-py3-none-any.whl.metadata (3.8 kB)
Collecting dash-core-components==2.0.0 (from dash>=2.6.0->open3d)
Downloading dash_core_components-2.0.0-py3-none-any.whl.metadata (2.9 kB)
Collecting dash-table==5.0.0 (from dash>=2.6.0->open3d)
Downloading dash_table-5.0.0-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: importlib-metadata in c:\users\a\.conda\envs\depthai\lib\site-packages (from dash>=2.6.0->open3d) (7.1.0)
Requirement already satisfied: typing-extensions>=4.1.1 in c:\users\a\.conda\envs\depthai\lib\site-packages (from dash>=2.6.0->open3d) (4.10.0)
Requirement already satisfied: requests in c:\users\a\.conda\envs\depthai\lib\site-packages (from dash>=2.6.0->open3d) (2.31.0)
Collecting retrying (from dash>=2.6.0->open3d)
Downloading retrying-1.3.4-py3-none-any.whl.metadata (6.9 kB)
Requirement already satisfied: nest-asyncio in c:\users\a\.conda\envs\depthai\lib\site-packages (from dash>=2.6.0->open3d) (1.6.0)
Requirement already satisfied: setuptools in c:\users\a\.conda\envs\depthai\lib\site-packages (from dash>=2.6.0->open3d) (68.2.2)
Requirement already satisfied: comm>=0.1.3 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipywidgets>=8.0.4->open3d) (0.2.2)
Requirement already satisfied: ipython>=6.1.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipywidgets>=8.0.4->open3d) (8.12.0)
Requirement already satisfied: traitlets>=4.3.1 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipywidgets>=8.0.4->open3d) (5.14.2)
Collecting widgetsnbextension~=4.0.10 (from ipywidgets>=8.0.4->open3d)
Downloading widgetsnbextension-4.0.10-py3-none-any.whl.metadata (1.6 kB)
Collecting jupyterlab-widgets~=3.0.10 (from ipywidgets>=8.0.4->open3d)
Downloading jupyterlab_widgets-3.0.10-py3-none-any.whl.metadata (4.1 kB)
Collecting fastjsonschema>=2.15 (from nbformat>=5.7.0->open3d)
Downloading fastjsonschema-2.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting jsonschema>=2.6 (from nbformat>=5.7.0->open3d)
Downloading jsonschema-4.21.1-py3-none-any.whl.metadata (7.8 kB)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in c:\users\a\.conda\envs\depthai\lib\site-packages (from nbformat>=5.7.0->open3d) (4.12.0)
Collecting MarkupSafe>=2.1.1 (from werkzeug>=2.2.3->open3d)
Downloading MarkupSafe-2.1.5-cp38-cp38-win_amd64.whl.metadata (3.1 kB)
Collecting Jinja2>=3.1.2 (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d)
Using cached Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB)
Collecting itsdangerous>=2.1.2 (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d)
Downloading itsdangerous-2.1.2-py3-none-any.whl.metadata (2.9 kB)
Collecting click>=8.1.3 (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d)
Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting blinker>=1.6.2 (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d)
Downloading blinker-1.7.0-py3-none-any.whl.metadata (1.9 kB)
Requirement already satisfied: zipp>=0.5 in c:\users\a\.conda\envs\depthai\lib\site-packages (from importlib-metadata->dash>=2.6.0->open3d) (3.17.0)
Requirement already satisfied: backcall in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.0)
Requirement already satisfied: decorator in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (5.1.1)
Requirement already satisfied: jedi>=0.16 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.19.1)
Requirement already satisfied: matplotlib-inline in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.1.6)
Requirement already satisfied: pickleshare in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (3.0.42)
Requirement already satisfied: pygments>=2.4.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.17.2)
Requirement already satisfied: stack-data in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.6.2)
Requirement already satisfied: colorama in c:\users\a\.conda\envs\depthai\lib\site-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.4.6)
Collecting attrs>=22.2.0 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting importlib-resources>=1.4.0 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting pkgutil-resolve-name>=1.3.10 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading pkgutil_resolve_name-1.3.10-py3-none-any.whl.metadata (624 bytes)
Collecting referencing>=0.28.4 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading referencing-0.34.0-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema>=2.6->nbformat>=5.7.0->open3d)
Downloading rpds_py-0.18.0-cp38-none-win_amd64.whl.metadata (4.2 kB)
Requirement already satisfied: pywin32>=1.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from jupyter-core!=5.0.*,>=4.12->nbformat>=5.7.0->open3d) (227)
Collecting tenacity>=6.2.0 (from plotly>=5.0.0->dash>=2.6.0->open3d)
Downloading tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
Requirement already satisfied: packaging in c:\users\a\.conda\envs\depthai\lib\site-packages (from plotly>=5.0.0->dash>=2.6.0->open3d) (24.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\a\.conda\envs\depthai\lib\site-packages (from requests->dash>=2.6.0->open3d) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\a\.conda\envs\depthai\lib\site-packages (from requests->dash>=2.6.0->open3d) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\a\.conda\envs\depthai\lib\site-packages (from requests->dash>=2.6.0->open3d) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\a\.conda\envs\depthai\lib\site-packages (from requests->dash>=2.6.0->open3d) (2024.2.2)
Requirement already satisfied: six>=1.7.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from retrying->dash>=2.6.0->open3d) (1.16.0)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in c:\users\a\.conda\envs\depthai\lib\site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.8.3)
Requirement already satisfied: wcwidth in c:\users\a\.conda\envs\depthai\lib\site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.13)
Requirement already satisfied: executing>=1.2.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in c:\users\a\.conda\envs\depthai\lib\site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.4.1)
Requirement already satisfied: pure-eval in c:\users\a\.conda\envs\depthai\lib\site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.2)
Downloading open3d-0.18.0-cp38-cp38-win_amd64.whl (62.9 MB)
---------------------------------------- 0.0/62.9 MB ? eta -:--:--
---------------------------------------- 0.7/62.9 MB 15.3 MB/s eta 0:00:05
...
--------------------------------------- 62.9/62.9 MB 21.1 MB/s eta 0:00:01
---------------------------------------- 62.9/62.9 MB 15.6 MB/s eta 0:00:00
Downloading dash-2.16.1-py3-none-any.whl (10.2 MB)
---------------------------------------- 0.0/10.2 MB ? eta -:--:--
--- ------------------------------------ 0.8/10.2 MB 16.3 MB/s eta 0:00:01
...
--------------------------------------- 10.2/10.2 MB 19.2 MB/s eta 0:00:01
---------------------------------------- 10.2/10.2 MB 18.1 MB/s eta 0:00:00
Downloading dash_core_components-2.0.0-py3-none-any.whl (3.8 kB)
Downloading dash_html_components-2.0.0-py3-none-any.whl (4.1 kB)
Downloading dash_table-5.0.0-py3-none-any.whl (3.9 kB)
Downloading ipywidgets-8.1.2-py3-none-any.whl (139 kB)
---------------------------------------- 0.0/139.4 kB ? eta -:--:--
---------------------------------------- 139.4/139.4 kB ? eta 0:00:00
Downloading nbformat-5.10.4-py3-none-any.whl (78 kB)
---------------------------------------- 0.0/78.5 kB ? eta -:--:--
---------------------------------------- 78.5/78.5 kB 4.5 MB/s eta 0:00:00
Downloading werkzeug-3.0.2-py3-none-any.whl (226 kB)
---------------------------------------- 0.0/226.8 kB ? eta -:--:--
---------------------------------------- 226.8/226.8 kB ? eta 0:00:00
Downloading ConfigArgParse-1.7-py3-none-any.whl (25 kB)
Downloading fastjsonschema-2.19.1-py3-none-any.whl (23 kB)
Downloading flask-3.0.3-py3-none-any.whl (101 kB)
---------------------------------------- 0.0/101.7 kB ? eta -:--:--
---------------------------------------- 101.7/101.7 kB 6.1 MB/s eta 0:00:00
Downloading jsonschema-4.21.1-py3-none-any.whl (85 kB)
---------------------------------------- 0.0/85.5 kB ? eta -:--:--
---------------------------------------- 85.5/85.5 kB ? eta 0:00:00
Downloading jupyterlab_widgets-3.0.10-py3-none-any.whl (215 kB)
---------------------------------------- 0.0/215.0 kB ? eta -:--:--
--------------------------------------- 215.0/215.0 kB 13.7 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.5-cp38-cp38-win_amd64.whl (17 kB)
Downloading plotly-5.20.0-py3-none-any.whl (15.7 MB)
---------------------------------------- 0.0/15.7 MB ? eta -:--:--
--- ------------------------------------ 1.2/15.7 MB 25.4 MB/s eta 0:00:01
...
--------------------------------------- 15.7/15.7 MB 21.8 MB/s eta 0:00:01
---------------------------------------- 15.7/15.7 MB 20.4 MB/s eta 0:00:00
Downloading widgetsnbextension-4.0.10-py3-none-any.whl (2.3 MB)
---------------------------------------- 0.0/2.3 MB ? eta -:--:--
---------- ----------------------------- 0.6/2.3 MB 19.5 MB/s eta 0:00:01
---------------------------- ----------- 1.6/2.3 MB 20.7 MB/s eta 0:00:01
---------------------------------------- 2.3/2.3 MB 18.3 MB/s eta 0:00:00
Downloading retrying-1.3.4-py3-none-any.whl (11 kB)
Downloading attrs-23.2.0-py3-none-any.whl (60 kB)
---------------------------------------- 0.0/60.8 kB ? eta -:--:--
---------------------------------------- 60.8/60.8 kB ? eta 0:00:00
Downloading blinker-1.7.0-py3-none-any.whl (13 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
---------------------------------------- 0.0/97.9 kB ? eta -:--:--
---------------------------------------- 97.9/97.9 kB 5.8 MB/s eta 0:00:00
Downloading importlib_resources-6.4.0-py3-none-any.whl (38 kB)
Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Using cached Jinja2-3.1.3-py3-none-any.whl (133 kB)
Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Downloading pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Downloading referencing-0.34.0-py3-none-any.whl (26 kB)
Downloading rpds_py-0.18.0-cp38-none-win_amd64.whl (206 kB)
---------------------------------------- 0.0/206.7 kB ? eta -:--:--
---------------------------------------- 206.7/206.7 kB ? eta 0:00:00
Downloading tenacity-8.2.3-py3-none-any.whl (24 kB)
Installing collected packages: fastjsonschema, dash-table, dash-html-components, dash-core-components, widgetsnbextension, tenacity, rpds-py, retrying, pkgutil-resolve-name, MarkupSafe, jupyterlab-widgets, itsdangerous, importlib-resources, configargparse, click, blinker, attrs, werkzeug, referencing, plotly, Jinja2, jsonschema-specifications, Flask, jsonschema, ipywidgets, dash, nbformat, open3d
Successfully installed Flask-3.0.3 Jinja2-3.1.3 MarkupSafe-2.1.5 attrs-23.2.0 blinker-1.7.0 click-8.1.7 configargparse-1.7 dash-2.16.1 dash-core-components-2.0.0 dash-html-components-2.0.0 dash-table-5.0.0 fastjsonschema-2.19.1 importlib-resources-6.4.0 ipywidgets-8.1.2 itsdangerous-2.1.2 jsonschema-4.21.1 jsonschema-specifications-2023.12.1 jupyterlab-widgets-3.0.10 nbformat-5.10.4 open3d-0.18.0 pkgutil-resolve-name-1.3.10 plotly-5.20.0 referencing-0.34.0 retrying-1.3.4 rpds-py-0.18.0 tenacity-8.2.3 werkzeug-3.0.2 widgetsnbextension-4.0.10
カレントディレクトリは、depthai_hand_trackerリポジトリの下に移動する必要がある。
%cd C:/work/oak-d_test/depthai_hand_tracker/
C:\work\oak-d_test\depthai_hand_tracker
コード変更
元のdemo.pyだと、画像ウィンドウのサイズが変えられなくてスクリーンキャプチャしづらいので、変更する。
下記参考で。
下記行を、コードの最初のほうに追加。
大体どこでも大丈夫だが、cv2.imshow()
を実行するより前のところで。
cv2.namedWindow('HandTracker', cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO)
実行
Jupyterセルでは、%run
コマンドでスクリプトファイル実行できるということだった。
ので、こんな感じ。
%run examples/3d_visualization/demo.py
また、スクリプトファイルだけ実行すると、OpenCVとOpen3Dの画像ウィンドウが残ってしまう。
下記のコード追加で、画像ウィンドウを閉じられる。
%run examples/3d_visualization/demo.py cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Open3Dのドキュメントも参考にした。
open3d.visualization.Visualizer - Open3D 0.18.0 documentation
うまく動かない…
色々動かしてみたが、どうしても動作してからしばらくすると止まってしまった。
- 手が少し認識されるか、これから認識されようというところでフリーズ、デバイス通信エラーのメッセージが出て終了する
- PCと接続するポートをThunderbolt4、USB3.0と変えても、状況は同じ、ポートの電力容量の問題ではない?
実行結果はこんな感じ。
%run examples/3d_visualization/demo.py
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
PD post processing blob : C:\work\oak-d_test\depthai_hand_tracker\custom_models\PDPostProcessing_top2_sh1.blob
Sensor resolution: (1920, 1080)
Internal camera FPS set to: 36
Internal camera image size: 1152 x 648 - pad_h: 252
Creating pipeline...
Creating Color Camera...
Creating Palm Detection pre processing image manip...
Creating Palm Detection Neural Network...
Creating Palm Detection post processing Neural Network...
Creating Hand Landmark pre processing image manip...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File C:\work\oak-d_test\depthai_hand_tracker\examples\3d_visualization\demo.py:173
170 while True:
171 # Run hand tracker on next frame
172 if not pause:
--> 173 frame, hands, bag = tracker.next_frame()
174 if frame is None: break
175 # Render 2d frame
File c:\work\oak-d_test\depthai_hand_tracker\HandTrackerEdge.py:477, in HandTracker.next_frame(self)
474 cv2.imshow("pre_lm_manip", pre_lm_manip)
476 # Get result from device
--> 477 res = marshal.loads(self.q_manager_out.get().getData())
478 hands = []
479 for i in range(len(res.get("lm_score",[]))):
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'manager_out' (X_LINK_ERROR)'
対処
結局、エッジモード(OAK-D側で手検出処理実施)をやめると、エラーが起こりにくくなった。(起こらなくなったわけではない)
元のdemo.pyが、強制的にエッジモードにするようにしている(-e
の引数設定のところがコメントアウトされて、後ろで強制的にTrue
が入れられている)ので、エッジモードをやめられるように変更した。
やっぱり電力的な問題なのか。
エッジでの推論を動かすと、瞬間的に大きな電流が流れるとか?
それがThunberboltポートの容量も超えてしまってる?
うまくいったときのログの一例。
%run examples/3d_visualization/demo.py cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
FPS : 11.0 f/s (# frames = 1038)
# frames w/ no hand : 797 (76.8%)
# frames w/ palm detection : 823 (79.3%)
# frames w/ landmark inference : 241 (23.2%)- # after palm detection: 26 - # after landmarks ROI prediction: 215
On frames with at least one landmark inference, average number of landmarks inferences/frame: 1.93
# lm inferences: 465 - # failed lm inferences: 20 (4.3%)
実行結果
実行結果のGIF。
-m
(--mode_3d
)のオプションで動作が少し変わるので、それを試してみる。
image
(デフォルト), world
, raw_world
, mixed
の4パターン。
各モードの説明は、リポジトリのREADME.mdに記載あり。ただ、読んでみてもいまいち分からない。
image
%run examples/3d_visualization/demo.py cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
FPS : 11.0 f/s (# frames = 1038)
# frames w/ no hand : 797 (76.8%)
# frames w/ palm detection : 823 (79.3%)
# frames w/ landmark inference : 241 (23.2%)- # after palm detection: 26 - # after landmarks ROI prediction: 215
On frames with at least one landmark inference, average number of landmarks inferences/frame: 1.93
# lm inferences: 465 - # failed lm inferences: 20 (4.3%)
指の動きはそれっぽい。
画像上の大きさに比例して奥行き位置を設定してる感じ?
カメラ面に平行な面内の位置も、画像に従う感じ?
raw_world
%run examples/3d_visualization/demo.py -m raw_world cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
FPS : 12.1 f/s (# frames = 899)
# frames w/ no hand : 752 (83.6%)
# frames w/ palm detection : 764 (85.0%)
# frames w/ landmark inference : 147 (16.4%)- # after palm detection: 12 - # after landmarks ROI prediction: 135
On frames with at least one landmark inference, average number of landmarks inferences/frame: 1.94
# lm inferences: 285 - # failed lm inferences: 12 (4.2%)
image
のときと違い、手(の手首)の奥行き位置は固定。手の大きさも変わらない。
また、次のworld
と違い、手首を中心に、カメラ面内で回転させても、3D上にはそれは反映されない。
手首とほかのどこかを結ぶ線を決まった向きに表示している?
world
%run examples/3d_visualization/demo.py -m world cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
FPS : 16.2 f/s (# frames = 1372)
# frames w/ no hand : 1283 (93.5%)
# frames w/ palm detection : 1294 (94.3%)
# frames w/ landmark inference : 89 (6.5%)- # after palm detection: 11 - # after landmarks ROI prediction: 78
On frames with at least one landmark inference, average number of landmarks inferences/frame: 1.91
# lm inferences: 170 - # failed lm inferences: 10 (5.9%)
raw_world
の結果に対して、手首中心の回転が反映されるようになっている。
回転角度は画像上の向きに合わせて、という感じか?
mixed
%run examples/3d_visualization/demo.py -m mixed cv2.destroyAllWindows() renderer3d.vis3d.vis.destroy_window()
Palm detection blob : C:\work\oak-d_test\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob : C:\work\oak-d_test\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating MonoCameras, Stereo and SpatialLocationCalculator nodes...
RGB calibration lens position: 131
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (1 thread)...
Pipeline created.
Pipeline started - USB speed: SUPER
FPS : 8.5 f/s (# frames = 722)
# frames w/ no hand : 498 (69.0%)
# frames w/ palm detection : 511 (70.8%)
# frames w/ landmark inference : 224 (31.0%)- # after palm detection: 13 - # after landmarks ROI prediction: 211
On frames with at least one landmark inference, average number of landmarks inferences/frame: 1.96
# lm inferences: 439 - # failed lm inferences: 11 (2.5%)
Spatial location requests round trip : 7.7 ms
これだけ、OAK-Dの3D撮影を利用している。
手首の位置の奥行き情報を使ってるのか、画像で、手首に3次元座標情報が重畳されている。
奥行き情報は、単純にステレオ画像から推定されたものを使ってるのかな?
検出したキーポイントでステレオマッチングしてる、ということはないか。
コードを見ればわかるかな。
また、3D表示には、カメラの位置と向きも重畳されている。
手首が画角外になると、3D表示の位置がおかしくなっている。
以上
いずれのモードでも、指の動きはそれなりに出ているような。
実際にピアノを弾きながら3D撮影してみたい。
次回どこまでできるか。