Unable to find the source frame ID of the keyframe

[SSLotfi]

I wanted to start by saying Great work on this model!

Main issue

I have a simple issue with the output, I need to view the keyframes as a 360 image and to do so I need to know the keyframe ID in order to extract it from the original video but the output doesn't contain the keyframe IDs but only the path.

Description

I had noticed in the earlier version of stella-vslam in the commit on 21 August, that you had removed the src_frm_ID from the output map.msg file which was the keyframe ID, so could you elaborate on what happened to this value, why was it removed and how can I obtain it again?

Environment

• commit: 14b1866
• OS: Ubuntu 22.04

Thanks in advance!

[ymd-stella]

Frame IDs were removed because the timestamps have enough information to identify the image. Is it possible to identify the frame from the timestamp?

[ymd-stella]

I need to view the keyframes as a 360 image

Why?

If you could clarify your objectives, we may be able to suggest a more suitable method. I don't think it is a good idea to use keyframes for other purposes.

[ymd-stella]

I need to know how the location looks like at specific points on the path

Why?
Do you want to draw the walking path on the image?

[SSLotfi]

Sorry if it wasn't clear enough my use case is a bit hard to explain, I will try to elaborate more.

After the path extraction, I'm able to obtain the x and y coordinates of the path, so imagine having a plot of x,y coordinates representing the extracted path, I would like to click on any point and view the corresponding frame at this point, so I need a link between the frames in the video and the points on the path. That concludes my first question.

My second question is what are keyframes exactly? what do they represent?

[ymd-stella]

I understand. For that application, instead of using keyframes, you should refer to the pose of the frames and sample them equally spaced.

My second question is what are keyframes exactly? what do they represent?

It is only part of the data structure that represents the map. The map is for tracking purposes only and is not intended for human use. You should not assume that keyframes will be placed exactly as you imagine.

[SSLotfi]

Okay then how can I obtain the frame IDs from the timestamp please?

This is a sample output, could you give an example on this?

The command used to get this output is the following, I set the --frame-skip to 3 and the video is 29.7 fps

./run_video_slam  -v ./orb_vocab.fbow -m  /path/to/video.MP4 -c  /path/to/config.yaml --frame-skip 3 --no-sleep --eval-log-dir /path/to/output_folder  --map-db-out map.msg

1661695788.91132 -0 -0 -0 0 0 0 1
1661695789.01132 -0.0266124379 0.00635749787 0.0241463001 0.00100252163 9.95488459e-05 -0.000310600583 0.999999444
1661695789.11132 -0.0554418823 0.0110921471 0.0568059602 0.00156517621 -4.14130338e-05 -0.00116874327 0.999998091
1661695789.21132 -0.0883599818 0.0173819152 0.0959747806 0.00103792236 -1.72606073e-05 -0.00234030677 0.999996723
1661695789.31132 -0.115218409 0.01874261 0.137738652 0.000957893967 -0.000249698461 -0.00334207936 0.999993925
1661695789.41132 -0.141547001 0.0180454328 0.183253272 0.00058188919 -0.00127368486 -0.00486931481 0.999987164
1661695789.51132 -0.150809062 0.0183660894 0.209293191 0.00157625091 -0.000133846209 -0.00466058601 0.999987888
1661695789.61132 -0.168981001 0.0178397117 0.252611263 0.00181377154 -0.000257508744 -0.00563695243 0.999982434
1661695789.71132 -0.186417071 0.0209167921 0.296421624 0.00193758222 0.000105293403 -0.00609759185 0.999979527
1661695789.81132 -0.204499331 0.0242739225 0.346753953 0.00200832468 0.000236211889 -0.00726515789 0.999971564
1661695789.91132 -0.224561406 0.0235497472 0.40150214 0.00192071893 0.000696959757 -0.00878151491 0.999959354
1661695790.01132 -0.239378215 0.0187244458 0.459944596 0.00176666483 0.00143779677 -0.0103705007 0.999943631
1661695790.11132 -0.248896609 0.0182591537 0.52257208 0.00239853629 0.00221206241 -0.0124765081 0.999916842
1661695790.21132 -0.244830654 0.0243643643 0.584518561 0.00271268577 0.00162917597 -0.0131695494 0.999908271
1661695790.31132 -0.232577644 0.0360277904 0.633274715 0.00346270058 0.00168389844 -0.0134473401 0.999902167
1661695790.41132 -0.218250942 0.0483799906 0.703868128 0.00374405169 0.00159084166 -0.0128475564 0.999909192
1661695790.51132 -0.201035504 0.0538505161 0.780160533 0.00297659353 0.00163747924 -0.0120960473 0.999921069
1661695790.61132 -0.188741165 0.0517338479 0.856347349 0.00266675655 0.0015580081 -0.0112237723 0.999932242
1661695790.71132 -0.181031068 0.0490746065 0.933517522 0.0029986164 0.0017205322 -0.0107454945 0.999936289
1661695790.81132 -0.179483396 0.0519792205 1.00539218 0.00430610216 0.00170534701 -0.0101635188 0.999937624
1661695790.91132 -0.177460755 0.0599298189 1.08456196 0.00523363646 0.0017752514 -0.0102380322 0.999932318
1661695791.01132 -0.18019655 0.0742235174 1.16976006 0.00526290496 0.00181580993 -0.0102751273 0.999931711
1661695791.11132 -0.180795271 0.0799971336 1.25694339 0.0048628442 0.00227374481 -0.010308858 0.999932453
1661695791.21132 -0.188950018 0.0749372284 1.34774328 0.00483473721 0.00241619773 -0.0103970804 0.999931342
1661695791.31132 -0.193737478 0.0656727686 1.41275256 0.00489236601 0.00216263789 -0.011696347 0.999917288
1661695791.41132 -0.196149487 0.056021847 1.49034909 0.0054316625 0.00195681251 -0.0120749203 0.999910428
1661695791.51132 -0.193886464 0.0560215591 1.57688532 0.00680414615 0.00194840278 -0.0118083381 0.999905231
1661695791.61132 -0.189114753 0.0651810416 1.66768814 0.00706980507 0.00220011022 -0.011586781 0.999905457
1661695791.71132 -0.182879984 0.0690656123 1.76458224 0.00672324664 0.00245939655 -0.0108475053 0.999915537
1661695791.81132 -0.176994693 0.0674607267 1.86074577 0.00726154398 0.00264485239 -0.0100934928 0.999919195
1661695791.91132 -0.179287948 0.0639561028 1.95905534 0.00761589207 0.00213962041 -0.00962976022 0.999922341
1661695792.01132 -0.18674747 0.0625035812 2.04696874 0.00760301445 0.00241219604 -0.00894712557 0.99992816
1661695792.11132 -0.197347469 0.0665234826 2.13341841 0.0079361498 0.00235984895 -0.00879936975 0.999927007
1661695792.21132 -0.206588078 0.0784774846 2.22245241 0.00819324978 0.00226618012 -0.00919210721 0.999921617
1661695792.31132 -0.219221912 0.0839032548 2.31789094 0.00794206451 0.00244564067 -0.00969323541 0.999918488
1661695792.41132 -0.233058392 0.0741032536 2.39143324 0.00732933012 0.00187677444 -0.0099134801 0.999922238
1661695792.51132 -0.240976389 0.0623423792 2.47790634 0.00795902476 0.00241334379 -0.0102285903 0.999913099
1661695792.61132 -0.248665901 0.0589461984 2.56064041 0.00900745174 0.00217329798 -0.0102695206 0.999904335

[ymd-stella]

how can I obtain the frame IDs from the timestamp please?

Please read the text carefully. If you are going to put me through more effort than necessary, I will stop replying.

Time stamps are not required. Frames should be selected based on pose.

[zoldaten]

@SSLotfi u can try to:

• record rosbag /image_raw while stella_vslam operates

• extract rosbag images from rosbag. it`ll looks like:
  

• links two datasets where first:

  • timestamps u got in (for example) stella_vslam

   1661695788.91132 -0 -0 -0 0 0 0 1
   1661695789.01132 -0.0266124379 0.00635749787 0.0241463001 0.00100252163 9.95488459e-05 -0.000310600583 0.999999444
   1661695789.11132 -0.0554418823 0.0110921471 0.0568059602 0.00156517621 -4.14130338e-05 -0.00116874327 0.999998091
   1661695789.21132 -0.0883599818 0.0173819152 0.0959747806 0.00103792236 -1.72606073e-05 -0.00234030677 0.999996723
  

  • timestamps in picture names from rosbag

  Timestamps as i understand - its the first column:

  timestamp x y z q_x q_y q_z q_w
  1661695789.01132 -0.0266124379 0.00635749787 0.0241463001 0.00100252163 9.95488459e-05 -0.000310600583 0.999999444
  

  @ymd-stella by the way is it possible to extact something usefull from map.msg with msgpack-python ? i know map.msg is for "technical purpose" but may be ?

[ymd-stella]

Would the following code be helpful?

https://gist.github.com/ymd-stella/a60edd4242dd99ee4bcbd0747031d524

[zoldaten]

yes, that`s what i need!

[zoldaten]

here is the way how to link keyframes to images in web-cam(or realsense) recording:

1. record rosbag with stella_vslam working:
   ros2 bag record /camera/color/image_raw.
2. start stella_vslam with map-out:
   ros2 run stella_vslam_ros run_slam -v ~/lib/stella_vslam_examples/build/orb_vocab.fbow -c /home/rock/lib/stella_vslam/example/tum_rgbd/TUM_RGBD_rgbd_1.yaml --map-db-out map1.msg
3. dont forget to start web_cam or realsense:
   ros2 launch realsense2_camera rs_launch.py
4. stop all nodes and covert ros_bag to ros1_bag (its needes to exctract images from it). may be there is ros2 bag coverter but i didnt find it. rosbags-convert <ros2_bag_dir_name> --dst ros1.bag`
5. extract images from rosbag (i used Kalibr docker image):

/catkin_ws/src/kalibr/aslam_offline_calibration/kalibr/python# python3 kalibr_bagextractor \
--image-topics /camera/color/image_raw --output-folder . --bag /data/data/ros1.bag

i`ll get all images from ros bag in *.png
6. convert map.msg u got with stella_vslam:
msg_to_traectory.zip

u`ll get txt like this:

1695364618.462801 0.0 0.0 0.0 0.0 0.0 0.0 1.0
1695364620.478818 0.0011010916949759438 -0.00041996785716774194 -0.001717261608303873 -0.000282630648990967 -0.0008260097666920766 -0.00045116872139978686 0.9999995171371667
1695364622.494783 0.00022203964554724881 -0.0003170794203157155 -0.0011760126576225091 -0.0002145761633387283 -2.9074966151891167e-05 -0.0003957251185042574 0.9999998982566684

7. use script that finds and copy-paste keyframe images from dir with all images:
   find_keyframe_images.zip
   it takes time from txt file in format 1695364618.46 finds in dir with all images the name of image with close time name to it and copy-paste it to out dir.

[zovelsanj]

Hi @SSLotfi, if you are still looking for src_frm_id in map.msg then I have modified the code accordingly in the latest version of the stella_vslam. You can find it in this commit.