上海北斗导航创新研究院官方网站

  1. 首页
  2. 数据集
  3. NetEase AR

NEAR: The NetEase AR Oriented Visual Inertial Dataset

Abstract

The existing datasets for evaluating Visual Inertial Odometry (VIO) have boosted the research of autonomous agents, but they don’t meet the prosperous research of Augmented Reality (AR) or Mixed Reality (MR) given that they are not collected at real AR scenes and do not account for affecting factors of mobile devices. This paper presents the NEAR dataset, an AR oriented visual-inertial dataset collected with commodity handheld phones with ground truthrts. The dataset has a total of 113 sequences in 49 elaborately designed collection cases at two typical indoor scenes, i.e. the living area and the table area. It also covers plenty of setting adjustments for comparison, including the comparisons of different level textures, illuminations, motion patterns, camera settings and the difference between the rolling shutter and the global shutter.

If you wanna acknowledge our work, please cite

  1. Cheng Wang, Yu Zhao, Jiabin Guo, Ling Pei, Yue Wang, Haiwei Liu. NEAR: The NetEase AR Oriented Visual Inertial Dataset. [Accepted by ISMAR 2019].
caseenvironmentscenelightingmotion patternspecial device settingdevices and link
1Easy[1]Rich-textured desktopuniform & stablecircle (slow+normal)default[5]iPhone 7, Huawei P20
2EasyRich-textured desktopuniform & stablehybriddefaultiPhone 7, Huawei P20
2_control[4]EasyRich-textured desktopuniform & stablehybriddefaultiPhone 7, Huawei P20, MYNT
3EasyRich-textured desktopuniform & stablecircle (alwayse points to center)defaultiPhone 7, Huawei P20
3_controlEasyRich-textured desktopuniform & stablecircle (alwayse points to center)defaultiPhone 7, Huawei P20, MYNT
4EasyMedium-textured desktopuniform & stablecircle (slow+normal)defaultiPhone 7, Huawei P20
5EasyMedium-textured desktopuniform & stablehybriddefaultiPhone 7, Huawei P20
6EasyMedium-textured desktop with reflectionuniform & stablecircle (slow+normal)defaultiPhone 7, Huawei P20
7EasyMedium-textured desktop with reflectionuniform & stablehybriddefaultiPhone 7, Huawei P20
8EasyRich-textured livingroomuniform & stabletranslation (slow+normal)defaultiPhone 7, Huawei P20
9EasyRich-textured livingroomuniform & stablecircle (slow+normal)defaultiPhoneXR_OnePlus5T_part1
iPhoneXR_OnePlus5T_part2
9_controlEasyRich-textured livingroomuniform & stablecircle (slow+normal)defaultiPhoneXR_OnePlus5T_MYNT_part1
iPhoneXR_OnePlus5T_MYNT_part2
10EasyRich-textured livingroomuniform & stablehybriddefaultiPhone XR, OnePlus 5T
10_controlEasyRich-textured livingroomuniform & stablehybriddefaultiPhone XR, OnePlus 5T, MYNT
11EasyRich-textured livingroomuniform & stableuser scan simulationdefaultiPhoneXR_OnePlus5T_part1
iPhoneXR_OnePlus5T_part2
11_controlEasyRich-textured livingroomuniform & stableuser scan simulationdefaultiPhoneXR_OnePlus5T_MYNT_part1
iPhoneXR_OnePlus5T_MYNT_part2
12Normal[2]Rich-textured desktopuniform & stablecircle (slow+normal+fast)defaultiPhone XR, OnePlus 5T
13NormalRich-textured desktopuniform & stablehybriddefaultiPhone XR, OnePlus 5T
14NormalRich-textured desktopuniform & stablefreedefaultiPhone XR, OnePlus 5T
15NormalRich-textured desktopuniform & stablegame simulation (YuMe)defaultiPhone XR, OnePlus 5T
16NormalMedium-textured desktopuniform & stablecircle (slow+normal+fast)defaultiPhone XR, OnePlus 5T
17NormalMedium-textured desktopuniform & stablehybriddefaultiPhone XR, OnePlus 5T
18NormalMedium-textured desktopuniform & stablefreedefaultiPhone XR, OnePlus 5T
19NormalMedium-textured desktopuniform & stablegame simulation (YuMe)defaultiPhone XR, OnePlus 5T
20NormalMedium-textured desktopdynamiccircle (slow+normal+fast)defaultiPhone XR, OnePlus 5T
21NormalMedium-textured desktopdynamichybriddefaultiPhone 7, Huawei P20
22NormalMedium-textured desktopdynamicfreedefaultiPhone 7, Huawei P20
23NormalMedium-textured desktop with reflectionuniform & stablecircle (slow+normal+fast)defaultiPhone 7, Huawei P20
24NormalMedium-textured desktop with reflectionuniform & stablehybriddefaultiPhone 7, Huawei P20
25NormalMedium-textured desktop with reflectionuniform & stablefreedefaultiPhone 7, Huawei P20
26NormalMedian-textured livingroomuniform & stabletranslation (slow+normal+fast)defaultiPhone 7, Huawei P20
27NormalMedian-textured livingroomuniform & stabletranslation (slow+normal+fast)30fpsiPhone 7, Huawei P20
28NormalMedian-textured livingroomuniform & stabletranslation (slow+normal+fast)autofocusiPhone 7, Huawei P20
29NormalMedian-textured livingroomuniform & stablecircle (slow)defaultiPhone XR, OnePlus 5T
30NormalMedian-textured livingroomuniform & stablecircle (normal)defaultiPhone XR, OnePlus 5T
31NormalMedian-textured livingroomuniform & stablecircle (fast)defaultiPhone XR, OnePlus 5T
32NormalMedian-textured livingroomuniform & stablecircle (normal)30fpsiPhone XR, OnePlus 5T
33NormalMedian-textured livingroomuniform & stablecircle (normal)autofocusiPhone XR, OnePlus 5T
34NormalMedian-textured livingroomuniform & stablehybriddefaultiPhone XR, OnePlus 5T
35NormalMedian-textured livingroomuniform & stablehybrid30fpsiPhone XR, OnePlus 5T
36NormalMedian-textured livingroomuniform & stablehybridautofocusiPhone XR, OnePlus 5T
37NormalMedian-textured livingroomuniform & stablefreedefaultiPhone XR, OnePlus 5T
38NormalMedian-textured livingroomuniform & stableuser scan simulationdefaultiPhone XR, OnePlus 5T
39NormalMedian-textured livingroomuniform & stablegame simulation (Shot)defaultiPhone 7, Huawei P20
40Normaltextureless floor and furnituresuniform & stabletranslation (slow+normal+fast)defaultiPhone 7, Huawei P20
41Normaltextureless floor and furnituresuniform & stablecircle (slow)defaultiPhone 7, Huawei P20
42Normaltextureless floor and furnituresuniform & stablecircle (normal)defaultiPhone 7, Huawei P20
43Normaltextureless floor and furnituresuniform & stablecircle (fast)defaultiPhone 7, Huawei P20
44Normaltextureless floor and furnituresuniform & stablehybriddefaultiPhone 7, Huawei P20
45Normaltextureless floor and furnituresuniform & stablefreedefaultiPhone 7, Huawei P20
46Hard[3]Rich-textured desktopnon-uniform & stablehybriddefaultiPhone 7, Huawei P20
48HardMedium-textured desktopnon-uniform & stablehybriddefaultiPhone 7, Huawei P20
50HardMedium-textured desktop with reflectionnon-uniform & stablehybriddefaultiPhone 7, Huawei P20
52Hardtextureless floor and furnituresnon-uniform & stablehybriddefaultiPhone 7, Huawei P20

notes:

  1. "Easy" means no repetitive texture, no reflection, no opened windows. Rich-textured carpets, sofas and walls.
  2. "Normal" means a common livingroom settings. Median-textured carpets, sofas and walls. Some opened windows and a mirror.
  3. "Hard" means Opened windows, mirrors, textureless floor, textureless furnitures, textureless walls, lamps.
  4. "_control" means we have an additinal MYNT camera attached to record data besides of the two mobile phones.
  5. default device setting means as below.

default_config

The full list of the case design can be downloaded with CaseTableEva.xlsx and the better resolution preview video can be found at NEAR-dataset-preview-video.

For full understand our design, please refer to our paper[1].

Tentative Grouping

We tested some open source VIOs including ICE-BA, VINS-Mono, OKVIS, ROVIO, MSCKF-Mono on our NEAR dataset tentatively. Note that the results are not for benchmarking.

Based on the results, we devide our dataset into several groups as below for reference.

Dataset structure

The sequences are organized following the structure of the EuRoC, so the usage is fair easy.

./Case ID
│
└───ip7 A folder with phone name.
 └───P_lp2lv.txt The extrinsic between phone camera and rigid body.
 └───rts_compensate_extrinsic_and_time.txt The ground truth trajectory (TUM format) of camera frame.
 └───timeoffset.txt Timestamp offset between phone camera and RTS system.
 │
 └───mav0
 │
 └───cam0
 │ └───data The folder contains image sequence.
 │ └───cam.yaml Camera settings when record this sequence.
 │ └───data.csv Timestamps of camera frames.
 │ └───sensor.csv Camera intrinsics and camera-IMU extrinsic.
 └───imu0
 └───data.csv Timestamps and IMU measurements.
 └───sensor.yaml IMU intrinsics and IMU-rig extrinsic which is identity.