Introduction to geo-AR + measuring position and orientation in the browser
We started a series of blog posts about the challenges of building a Real World Browser, at the technical level of course, but also in terms of human-computer interface, data, etc.
We will talk about mobile sensors, maps, machine learning and data, mobile and web development, orientation, positioning systems, buildings, cities, design and information. And much more :)
To begin, we will dive into the technical challenges to obtain an accurate attitude estimation in the context of a smartphone web browser. Firstly, we will introduce some reminders on how Geographical Augmented Reality (Geo AR) is built.
At Wemap, our vision is to propose a light (no download) AR experience which works worldwide and is compatible with a large number of smartphones. This is the reason why part of our research efforts on Geo AR was spent on web specific issues.
Geo AR Principles
Augmented Reality for exploring the Real World means that you do not know beforehand what scene the user is looking at beforehand, and that you have to be based on “ground truth” (the information you are going to display / engage with is rooted in the Real World, with a specific location). These are the basic tenets of Geo AR: the digital information or interaction that you are offering to the user relates to a location.
To make the Augmented Reality for exploring the Real World possible, the well-known approach is to combine data from: (1) GNSS (GPS, GLONASS, Galileo…) and (2) the Inertial Measurement Unit [IMU] (accelerometer, gyroscope, magnetometer). The GNSS allows to know the position of the smartphone while the IMU allows to estimate the smartphone attitude (aiming north, east, up, down…).
In this blogpost we will focus on this second part: attitude estimation of the smartphone.
Part 1.1 — What is the attitude of a smartphone and how it is estimated?
Smartphone attitude
The smartphone attitude is determined when the axis orientation of the Smartphone-Frame (SFx, SFy, SFz) is specified with respect to the ENU Frame EF (EFx, EFy, EFz), see Figure below.
The Smartphone-Frame SF (dashed line) and ENU-Frame EF (solid line).
This rotation can be expressed with four different mathematical representations: a rotation matrix, quaternion, Euler angles or axis-angle. To learn more, I invite you to read documents on euclideanspace.com.
Estimation
There are several ways to estimate attitude of an object (image processing, IMU, inclinometer…). In the case of a smartphone, attitude estimation that we will describe here is based on the IMU. For this purpose, we need to use at least two sensors: a 3-axis accelerometer and a 3-axis magnetometer. The axes are the same as those in blue on the previous figure.
- Accelerometer: It measures accelerations along the 3 axis of the smartphone: gravity (~9.8 m.s-2) and external accelerations (translations). The unit is G or m.s-2.
- Magnetometer: It measures the magnetic field around it: the Earth magnetic field and all the sources emitting a magnetic field (e.g the structure of a building, a belt…). The unit is Tesla or Gauss.
A third sensor is fused to smoothed data from the two previous sensors:
- Gyroscope: It measures the rotation speed around the 3 axes of the smartphone. The unit is radian.s-1.
These three sensors are fused using an attitude estimation filter. This filter can be designed using: a Kalman filter, an observer, a gradient descent … Their common point is that they all implement a solution to the Wahba’s problem (1965).
Another common point to all these algorithms is assumptions they make during estimation:
- The smartphone must be static (no translation).
- There are no magnetic disturbances. That is, the magnetometer must only measure the Earth’s magnetic field and not other magnetic fields emitted by ferromagnetic objects (e.g: the walls / floors of a building).
The first hypothesis (even if it is not exactly respected) does not really pose a problem in the case of Augmented Reality. However, this is not the case for the second, it is the source of most estimation errors in AR. In Europe, the Earth’s magnetic field is of the order of 45 microTesla. An uncalibrated smartphone magnetometer can measure a magnetic field of about 300 microTesla, just because of the battery and speaker that are only a few inches away from it. Variations due to real estate structures can disturb the measures up to +/- 100 microTesla. The consequence is that magnetic disturbances of a few tens of microTeslas can change the orientation of the smartphone a few (tens of) degrees.
No filter is really able to ignore these two assumptions, however some algorithms can reduce the impact of magnetic disturbances (https://hal.inria.fr/hal-01650142/document).
Part 1.2 Attitude estimation in web browsers
Fortunately, whenever a developer wants to use the orientation data of a smartphone he is not obliged to re-implement a Kalman filter. Android, iOS and Microsoft smartphones allow us to easily access the orientation estimate in our native applications. On the web side, access to this data is more tedious, browsers do not give access to all the necessary information. From one browser to another, or even from one phone to another, the data we have access to is significantly different. In addition, it is impossible for us to know if the filters used by manufacturers take into account the reduction of the impact of magnetic disturbances. Here are some of the constraints related to the web:
Access constrains
- Current web APIs do not provide access to quaternion or rotation matrix, only to Euler angles. Theoretically this is not a constraint, but related to inconsistencies (next section), it makes the use of these data very cumbersome.
- There is no access to the magnetometer on almost all browsers. It is then impossible to directly use the best orientation estimation filters in web.
Limiting inconsistencies
Smartphones operating systems have varying standards and inconsistent treatments when it comes to handling and processing orientation:
- On an iPhone, when the inclination of the phone exceeds 62 °, the attitude returned is an orientation with respect to the EUS (East-Up-South) frame. When the phone goes below 30 ° it reuses the classic ENU (East-North-Up) frame. When the web page is opened with an inclination between 30 ° and 62 ° it is impossible to know which frame is used.
- On some phones with Android Chrome, the relative orientation (from deviceorientation) is randomly set to 0 ° or -90 °…
Heterogeneity inconsistencies
These inconsistencies are not limiting per se, they just have to be taken into account when implementing the algorithms.
Safari iOS and Chrome Android do not use the same rotations to express Euler angles (ZXY vs -ZXY):
- For absolute guidance, Safari iOS uses the deviceorientation event while Chrome Android uses the deviceorientationabsolute event.
- On Safari iOS the vector of acceleration is the opposite of that of Chrome Android.
Recommended approach
To avoid the edge effects described above, we decided to set up our own algorithm. This allows us to have full control over the AR experience we offer. Our algorithm is notably based on an extended Kalman filter (EKF) which has been specially designed for the Geo RA. It reuses directly the data from the accelerometer, the gyroscope and the magnetometer. It is an opportunistic algorithm, that is, it adapts to the sensors to which it has access. For example, if the magnetometer is accessible we will use it. If it is not accessible (as it is mostly the case), we use an EKF without magnetometer that merges with data from “deviceorientationabsolute”.
Most of our algorithms have been tested and compared to dozens of others in a controlled environment (http://tyrex.inria.fr/mobile/benchmarks-attitude/).
Conclusion
The web is a complex environment to work with sensors from the inertial unit, however it is still possible to provide the smartphone user with a light AR experience. The filters described above are not resource intensive compared to a pure image processing approach, which gives a significant advantage for their Web implementation. Contact us if you want more information.
For Wemap AR, we have implemented hybrid solutions to merge data from the inertial unit, GNSS, computer vision and mapping. This work will be the subject of a future blogpost. Stay tuned 😊
Wemap is a recognized digital platform for maps publishing on the French market working with brands like Air France, Hachette Publishing or the French government. Its research on Augmented Reality allows Wemap to provide an unparalleled world-exploration experience to create a Real-World Browser.
