open_AR_Sandbox

Welcome to the Open_AR_Sandbox repository. If you do not know what this is all about, have a look at this video:

:warning: Warning! It is unfortunate that we have to state this here, but: downloading the software and presenting it somewhere as your own work is serious scientific fraud! And if you develop content further, then please push these developments back to this repostory - in the very interest of scientific development (and also a requirement of the license). For more details, please consult the information below and the license.

The CGRE Sandbox in action

What is AR-Sandbox?

Python 3 License: LGPL v3

Table of Contents

Introduction

Augmented Reality Sandboxes are a great tool for science outreach and teaching due to their intuitive and interaction-enhancing operation. Recently AR Sandboxes are becoming increasingly popular as interactive exhibition pieces, teaching aids and toys.

AR Sandboxes consist of a box of sand that can be freely sculpted by hand. The topography of the sand is constantly scanned with a depth camera and a computed image is projected back onto the sand surface, augmenting the sandbox with digital information.

However, most of these common AR Sandboxes are limited to the visualization of topography with contour lines and colors, as well as water simulations on the digital terrain surface. The potential for AR Sandboxes for geoscience education , and especially for teaching struc- tural geology, remains largely untapped.

For this reason, we have developed open-AR-Sandbox, an augmented reality sandbox designed specifically for the use in geoscience education. In addition to the visualization of topography it can display geologic subsurface information such as the outcropping lithology, creating a dynamic and interactive geological map. The relations of subsurface structures, topography and outcrop can be explored in a playful and comprehensible way.

Features

Some of the modules already implemented include:

Check the video below for some of the features in action: Open AR Sandbox Features

The open_AR_Sandbox as well as GemPy are under continuous development and including more modules for major outreach. Some of the features we are currently working on include:

License, use and attribution

Feel free to download and use the Open_AR_Sandbox software! We do not provide any warranty and any guarantee for the use. We also do not provide professional support, but we aim to answer questions posted as Issues on the github page as quickly as possible.

Open_AR_Sandbox is published under an GNU Lesser General Public License v3.0, which means that you are free to use it, if you do not do any modifications, in a wide variety of ways (even commercially). However, if you plan to modify and redistribute the code, you also have to make it available under the same license!

Also, if you do any modifications, especially for scientific and educational use, then please provide them back to the main project in the form of a pull request, as common practice in the open-source community. If you have questions on the procedure, feel free to contact us about it.

These are the main conditions for using this library:

For more details on the licsense, please see provided license file.

If you use Open-AR-Sandbox in a scientific abstract or publication, please include appropriate recognition of the original work. For the time being, please cite:

Virgo, S., De La Varga Hormazabal, M., & Wellmann, F. (2019). Open-AR-Sandbox: An open-source Augmented Reality platform for geoscience. In Geophysical Research Abstracts (Vol. 21).

Requirements

You will need:

Mount the kinect and projector facing down vertically in the center above of the box. The optimal distance will depend on the size of your sandbox and the optics of the projector, from our experience a distance of 150 cm is well suited for a 80 cm x 100 cm box. More details on how to set up the kinect and projector can be found in the 1_calib_projector.ipynb and 2_calib_sensor.ipynb notebooks, and if you want to use the ArUco markers 3_calib_arucos.ipynb.

Installation

First of all you will need a healthy Python 3 environment. We recommend using Anaconda. In addition to some standard Python packages, you will need a specific setup dependent on the Kinect version you are using. In the following we provide detailed installation instructions.
Now download or clone this repository open_AR_Sandbox from github.

  1. First clone the repository: 
    git clone https://github.com/cgre-aachen/open_AR_Sandbox.git
  2. Create a new anaconda environment 
    conda create -n sandbox-env python=3.7
  3. Now when you want to use the sandbox and the packages we are about to installl you will have to activate the environment before starting anything 
    conda activate sandbox-env

    Standard packages

To install all the standard packages please use the requirements.txt file:

pip install -r requirements.txt

You can also have a local installation of the sandbox by using the File “setup.py” by doing:

pip install -e .

Kinect Installation

For Windows

Kinect v1 - Future

There is still no support for kinect V1…

Kinect V2 - PyKinect2

(Tested on Windows 10). First, install the current Kinect SDK including drivers. You can use the software bundle to test the connection to your kinect, before you continue.

To make Python and the Kinect SDK communicate, install the related PyKinect2 wrappers which can be easily installed via:

pip install pykinect2

Unfortunately, the configuration of PyKinect2 needs to be adjusted to work on a 64 bit System. Therefore, edit the Lib/site-packages/pykinect2/PyKinectV2.py file, go to line 2216 and comment it:

# assert sizeof(tagSTATSTG) == 72, sizeof(tagSTATSTG)

Add the following lines below:

import numpy.distutils.system_info as sysinfo
required_size = 64 + sysinfo.platform_bits / 4
assert sizeof(tagSTATSTG) == required_size, sizeof(tagSTATSTG)

For Linux

Kinect v1 - libfreenect

To make open_AR_Sandbox talk to the first generation kinect you will need the Libfreenect Drivers with Python Wrappers. The installation is kind of straight forward for Linux and MacOS but challenging for Microsoft (in fact: if you pull it off, let us know how you did it!) The steps can be summarized as follows (refer to any problems regarding installation in to link) To build libfreenect, you’ll need

Once these are installed we can follow the next commands

sudo apt-get install git cmake build-essential libusb-1.0-0-dev
sudo apt-get install freeglut3-dev libxmu-dev libxi-dev
git clone https://github.com/OpenKinect/libfreenect
cd libfreenect
mkdir build
cd build
cmake -L .. # -L lists all the project options
cmake .. -DBUILD_PYTHON3=ON
make 


cd ../wrappers/python
python setup.py install
# now you can see if the installation worked running an example
python demo_cv2_async.py

Kinect v2 - freenect2

or pylibfreenect2
For this we are going to use a python interface for the library libfreenect2 called freenect2.

<your_path>/anaconda3/envs/<sandbox-env>/lib

sudo cp $HOME/freenect2/lib/libfreenect2{.so,.so.0.2,.so.0.2.0} $HOME/anaconda3/envs/sandbox-env/lib/

LiDAR L515 Installation

Installing in Windows

First, go to the latest release page on GitHub and download and execute the file:


Follow the instructions for the installation and update the Firmware of your sensor.  You should be able to use and see the depth and RGB image.

#### Installing in Linux 

Detailed installation steps can be found in the 
[linux installation guide](https://github.com/IntelRealSense/librealsense/blob/development/doc/distribution_linux.md). 
The steps are as follows:

- Register the server's public key:  
`sudo apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE`
In case the public key still cannot be retrieved, check and specify proxy settings: `export http_proxy="http://<proxy>:<port>"`  
, and rerun the command. See additional methods in the following [link](https://unix.stackexchange.com/questions/361213/unable-to-add-gpg-key-with-apt-key-behind-a-proxy).  

- Add the server to the list of repositories:  
  Ubuntu 16 LTS:  
`sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo xenial main" -u`  
  Ubuntu 18 LTS:  
`sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u`  
  Ubuntu 20 LTS:  
`sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo focal main" -u`

- Install the libraries:  
  `sudo apt-get install librealsense2-dkms`  
  `sudo apt-get install librealsense2-utils`  
  
Reconnect the Intel RealSense depth camera and run: `realsense-viewer` to verify the installation.

#### Running with python

After the sensor is installed on your pltaform, the Python wrapper can be easily installed via:

```pip install pyrealsense2```

If any problems with the installation reference to 
[Intel RealSense Python Installation](https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python#installation)

Download sample data
-------

You have the option to download some publicly shared files from our open_AR_Sandbox shared folder. 
You will need to do this if you want to run the tests, use the landslides simulations and/or get the trained models for 
the the use of the Landscape generation module.

In the terminal type:

python3 sandbox/utils/download_sample_datasets.py


and follow the instruction on the terminal to download the specific files you need. We use 
[Pooch](https://github.com/fatiando/pooch) to help us fetch our data files and store them locally in your computer 
to their respective folders. Running this code a second time will not trigger a download since the file already exists.

External Packages
---------
### GemPy

To use implicit geological models inside the sandbox, go to [GemPy](https://github.com/cgre-aachen/gempy),
clone or download the repository and follow the Gempy Installation instructions. With gempy installed 
you can follow the tutorial [GempyModule](notebooks/tutorials/04_GempyModule/).

pip install gempy

If using windows you will need to install `Theano` separately as instructed in [here](https://www.gempy.org/installation)

conda install mingw libpython m2w64-toolchain conda install theano pip install theano –force-reinstall

### Devito

This package uses the power of [Devito](https://github.com/devitocodes/devito) to run wave proppagation simmulations.
More about this can be found in `notebooks/tutorials/10_SeismicModule/`. Follow the Devito installation instructions. 
* This module so far have only support in linux

pip install –user git+https://github.com/devitocodes/devito.git


### PyGimli
This library is a powerful tool for Geophysical inversion and Modelling. Some examples can be found in 
`notebooks/tutorials/11_Geophysics/`. 
[PyGimli](https://www.pygimli.org/) can be installed following the installation intructions 
[here](https://www.pygimli.org/installation.html)

We recomend creating a new environment where PyGimli is already installed and over that one install the sandbox 
dependencies.

conda create -n sandbox-env -c gimli -c conda-forge pygimli=1.1.0

* And now go back to [installation](/open_AR_Sandbox/README.md#installation) and follow all over again the instruction but skipping 
step 2. 

### PyTorch

To use the LandscapeGeneration module we need to install [PyTorch](https://pytorch.org/). This module use the power 
of [CycleGAN](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) 
to take a topography from the sandbox, translate this as a DEM and then display it again on the sandbox as a Landscape 
image. 
To install the dependencies for this module do:

#For Windows pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

#For Linux pip install torch torchvision

git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix cd pytorch-CycleGAN-and-pix2pix pip install -r requirements.txt ``` Once this is installed, copy the trained model in /notebooks/tutorials/09_LandscapeGeneration/checkpoints folder, and then follow the notebook. Get in contact with us to provide you with the train model for this module.

Project Development

Open-AR-Sandbox is developed at the research unit Computational Geoscience and Reservoir Engineering (CGRE) at RWTH Aachen University, Germany.

CGRE

Project Lead

Prof. Florian Wellmann, PhD

Maintainers (also external to CGRE)

Obtaining a full system

If you are interested in buying a fully operating set-up including appropriate hardware, pre-installed software, and set-up and maintenance, please contact Terranigma Solutions GmbH.

This site is open source. Improve this page.