We show you how to make your own AR sandbox! Great for hands-on interactive fun and education.
BUILD TIME: A weekend
DIFFICULTY RATING: Advanced
The AR Sandbox is a visual display that combines 3D sensors, light projection, and geographic information system software to create a dynamic simulation that merges physical and virtual elements. The Sandbox can be used to simulate and explore various environmental phenomena such as flooding and erosion. While the Sandbox can be a fantastic teaching tool it is also extremely fun to play with and will entertain all ages young or old.
The AR Sandbox allows you to create topographic landscapes in the physical world and see how the virtual world responds with accurate water simulations and height projection. This innovative idea was one of the first successful implementations of augmented reality.
The first Augmented Reality Sandbox prototype was created by a team of researchers from the Keck Center for Active Visualization in Earth Science at the University of California, Davis. The researchers were inspired by the previous efforts of the Sandy Station project from a team from the Czech Republic who first thought of the idea in 2011. The AR Sandbox prototype developed into a public exhibit after receiving funding from the LakeViz3D project. The code and instructions to build an AR Sandbox were then published online for anyone to make their own. https://arsandbox.ucdavis.edu/ https://web.cs.ucdavis.edu/~okreylos/ResDev/SARndbox/
There are now over 800 AR Sandboxes at various museums, universities, and businesses across the globe. While building an AR Sandbox was previously out of reach for most people with the cost easily exceeding $5000, thanks to recent advancements in projection and computing hardware, that price has fallen considerably. We will be looking at how to build our very own AR Sandbox.
How it works
The AR Sandbox consists of four main components. The sandbox itself measuring 1m x 0.75m or any other size in the same 4:3 ratio. The Xbox 360 Kinect sensor mounted in the centre of the sandbox, one metre above the surface, and the digital projector mounted above the middle of the longest edge of the sandbox. These components all come together with a powerful PC running the AR Sandbox software which performs all the simulations and projection mapping.
The Sandbox
The recommended size for a typical AR sandbox is 1m x 0.75m as this means the Kinect will be mounted 1m above the surface and produce a horizontal resolution of 1.56mm, which is enough to resolve features on the order of 5mm, while the vertical resolution is 2.79mm. With a larger sandbox, the resolution will decrease, limiting the usability and practicality.
The sandbox can also be made smaller than 1m x 0.75m to increase the Kinect resolution and reduce the overall footprint and cost of the build. The Xbox 360 Kinect features a 90° field-of-view, which is why it needs to be mounted as high as the sandbox is wide, directly above the centre-point.
Xbox 360 Kinect Sensor
The camera is used due to its low cost and relatively high accuracy and resolution from its IR depth sensor. While the AR Sandbox software also supports the newer Xbox One Kinect it does not provide a better experience and is less stable. The software is also compatible with some early Intel Realsense cameras but these couldn’t produce a usable depth map. The Xbox 360 Kinect had various revisions with some featuring separate USB and power connections while later models had a combined proprietary connector. We will need an earlier model with USB and power or an adapter that splits the proprietary connector out.
Digital Projector
The AR Sandbox team recommends the use of a digital projector with a short-throw length and a native aspect ratio of 4:3. The resolution is less of a concern with XGA (1024x768) proving sufficient as the limiting factor is the Kinect with a resolution of 640x480. The recommended projector is the BenQ MX631ST DLP projector, however, as this projector is long out of production it is difficult to source.
We decided to use the SONY VPL-EX290 as it featured the required 4:3 resolution and 1024x768 resolution while also offering an HDMI video input. The projector features a brightness of 3800 lumens which is sufficient for our needs, and a zoom of 1.6x which determines our mounting height.
You can use any 4:3 1024x768 projector. Just try to get the shortest throw length and highest brightness possible, as that will improve the experience.
Graphics Processing
The AR Sandbox software runs complex water simulations to ensure the water projections are accurate, however, this does require a powerful graphics card to run. The software requires a modern NVIDIA graphics card like the GTX 970 or higher, we used a GTX 980. The remaining system requirements are less strenuous with any recent laptop or desktop computer proving sufficient.
The Build:
The AR Sandbox project includes no instructions for construction other than the size and mounting requirements for the various components.
We decided to create a wooden AR Sandbox, however, it would also be possible to use various other materials and combinations such as metals or plastics. The main considerations for our design were ease of construction, availability of materials, and the ability to disassemble the AR Sandbox for long-term storage.
Sandbox Base
We considered various timber to use as our base platform including MDF, Plywood, and Melamine. These products were all available from our local hardware store and each offered various advantages. The MDF and Plywood sheets were the cheapest but both would have required sealing and painting to create an acceptable surface for projection. We decided to go with Melamine sheeting for the main sandbox as it had a perfect projection surface that wouldn’t require any post-processing.
| Electronic Parts Required: |
|---|
| 1 x Xbox 360 Kinect Sensor |
| 1 x PC with high performance graphics card (We used a GTX 980) |
| 1 x Digital projector (We used a SONY VPL-EX290) |
| Build Materials Required: | PURPOSE |
|---|---|
| 1 x Sheet of 2400x1200x16mm Melamine | Sandbox |
| 4 x 400mm lengths of 70x35mm Pine | Table Legs |
| 2 x 2m lengths of 70x35mm Pine | Mounting Post |
| 1 x 2m length of 40x20mm Lightweight Pine | Kinect Frame |
| Packets of short & long countersunk Wood Screws | Entire Build |
| 6 x Metal Right-angle Brackets | Kinect / Projector Supports |
| 1 x Packet of Long Cable Ties | Mounting/Tidy Wiring |
| 5 x 20kg Bags of Play Sand | Sandbox |
| Wood glue | Sandbox |
| Wood filler | Sealing Sandbox |
| 1 x CD or DVD | Calibration Tool |
| 1 x Chopstick or similar | Calibration Tool |
| 1 x Marker Pen | Calibration Tool |
| Tools & Hardware Required: |
|---|
| Electric Drill and Sander |
| Scraper |
| Fine Sandpaper |
* Quantity shown, may be sold in packs.
We also had to select a suitable thickness of Melamine sheeting so that it could support the sand and hardware required without sagging. We will be using 100kg of sand which provides an average depth of 10cm across the entire base. With this in mind, we selected a 16mm sheet that will ensure that the AR Sandbox remains solid when fully assembled.
To create the base of the sandbox, we started with a 2400 x 1200 x 16mm Melamine sheet which we got cut down into two 1000mm x 750mm sheets. One for the sandbox base, and the other as a lid to protect the sand when not in use.
The remaining material was cut into 4 lengths which we used as sides for our base to create our sand area. The two sets of sides measured 1040 x 200mm and 750 x 200mm. We attached these edge pieces to the side of our base plate with wood glue and nails.
The AR Sandbox requires a sealed base area so that sand doesn’t escape the enclosure, to seal the sandbox we applied wood filler along the edges of our base and let it harden. Once the wood filler had set we removed the excess with a scraper and sanded down the remains for a smooth surface.
Timber Legs
This base will serve as the core component of our AR Sandbox, so we wanted to raise it above the ground to make it easier to interact with. We couldn’t significantly increase the height as we needed room for the Kinect and projector mounting hardware and we were constrained by the height of our building. We decided to create some base legs from a length of 70 x 35mm pine. We cut four 400mm lengths and attached them to our base using two countersunk screws each so that they could be removed or changed later.
Mounting Post
The next step was to create the mounting hardware for our Kinect sensor and projector.
First, we first needed to find a strong supporting post that would run along the edge of the sandbox and serve as a point to attach our mounting hardware.
We decided to use another length of 70mm x 35mm pine. To determine the length required, we first calculated where we would mount the projector and worked backwards from that.
The SONY VPL-EX290 projector we are using features an adjustable zoom of 1.6x, which gives us a throw ratio of 1.40 - 2.27. This means that the projector will need to be vertically located 1.4x - 2.27x times the width of our image. We will be projecting a 1m x 0.75m image so we will need to mount the projector 1.4m - 2.27m above our projection surface. We decided to mount our projector 1.6m above the sandbox as this allowed for minor adjustments.
We added our projector height to our leg height to get a total length of 2m for our main mounting post. We cut our length of pine and attached it along the centre of the longest edge and secured it to the table with several countersunk screws so it could later be removed for storage. We were initially only going to have one post, however, we noticed some instability during our initial tests, so we decided to attach a second length of pine diagonally to the post and AR Sandbox to help secure the assembly.
Kinect Sensor Mount
With all the structural components of the AR Sandbox completed, we were ready to create the mounting hardware for our Kinect sensor and projector. We started with the Kinect by creating a frame out of 40mm x 20mm lightweight pine so that the Kinect would be directly in the centre of the sandbox, 1m above the surface. We had to ensure that the pine did not block the projection by creating a square mount.
Secure the Kinect using a few long cable ties.
Projector Mount
The final piece of hardware we needed to create was the projector mount, we started with a spare piece of the white Melamine sheet and created a 70mm x 35mm rectangular hole so that we could slide it over the mounting post.
We secured two brackets to the underside of the sheet so that we could screw it into the post. With a secure base to hold the projector, we created a cutout for the projector lens and controls, and finally created a cradle for the projector with more sheets of Melamine. This design allows us to quickly access and troubleshoot with the projector and can easily accommodate various projector sizes.
Computer Location
We decided against creating any mounting hardware for our desktop computer and instead opted to simply place it by the side of the AR Sandbox to allow for easy access and wiring.
100kg of Sand
With the initial construction complete, we loaded up the Sandbox with five 20kg bags of play sand and wired up our desktop computer, Kinect, and projector.
The Kinect was connected to our USB/power adapter, which connected to our computer’s USB port. The projector was attached to HDMI from the GTX 980.
Operating System Setup
The AR Sandbox software runs on Linux Mint so we will need to install the operating system on our computer.
Important note: Backup any data on the computer beforehand, as all data will be overwritten during installation.
We will be installing the recommended 64-bit Linux Mint 19.3 with the MATE desktop environment. You can download the ISO file from https://linuxmint.com/edition.php?id=276 to your computer and flash it to a USB.
Use balenaEtcher to flash the Linux Mint installation ISO to a spare USB. The latest version of Etcher can be downloaded from https://www.balena.io/etcher/
With Etcher installed, select the linuxmint-19.3-mate-64bit.iso file and your USB flash drive, then select flash. This will flash the Linux Mint installer to your USB, ready to use with your AR Sandbox computer.
Next, insert the flashed Linux Mint installer USB to the computer you want to power the AR Sandbox and boot into the installation environment.
Enter the computer BIOS by tapping the BIOS key when the computer is booting (the most common BIOS keys are F8 or DEL). You can then set your installer USB as the priority boot device so that the computer will boot into the Linux Mint installation environment.
Note: The methods of changing boot priority vary between computers but most will have a boot menu with the option.
Once you have adjusted the boot priority, save the changes and reboot.
Once your computer has booted into the Linux Mint installation environment running on our USB stick, you can begin the installation process.
Select the Install Linux Mint icon, which will start the installer program. Set your language, and keyboard layout in the next screens, before disabling the third-party software option.
When the installer asks for the installation type, select “Erase disk and install Linux Mint”, which will then ask you to select a drive and confirm. The installer will begin to install Linux Mint and will allow you to select a timezone and set up the user profile. We can select “Log in automatically” and choose a name, username, and password for our local user profile.
Once the installation is complete, remove the USB and restart the computer.
The computer should now boot into the Linux Mint environment from its internal drive.
Connect an ethernet cable to install the necessary drivers and scripts.
Note: Once the AR Sandbox computer is configured, it does not require an internet connection to run.
The computer will show a welcome screen on the first boot, which will allow you to install the necessary NVIDIA drivers for your graphics card. You can launch the driver manager which will scan for devices and available drivers. Install the latest recommended nvidia-driver which will allow the AR Sandbox software to function correctly. You can now reboot the computer. It should then be ready for you to begin installing the AR Sandbox software.
AR Sandbox Installation
Note: The AR Sandbox project has an official installation guide available that can offer you more advanced configuration and troubleshooting options if you get stuck: https://web.cs.ucdavis.edu/~okreylos/ResDev/SARndbox/LinkSoftwareInstallation.html
You can now install the AR Sandbox software onto your machine. The software uses the Vrui VR Development Toolkit to render and configure the AR Sandbox so that it will be installed first. Open a new terminal window and enter the following commands to download and run the Vuri installation script.
When the script has completed the installation process, a new window will open with a spinning globe which will allow us to test the Vuri controls we will use later for calibration. You can access the main menu in a Vuri application by holding the right mouse button and navigating to the option you want and releasing the button. Pan by holding the left mouse button and holding the ‘Z’ key at the same time.
Finally, you can set hotkeys for different tools. Hold the keyboard key you want to bind the tool to, navigate to the tool you want with the mouse, and release the keyboard key.
Note: These tools and controls will be used during the setup of the AR Sandbox so be sure to familiarise yourself with them.
Exit the globe window and return to our terminal window to continue the AR Sandbox software setup.
The Vuri toolkit will attempt to automatically recognise your screen size, however, projectors are not supported. Instead, you need to manually set this by editing the configuration file.
Run the following command and change the value of autoScreenSize from true to false.
Next, install the Kinect 3D Video Package software by running the following commands and entering your user password if prompted.
You should now have all the necessary prerequisites for your AR Sandbox. Next is to install the sandbox scripts.
Run the following commands in a terminal window.
Note: The final command should print CalibrateProjector, SARndbox, and SARndboxClient which are the three AR Sandbox scripts we need.
AR Sandbox Calibration
If you haven’t already done so, connect and place your projector and Kinect in their final positions so you can begin the calibration process.
The first step is to load the Kinect configuration file by running the following command in the terminal.
Align your camera so that it captures the entire interior of your AR sandbox. Do this by running the following command and moving the Kinect sensor until the left depth image shows on the entire sandbox.
You can now fix the Kinect in this final position for the next calibration steps.
Note: You can leave this window open for the next step.
You now need to measure your AR Sandbox’s base plane equation using the Kinect sensor, so that the bounds of your sandbox are correctly defined in the software.
Open the BoxLayout.txt configuration file by running the following commands.
Use the depth image stream we opened previously, along with the “Extract Planes” and “Average Frames” tools to find your base plane equation.
Note: The AR Sandbox team has also created a video walkthrough if you get stuck, which can be found here https://www.youtube.com/watch?v=EW2PtRsQQr0.
First, select “Average Frames” tool by holding the first mouse button, navigating to it and releasing it, which will cause the Kinect to average its depth readings and create a static image. You can then bind the “Extract Planes” tool to another key and capture the four corner points of the sandbox by holding the tool key and dragging the mouse to the corners of the sandbox creating a rectangle. You can then release the tool key to capture the depth plane. Once you have captured the plane, you can return to the terminal where the depth space and camera space should be printed.
Copy the camera-space value into the BoxLayout.txt file we opened, slightly modifying it from the original form to be in the following format.
You will now need to bind the “Measure 3D Positions'' to another key and capture the four corners of your sandbox. You will need to do this in the order lower left, lower right, upper left, and upper right by moving your mouse to position and pressing the tool key. You can then copy the four corner positions into the BoxLayout.txt file after the camera space value to get a final file with the following layout but different values.
(0.0123403, -0.02516, 0.999607), -105
(-52.6878, -33.5753, -99.6237)
(48.0742, -33.7274, -100.112)
(-53.1308, 39.9727, -102.836)
(49.6205, 40.9425, -102.791)
You can now align our projector by running the Vuri calibration grid and physically adjusting the projector so that it fills our AR Sandbox. Be sure to disable any digital image distortion features on the projector as these will cause issues. You can have some overflow projection, as this will be fixed during the final calibration step.
Calibrate the Kinect and projector with respect to each other so that the final image will correctly line up with the depth map. To do this, you will need to create a calibration stick.
Make the calibration stick using an old CD or DVD and a chopstick or similar to hold it. Cover the CD or DVD in paper and mark the centre of the disk using a marker pen.
Run the following command to boot the calibration program.
You need to first full-screen the CalibrateProjector application using F11 and then bind the "Capture" tool to two unused keys. For example, we set these to ‘1’ and ‘2’ by holding one and navigating to Capture in the menu before releasing the key, and then pressing the ‘2’ key to set it as our secondary tool key.
You are now ready to begin the calibration process. First, press ‘2’ to capture the background and then position the crosshair of your calibration tool with the two projected lines just above the surface of your sandbox. Once a yellow projection appears, you can press ‘1’ to capture the point and move onto the next. You will also need to capture some points lower down by digging a hole in the sand and recapturing the background with the ‘2’ key. The video guide has a visual demonstration of this calibration procedure.
Using the AR Sandbox
Once the calibration procedure is complete, you will be ready to launch the AR Sandbox application by entering the following command. You will, once again, want to full-screen the application once it has launched using F11 to ensure the projection lines up.
You can try using the AR Sandbox by moving the sand and watching as the projection updates in real-time. You can create mountains and ravines and watch as topographic lines are projected, along with corresponding colour height values.
You can create rain by holding your hand above the area we want the rain to appear. The water will naturally flow across your landscape, which allows you to create dams and rivers.
Where to from here?
You should now have a fully functioning AR Sandbox without the need for any further modification, however, the current sandbox setup requires a keyboard and mouse to manually launch the application each time the computer is restarted. This process can be automated by creating a desktop shortcut for the application and setting it to launch on boot. Instructions to configure this can be found on the AR Sandbox page.
There are also various other options to add more features to your AR Sandbox, including displaying a 3D version of the depth map on a second monitor which can be navigated around and explored.
The default installation is targeted towards education with accurate water simulation, but for a bit more fun, the configuration files can be edited to add lava.