BlockBuildAR
BlockBuildAR is an augmented reality sandbox game for your phone created by Andreas Baldinger. You can mine blocks to build whatever your heart desires!
By pressing the mode switch button you can toggle between the following three modes:
-
Build Mode
- In this mode you can tap the screen to place blocks in your real life environment. A green indicator cube shows you where your block will be placed. Simply rotate your device to choose your next spot
-
Delete Mode
- In this mode you can tap the screen to remove placed blocks. A red indicator cube shows which cube will be destroyed
-
Mine Mode
- In this mode you can tap the screen to mine (i.e. scan the material of) your environment. Tap while focusing on a tree to receive wood, tap while focusing on a building or wall to receive stone
Please be aware that the scanning process does not work equally well for all materials and can fail sometimes!
-
Inside
- Play in your living room, office, bathroom, etc.! Mining materials, however, will be easier outside.
-
Outside
- Play at the parc, a stadium, school, in your garden, etc.! Be wary of your environment (e.g. do not play on a busy street)!
-
Hardware
- A phone (both IOS and Android should work but this recipe focuses on the latter)
- A laptop with enough space for Unity, Unity packages and your project
- A decent internet connection
-
Software
- An IDE or editor for editing C# scripts
- An email address to create both a Unity and a Niantic account
- For Unity Hub, Unity Editor, Niantic SDK, etc. you can find the versions in the recipe under Project Setup
-
Skills
- Basic understanding of C#
- Beginner experience with Unity
-
Time
- By copying the code from this tutorial I would estimate that you can recreate this project in a few hours
-
Permissions
- Your phone will ask for camera permissions for running the AR application
This recipe will first show you how to setup the project and create the assets, then how to create the main script of the application and setup the scene. At the end it will teach you how to build the APK so that you can run the game on your mobile device.
-
Project Setup
If you run into any problems during the setup, you can find a more indepth tutorial here
- Create a Niantic Spatial Developer account here
- Once signed in, create a new project to get an API key
- Download Unity Hub here
- Install the editor version 6000.0.58f2
- Create a new Unity project using the 3D (Built-In Render Pipeline) template
- Inside the project open the Package Manager and add a package using the following git URL: https://github.com/niantic-lightship/ardk-upm.git. At the time of writing version ARDK 3.17 was current. Please make sure that the link still points to that version or find an alternative one that does yourself to guarantee compatibility with the downloaded Unity editor version
- Additionally add a package using the following git URL: https://github.com/niantic-lightship/sharedar-upm.git. At the time of writing version ARDK 3.17 was current. Please make sure that the link still points to that version or find an alternative one to guarantee compatibility with the downloaded Unity editor version
- Open Settings under Ligthship in the top menu bar and paste your API key into the API key field
- Open XR Plug-in Management under Lightship in the top menu bar, select the Android tab and check the box labeled Niantic Lightship SDK + Google ARCore
- Open Build Profiles under File in the top menu bar, select Android and the click Switch Platform.
- In the same menu click Player Settings and change the following settings in Other Settings: a) Uncheck Auto Graphics API and remove Vulkan from the list b) Set Minimum API Level to Android 7.0 'Nougat' API Level 24) or higher c) Set Scripting Backend to IL2CPP and enable both ARMv7 and ARM64
-
Asset Creation
- Download a dirt texture pack, a wood texture pack, a grass texture pack and a stone texture pack
- In your editor in your Assets folder create a Textures folder
- For each pack you downloaded create a folder in your Textures folder
- Extract the downloaded files and drag and drop them into their corresponding folders
- In each folder find the mostly blue colored and normal named textures, click them to open the Inspector window and change Texture Type from Default to Normal Map
- In your editor in your Assets folder create a Materials folder
- In your Materials folder right click, and select Create and Material. Repeat this until you have 1 material per Textures folder
- Click one of the materials (there should be 4 of them now) to open its Inspector window. Navigate to one of the Textures folders and drag and drop the actual texture file (i.e. the normally colored one) into the Albedo field. Perform the same action for the Normal Map using the blue normal map you created earlier. You can also add the rest of the texture files (e.g. Metallic, Height Map and Occlusion) but this is optional. Repeat this for the rest of the Textures folders
- Copy all of your materials and add _transparent to the name of the copies
- For each transparent material (there should be 4 of them now, 8 materials overall) click it to reveal the Inspector window, set Rendering Mode to Transparent, select the color field next to the Albedo field and set the alpha value to 100 instead of 255 and set Smoothness under Metallic to 0
- In your editor in your Assets folder create a Prefabs folder
- In your Hierarchy right click, and select 3D Object and Cube. Drag and drop the cube you just created into your Prefabs folder. Delete the cube in Hierarchy.
- Copy the prefab you just created to create one for each of your (non-transparent) materials.
- Double click a prefab and drag and drop one of the materials from the Materials folder onto the cube in the scene. Set Scale to 0.25 on all axes in the Inspector. Repeat this for all of the other prefabs until you have one textured cube prefab for each material (there should be 4 of them)
-
Code
- In your editor in your Assets folder create a Scripts folder and in the folder right click and select Create and MonoBehaviour Script. Name the script Depth_ScreenToWorldPosition and open it in your preferred IDE / code editor
- Add the following fields which you will later use to assign all the necessary assets to this script at the top of the class:
[Header("AR Components")] [SerializeField] private AROcclusionManager _occlusionManager; [SerializeField] private ARSemanticSegmentationManager _semanticManager; [SerializeField] private Camera _camera; [Header("Block Prefabs")] [SerializeField] private GameObject _dirtPrefab; [SerializeField] private GameObject _woodPrefab; [SerializeField] private GameObject _grassPrefab; [SerializeField] private GameObject _stonePrefab; [Header("UI & Mode")] [SerializeField] private TextMeshProUGUI _modeButtonText; [SerializeField] private TextMeshProUGUI _feedbackText; [Header("Material Specific Previews (Transparent versions)")] [SerializeField] private Material _dirtPreviewMat; [SerializeField] private Material _woodPreviewMat; [SerializeField] private Material _grassPreviewMat; [SerializeField] private Material _stonePreviewMat; [Header("Colors")] [SerializeField] private Color _buildColor = new Color(1, 1, 1, 0.8f); [SerializeField] private Color _deleteColor = new Color(1, 0, 0, 0.5f);
- You will also need an enum to distinguish between the three modes:
public enum ToolMode { Build, Delete, Mine }
- Add the following variables to track all internal states (e.g. the cubes that have been placed in the scene):
private List<GameObject> _placedCubes = new List<GameObject>(); private GameObject _previewCube; private Renderer _previewRenderer; private bool _previewIsValid = false; private GameObject _currentPrefabToSpawn; private Material _currentPreviewMaterial; private XRCpuImage? _depthImage; private Matrix4x4 _displayMatrix; private ScreenOrientation? _latestScreenOrientation; private ToolMode _currentMode = ToolMode.Build;
- Create a first helper function for setting the currently selected block including its preview block (i.e. the transparent cube prefab):
private void SetSelectedBlock(GameObject prefab, Material previewMat) { _currentPrefabToSpawn = prefab; _currentPreviewMaterial = previewMat; if (_previewRenderer != null && previewMat != null && _currentMode == ToolMode.Build) { _previewRenderer.material = previewMat; } }
- Create a second helper function for updating the the text of the mode selection button:
private void UpdateModeText() { if (_modeButtonText != null) _modeButtonText.text = "Mode: " + _currentMode.ToString().ToUpper(); }
- Create a third helper function for creating the preview cube:
private void CreatePreviewCube() { _previewCube = Instantiate(_dirtPrefab); _previewCube.name = "Preview Ghost"; Destroy(_previewCube.GetComponent<Collider>()); _previewRenderer = _previewCube.GetComponent<Renderer>(); if (_currentPreviewMaterial != null) _previewRenderer.material = _currentPreviewMaterial; _previewCube.SetActive(false); }
- You can now use those three methods in Start() to initialize the scene:
void Start() { SetSelectedBlock(_dirtPrefab, _dirtPreviewMat); UpdateModeText(); CreatePreviewCube(); }
- Before you can write the update function you need four helper methods as well. Write the following one which aquires and sets the latest depth image:
private void UpdateImage() { if (!_occlusionManager.subsystem.running) return; if (_occlusionManager.TryAcquireEnvironmentDepthCpuImage(out var image)) { _depthImage?.Dispose(); _depthImage = image; } }
- You will need the current display matrix to transform screen points into world positions. Write the following function for updating it:
private void UpdateDisplayMatrix() { if (_depthImage is { valid: true }) { if (!_latestScreenOrientation.HasValue || _latestScreenOrientation.Value != XRDisplayContext.GetScreenOrientation()) { _latestScreenOrientation = XRDisplayContext.GetScreenOrientation(); _displayMatrix = CameraMath.CalculateDisplayMatrix( _depthImage.Value.width, _depthImage.Value.height, Screen.width, Screen.height, _latestScreenOrientation.Value, invertVertically: true); } } }
- The preview cube will show where the next cube will be placed (i.e. next to placed blocks) in build mode but which cube will be deleted (i.e. on placed blocks) in delete mode. Write the following function for updating the preview cube's position:
private void UpdatePreviewPosition() { if (_currentMode == ToolMode.Mine) { _previewCube.SetActive(false); return; } Vector2 screenCenter = new Vector2(Screen.width / 2f, Screen.height / 2f); _previewIsValid = false; Ray ray = _camera.ScreenPointToRay(screenCenter); RaycastHit hit; // --- DELETE MODE LOGIC --- if (_currentMode == ToolMode.Delete) { if (Physics.Raycast(ray, out hit)) { if (_placedCubes.Contains(hit.collider.gameObject)) { _previewCube.transform.position = hit.collider.transform.position; _previewCube.transform.rotation = Quaternion.identity; Renderer targetRenderer = hit.collider.GetComponent<Renderer>(); if (targetRenderer != null && _previewRenderer != null) { _previewRenderer.material = _currentPreviewMaterial; _previewRenderer.material.mainTexture = targetRenderer.sharedMaterial.mainTexture; _previewRenderer.material.color = _deleteColor; } _previewIsValid = true; } } _previewCube.SetActive(_previewIsValid); return; } // --- BUILD MODE LOGIC --- if (_previewRenderer && _previewRenderer.sharedMaterial != _currentPreviewMaterial) { _previewRenderer.material = _currentPreviewMaterial; } if (_previewRenderer) _previewRenderer.material.color = _buildColor; if (Physics.Raycast(ray, out hit)) { if (hit.collider.gameObject != _previewCube) { Vector3 rawPos = hit.point + (hit.normal * (_currentPrefabToSpawn.transform.localScale.x * 0.5f)); Vector3 snappedPos = SnapToGrid(rawPos); if (!IsPositionOccupied(snappedPos)) { _previewCube.transform.position = snappedPos; _previewIsValid = true; } } } else if (_depthImage.HasValue) { var uv = new Vector2(screenCenter.x / Screen.width, screenCenter.y / Screen.height); var eyeDepth = _depthImage.Value.Sample<float>(uv, _displayMatrix); if (eyeDepth > 0) { var worldPosition = _camera.ScreenToWorldPoint(new Vector3(screenCenter.x, screenCenter.y, eyeDepth)); Vector3 snappedPos = SnapToGrid(worldPosition); if (!IsPositionOccupied(snappedPos)) { _previewCube.transform.position = snappedPos; _previewIsValid = true; } } } _previewCube.SetActive(_previewIsValid); }
- Write the following function to handle touch input for both when working with the editor but also phones:
private void HandleTouch() { Vector2 screenPosition = Vector2.zero; bool inputDetected = false; #if UNITY_EDITOR if (Input.GetMouseButtonDown(0)) { screenPosition = Input.mousePosition; inputDetected = true; } #else if (Input.touchCount > 0) { Touch touch = Input.GetTouch(0); if (touch.phase == TouchPhase.Began) { screenPosition = touch.position; inputDetected = true; } } #endif if (!inputDetected) return; if (IsPointOverUIObject(screenPosition)) return; if (_currentMode == ToolMode.Mine) { PerformSemanticSample(); return; } if (!_previewIsValid) return; if (_currentMode == ToolMode.Build) { SpawnCube(_previewCube.transform.position); } else if (_currentMode == ToolMode.Delete) { for (int i = _placedCubes.Count - 1; i >= 0; i--) { if (Vector3.Distance(_placedCubes[i].transform.position, _previewCube.transform.position) < 0.05f) { GameObject target = _placedCubes[i]; _placedCubes.RemoveAt(i); Destroy(target); _previewCube.SetActive(false); break; } } } }
- You are now ready to write the Update function:
void Update() { UpdateImage(); UpdateDisplayMatrix(); UpdatePreviewPosition(); HandleTouch(); }
- You might notice at this point that you are missing some helper functions. Write the first one used by the mode selection button to toggle between the modes:
public void ToggleMode() { switch (_currentMode) { case ToolMode.Build: _currentMode = ToolMode.Delete; break; case ToolMode.Delete: _currentMode = ToolMode.Mine; break; case ToolMode.Mine: _currentMode = ToolMode.Build; break; } UpdateModeText(); }
- Write the following function for checking whether there is already a cube placed at a certain position:
private bool IsPositionOccupied(Vector3 position) { foreach (var cube in _placedCubes) { if (cube == null) continue; if (Vector3.Distance(cube.transform.position, position) < 0.05f) { return true; } } return false; }
- Write the following function for detecting certain objects in the mine mode to select corresponding cubes to place in the build mode:
private void PerformSemanticSample() { int x = Screen.width / 2; int y = Screen.height / 2; Ray ray = _camera.ScreenPointToRay(new Vector2(x, y)); RaycastHit hit; if (Physics.Raycast(ray, out hit)) { if (_placedCubes.Contains(hit.collider.gameObject)) { Renderer hitRenderer = hit.collider.GetComponent<Renderer>(); if (hitRenderer != null) { if (IsSameMaterial(hitRenderer, _woodPrefab)) { SetSelectedBlock(_woodPrefab, _woodPreviewMat); if (_feedbackText) _feedbackText.text = "mined -> wood"; return; } if (IsSameMaterial(hitRenderer, _stonePrefab)) { SetSelectedBlock(_stonePrefab, _stonePreviewMat); if (_feedbackText) _feedbackText.text = "mined -> stone"; return; } if (IsSameMaterial(hitRenderer, _grassPrefab)) { SetSelectedBlock(_grassPrefab, _grassPreviewMat); if (_feedbackText) _feedbackText.text = "mined -> grass"; return; } if (IsSameMaterial(hitRenderer, _dirtPrefab)) { SetSelectedBlock(_dirtPrefab, _dirtPreviewMat); if (_feedbackText) _feedbackText.text = "mined -> dirt"; return; } } } } if (!_semanticManager) return; var channels = _semanticManager.GetChannelNamesAt(x, y); if (channels.Count == 0) { if (_feedbackText) _feedbackText.text = "unknown -> previous"; return; } foreach (var channel in channels) { if (channel == "building" || channel == "wall") { SetSelectedBlock(_stonePrefab, _stonePreviewMat); if (_feedbackText) _feedbackText.text = "structure -> stone"; return; } if (channel == "foliage") { SetSelectedBlock(_woodPrefab, _woodPreviewMat); if (_feedbackText) _feedbackText.text = "foliage -> wood"; return; } if (channel == "grass" || channel == "natural_ground") { SetSelectedBlock(_grassPrefab, _grassPreviewMat); if (_feedbackText) _feedbackText.text = "terrain -> grass"; return; } if (channel == "ground" || channel == "dirt_experimental" || channel == "artificial_ground" || channel == "manmade_ground") { SetSelectedBlock(_dirtPrefab, _dirtPreviewMat); if (_feedbackText) _feedbackText.text = "ground -> dirt"; return; } } if (_feedbackText) _feedbackText.text = channels[0] + " -> previous"; }
- Write the following function for determening whether a (touch) point is intersecting a UI object without using the unstable default functions:
private bool IsPointOverUIObject(Vector2 pos) { if (EventSystem.current == null) return false; PointerEventData eventDataCurrentPosition = new PointerEventData(EventSystem.current); eventDataCurrentPosition.position = pos; List<RaycastResult> results = new List<RaycastResult>(); EventSystem.current.RaycastAll(eventDataCurrentPosition, results); return results.Count > 0; }
- Write the following function for conveniently producing and placing cubes:
private void SpawnCube(Vector3 position) { GameObject newCube = Instantiate(_currentPrefabToSpawn, position, Quaternion.identity); _placedCubes.Add(newCube); }
- Write the following function for conveniently placing cubes along the virtual 3D cube grid:
private Vector3 SnapToGrid(Vector3 rawPosition) { Vector3 cubeScale = _currentPrefabToSpawn.transform.localScale; return new Vector3( Mathf.Round(rawPosition.x / cubeScale.x) * cubeScale.x, Mathf.Round(rawPosition.y / cubeScale.y) * cubeScale.y, Mathf.Round(rawPosition.z / cubeScale.z) * cubeScale.z); }
- Write the following function for checking whether the same material is used:
private bool IsSameMaterial(Renderer hitRenderer, GameObject prefab) { if (prefab == null) return false; Renderer prefabRenderer = prefab.GetComponent<Renderer>(); if (prefabRenderer == null) return false; return hitRenderer.sharedMaterial == prefabRenderer.sharedMaterial; }
- If your IDE can not find the right namespaces or suggests different ones add these at the top of your file:
using Niantic.Lightship.AR.Semantics; using Niantic.Lightship.AR.Utilities; using System.Collections.Generic; using TMPro; using UnityEngine; using UnityEngine.EventSystems; using UnityEngine.XR.ARFoundation; using UnityEngine.XR.ARSubsystems;
-
Scene Setup
- In your editor in your Assets folder create a Scenes folder and create a new scene
- In your scene in the Hierarchy delete the Main Camera
- In your Hierarchy right click and select XR and AR Session
- In your Hierarchy right click and select XR and XR Origin (Mobile AR)
- Click your XR Origin to open the Inspector and add the Depth_ScreenToWorldPosition script as a component
- Right click your XR Origin game object and select Create Empty. Name the new object Meshing. Click it to open the Inspector window and both an ARMeshManager and a LightshipMeshingExtension component
- Click your Main Camera to open the Inspector window and add an AROcclusionManager for hiding blocks behind real world geometry, a LightshipOcclusionExtension and an ARSemanticSegmentationManager component for classifying real world materials. For the occlusion extension drag and drop one of your cube prefabs in the Prefabs folder into the Principal Occludee field. For Requested Suppression Channels set Element 0 to sky and Element 1 to ground. Last but not least under Occlusion Stabilization drag and drop the Meshing object you created previously into the Mesh Manager field.
- Drag and drop the AROcclusionManager and the ARSemanticSegmentationManager component from the Inspector into their respective fields in the Depth_ScreenToWorldPosition script. Repeat this for the Main Camera which you can find in the Hierarchy by fully opening the XR Origin game object. Repeat this for the prefabs in your Prefabs folder and for the transparent materials (i.e. the preview materials) in the Materials folder. You can ignore UI & Mode for now
- In your Hierarchy right click and select UI and Canvas
- Right click your Canvas and under UI select Button - TextMeshPro. Click the button and in the Inspector in Rect Transform set Left, Top, Right, Bottom and PosZ to 0. In Anchors set Min X to 0.75, Max X to 0.95, Min Y to 0.05 and Max Y to 0.15. Set Pivot X to 1 and Pivot Y to 0. Add a runtime only on click event for Depth_ScreenToWorldPosition.ToggleMode and drag and drop your XR Origin from the Hierarchy into its field. In the Hierarchy below the button you can click its text and set the initial string in TextMeshPro - Text (UI) to Mode: Build
- Right click your Canvas and under UI select Text - TextMeshPro. Click the text and in the Inspector in Rect Transform set Left, Top, Right, Bottom and PosZ to 0. In Anchors set Min X to 0.05, Max X to 0.2, Min Y to 0.05 and Max Y to 0.15. Set the initial string in TextMeshPro - Text (UI) to dirt
- Right click your Canvas and under UI select Image. Click the image and in the Inspector in Rect Transform set PosX, PosY and PosZ to 0. Set Width and Height to 10. In Anchors set Min X to 0.5, Max X to 0.5, Min Y to 0.5 and Max Y to 0.5. In Image set Source Image to Knob
- In the Hierarchy click your XR Origin and in Depth_ScreenToWorldPosition (Script) under UI & Mode drag and drop both your button and your text from the Canvas into their respective fields
-
APK Build
- Save everything in your project
- Open Build Profiles under File in the top menu bar
- Under Platforms click Android and Switch Platform
- Once Android is the active platform click Build and choose a name and location for your APK. If you encounter errors regarding the input system, make sure to click Edit, Project Settings, Player, the Android icon and under Configuration set Active Input Handling to Both. Doing this might trigger a warning during the build process which you may ignore.
- You can now send / upload the APK to your Android device and run it. I suggest using WhatsApp Web to send the apk. After allowing WhatsApp to run APKs from unknown sources, you can simply click the APK on your phone to install and start it. For security reasons, don't forget to prohibit the execution of external apps afterwards!