Introduction
About
DirtScan is a open source project, that aims to develop a process to turn mountain bike trails into 3D models.
These 3D models can then be displayed on the web, for riders to find and explore.
Goals
- Be able to display a mountain bike trail digitally
- Be as easy and low cost as possible
- Have a reasonable level of accuracy.
How it works
DirtScan is at a high level, a customised Photogrammetry pipeline. For those interested, it is based off of the COLMAP software.
Photogrammetry, is the process of reconstructing 3D models from lots of individual images.
I won’t go into detail, but at a high level these are the main steps to be aware of. Understanding this will allow you to understand the why behind lots of the decisions made.
Main Steps
1 - Photo taking
Photos are taken of the intended object, here, a mountain bike trail. Lots of photos are taken, from as many angles as possible.
The rule is, if the camera can’t see it, it doesn’t end up in the 3D model - the camera has no concept of space like a human does.
2 - Feature Extraction
Features here are simply parts of the photo, that are stable under different lighting conditions. Think: leaves, rocks, sticks on the ground; the corner of a log; the edge of a tree.
A algorithm runs over all the photos and finds what could be useful reference points in each photo.
3 - Matching
After features are extracted, another algorithm tries to pair images up by comparing their features. Basically, it tries to work out what images saw the same objects.
4 - Mapping
After matching, an algorithm then runs and tries to place these photos in 3D space, such that their matching features line up. It tries to build a “map” of where you walked with the camera. This is not a perfect process - often features are ever so slightly misaligned due to external factors such as lens distortion, bluriness, lighting, etc. The mapping algorithm tries to minimise the error, but it can never fully remove it.
5 - Dense
Now that each photo is placed in a 3D world, another algorithm then runs and projects every (or nearly every) pixel into the 3D world. It compares the output of all the cameras that see that pixel, and tries to find the optimal location.
This produces a dense point cloud - a large (sometimes 10+ million) cloud of 3D points, that form the “surface” of the model.
6 - Meshing
This step takes the dense point cloud and runs an algorithm over it to get a proper 3D mesh.
7 - Texturing
This step takes the mesh, and the actual photos from the cameras, and projects the actual images to form actual textures on the surface of the model.
Additional optional steps
There are other tangential/optional steps that can be run, such as:
- Cleaning the point cloud to remove tree foliage (this often appears as blobs)
- Removing sun-cast shadows from textures
- Selectively improving detail in some parts of the model, either via more detailed dense triangulation, or by supplying additional photos.
- Automatic detection of objects in the images/model (coming soon)
Equipment
Photo Taking
Actually taking photos of a trail only requires some common equipment:
GoPro
- GoPro with GPS (newer models do not have GPS anymore…)
- Micro SD Card (for reference, a 1KM trail takes about 12GB of storage), I use a 512GB.
- Enough batteries (usually 1 battery per 800m).
A GoPro is used because:
- It is relatively common.
- It includes lots of extra useful data (such as gyoroscope and GPS).
- It has a removable SD card with decent write speeds (there is a lot of data to transfer).
- It has swappable batteries.
- It is quite rugged and weather proof.
- It is a simple, single lens.
Currently, only the Hero 10 Black has been verified to work. It is important that you know that using any other type of GoPro will likely require some adjustments, due to differences in lenses.
Scale Bar
A scale bar is a reference object used in scans, in order to give the camera something of known size.
It is extremely important, that the scale bar is as rigid as possible. Even millimeteres of bow/flex is enough to throw off the accuracy of the reconstruction.
Given this, my current scale bar is as follows:
- 1m square steel tube
- 3x flat builders shim (just a flat piece of plastic)
- 3x APRILtags, attached to the shims with double sided tape
3 tagStandard41h12 tags are printed (see https://novanix.github.io/apriltag-pdf-gen/) and placed onto 3 of the flat builders shims. These shims are then bolted onto the square steel tube (1 at start, 1 center, 1 at end).
Then, the distances between these 3 tags are measured extremely carefully, and entered into the reconstruction settings.
Fluro token
This is just a piece of plastic that is fluroescent and easily visible. Because the scanning requires 2 passes of the trail (forwards and backwards), the token is dropped onto the ground to mark where the turnaround point was, so you don’t need to remember it.
Processing scans
Unfortunately, processing these models does require a decently equipped PC due to the sheer amount of computation required.
I have had success on my mid-range PC:
- 16GB RAM
- RTX3070 8GB Graphics Card
- AMD Ryzen 5 5600X 6-Core Processor
- Linux
- Enough storage (a 1KM trail takes about 50GB of total storage for all the intermediary data)
The only difference really will be the processing time. Currently with the above setup, the end to end processing time is about 20 hours per KM of trail (I aim to get this as low as possible though!).
The only hard constraint really is a modern-ish graphics card - due to Nvidia CUDA driver support being dropped for older graphics cards.
Capturing a scan
Equipment
In addition to the actual scanning equipment noted in equipment, you’ll also want the usual essentials (you should already have these if you are a mountain biker!):
- Enough water
- First aid kit
- Good shoes
- Knowledge of the area and routes, and a map.
Planning
Direction & Starting point
It is best to start a scan at the end of a trail, and work upwards where possible (obviously if a trail is largely flat, either direction works). This is done to improve the visibility of the ground in the camera - the camera can see the ground better if the ground is rising in front of it (i.e. you are going uphill).
Time required
Ensure you have enough time - scanning a trail takes about 90 minutes per km + additional time to walk to the start/end of trails.
Weather
The best weather for a scan is a overcast but dry day, because this gives soft lighting with no cast shadows.
However, waiting for an overcast day is unrealistic. The scan will still work in a normal sunny day, only that there will be cast shadows in the texture. I am developing software based methods to remove these shadows after the fact.
The only weather you cannot do a scan in is:
- In the rain, because rain drops drop onto the camera ruining image quality (also, its just not great to get wet)
- Shortly after rain - as there will be puddles on the trail that you likely do not want to scan.
Day of the week
Because you will be walking on the trail to scan it, you MUST be mindful of other riders, and always move to the side if a rider is coming down.
Given the interruption this causes, it is also best to do a scan on a non-peak day (generally this means on a weekday, and avoiding weekends and public holidays).
Time of day
Just because you, a human, can see - does not mean the camera can (without bluriness). Aim for middle of the day / afternoon, avoid early morning or late afternoon as there often is not enough sunlight.
Actual scanning process
Assuming you have a gopro, scale bar, and fluro token, and are positioned at the start of a trail, here is what you do.
GoPro Setup
- I like to factory reset the SD card, so the only photos on there are from this run.
- Switch to timelapse mode, 0.5s per image
- Use the wide lens setting (never used the others, as they are software based corrections, which ruins the image for our purposes)
GoPro GPS Quirk GoPros do not maintain an accurate GPS fix even if switched on - they must be recording to maintain a good fix. Given this, it is often good to record anything for about 5 minutes, then discard this, to prompt it to get a good GPS lock before starting a scan.
Optional: the scan helper The scan helper is a small web page that connects to the GoPro over Bluetooth and speaks cues (photo count at 110/120/140/270/280, walking too fast or slow, no GPS lock), so you don’t have to watch the camera. You still start and stop the camera yourself. It needs a Web Bluetooth browser (Bluefy on iPhone, Chrome on Android).
The 4 step loop
Because mountain bike trails are very long, processing is done in pieces (called stations)
Each station:
- Starts with a view of the scale bar
- Ends with a view of the scale bar
- Has a forward pass and a reverse pass, which see the same objects but from opposite sides.
It is important that the above is always true, otherwise the software does not correctly know how to match the images correctly.
Starting a station
Place the scale bar on the ground in a position that is:
- Not on the trail itself. I.e. not in the way of where a rider would ride.
- Has little/no uneven shadows cast on it. These can affect detection.
- Is clear of obstructions (e.g. no leaves or grass in front of the tags)
- Is secure, and won’t shift/move during the station scan.
Station scan - forward and reverse pass
- Starting with all 3 tags in the scale bar in view, start the timelapse.
- Slowly move the camera left and right looking at the scale bar - for about 10 photos
- Then slowly move backwards to “show” the scale bar in its environment - for about 10 photos
- Slowly pan towards forward, facing the trail
- Ensure the GoPro is facing (assuming right handled), 45 degrees to the LEFT. You do not want to have the camera face directly forward - Facing on the diagonal creates more parallax and results in a better reconstruction.
- Start walking, at about 1 step per 0.5s (i.e. matching the timelapse). Not too fast but not too slow.
- Monitor the timelapse photo counter. Once it reaches 120 photos, out of frame of the camera, drop the fluro token on the ground somewhere. This is a marker for where to return to later.
- Keep walking forwards past the fluro token until you reach 140 photos.
- Slowly turn around, and then continue walking back to the scale bar (you will walk past the fluro token - leave it where it is). Continue holding the camera at a 45 degree angle to the left.
- Walk all the way back to the scale bar
- Do the reverse of what you did at the start - slowly bring the scale bar into view, showing it in its surroundings and then coming in closer.
- Stop the timelapse. You should have about 280 photos per chunk (140 forward, 140 reverse).
Next chunk
You have just scanned one chunk!
- With the camera off, pick up the scale bar
- Walk back up the trail to the fluro token - you will now place the scale bar down in a location near the fluro token.
- Pick up the token, and restart (from the first step, placing the scale bar).
General tips
If the camera does not see it, it does not exist.
Ensure the camera has enough of a view of all parts you wish to capture.
A common culprit here is medium/tall berms. You should raise the camera high enough to see behind the berm, otherwise in the processed model behind the berm may be blurry/filled with generic/guessed data.
Detailed objects (e.g. wooden features)
Objects that raise above the ground, e.g. Wall rides, bridges, etc - these also have an underside. You need to also capture the underside correctly, otherwise these will not be visible in the processed model.
There are 2 ways you can do this:
- During the scan - You can do 2 loop passes, one of the front and one of the back.
- Supplemental photos - The processing software allows additional photos to be added to an existing path. Say you scanned the front of a wallride, you could then add in additional photos of the rear of the wallride.
For wallrides or any other tall object, ensure to capture some photos that show both the front and the back of the wallride in one photo, as otherwise these two parts can become disconnected (because the computer does not know it is one object, unless it can see them both in one photo).
Splits / Joins
These are not yet properly supported, we assume trails at the moment to be single paths.
Safety advice
You will be walking on a mountain bike trail, and there is a risk of injury or collision with a rider.
Given this, you MUST be aware of these risks and how they are managed:
- Always try to do this on a off-peak day. The best way to avoid a rider is to never encounter one in the first place.
- When walking UP the trail, always be on the lookout for a rider coming down.
- When walking DOWN the trail, always be vigilant of looking BEHIND you at a potential rider coming down.
- As suggested earlier, always aim to start from the end of a trail and move towards the start. This means that 66% of your walking is upwards, meaning you have good visibility if a rider is approaching.
- Never have headphones or loud music that impair you sense of sound, this is often the best early warning of a rider approaching.
- If you suspect a rider is approaching, immediately move to the side of the trail, even if it turns out to be a false alarm (e.g. a branch falling).
- Never be on the trail unless you are actively scanning. If you need to check your phone, or tweak settings, or put in a new battery, do this standing off to the side.
- Be acutely aware and familiar with blind corners, blind jumps - anywhere you do not have good visibility of - proceed in these areas with extra caution.
Processing a scan
Now you have captured the images, how is this turned into a 3D model?
This is done using the tfi tool I have developed. See the code releases page for the latest version.
You must have somewhat of a technical capability as it does require installation and compiling on your own machine - I do not distribute prebuilt binaries.
Please check the README.md in the repository for instructions on what you need to run it.
To process a scan simply go through each of the steps as prompted:
- Select/load a workspace (this just stores the file paths and configuration)
- Select the first step,
scan image data, and enter the images folder - Run this step, and all corresponding steps as they complete.
Improving scans in certain areas
If a certain area of a scan did not work well, maybe because the photos were bad or there were not enough, you are able to add “supplemental photos” by linking their folder in the relevant step and specifying the station numbers they affect.
The tool will automatically detect what needs to be rebuilt for each step and rebuild it.
Final output
Once you have completed all the steps, you should have a number of outputs:
- The route the camera took as a line
- The mesh of the route, both with and without textures.
- 3D tiles, which is used to display the route in an optimised way through the web.
If you wish to display these yourselves, go ahead!
Otherwise, please contact me if you’d like to add them to our database ❤️