Skip to content
GitHubYouTubeInstagram

How a robot sees.light → pixels → tag corners → a pose on the field

A camera sees an AprilTag, finds its four corners, and works out where the robot must be standing to see it that way. This course follows that path from the light hitting the sensor to a pose in the robot code. The ideas are the same on a Limelight, on PhotonVision, and on the Limelight vision built into SystemCore, the controller FRC uses from 2027.

7 chapters · about 49 minutes

  1. 01 · 6 minWhy robots need visionOdometry drifts; a tag at a known spot fixes it, if it covers enough pixels.
  2. 02 · 8 minInside the cameraGlobal shutter, mono sensors, lenses, and the settings that keep a moving tag sharp.
  3. 03 · 7 minFocus and calibrationFocus and lock the lens, then calibrate it with board snapshots that reach every corner of the picture.
  4. 04 · 7 minFrom corners to posePnP turns four corners into a pose; one tag can flip, and more tags or the gyro fix it.
  5. 05 · 8 minGetting it into the robotaddVisionMeasurement with a timestamp and a trust level, in PhotonLib and LimelightHelpers.
  6. 06 · 8 minMatch dayPit checks for Limelight and PhotonVision, a clean lens, and the course quiz.
  7. 07 · 5 minAdvanced: going fasterFewer pixels, GPUs, accelerators, and searching only where the tags were.
Chapter 1 of 7 6 min

Why robots need vision

Odometry drifts; a tag at a known spot fixes it, if it covers enough pixels.

Here is the whole course on one screen: a robot on the real 2026 field, what its camera sees, and where it works out it is.

LabDrive the robot. See what it sees.

Drag the robot, use the Turn slider, or click the field and use the arrow keys. Try getting close to a tag, seeing two at once, or facing an empty wall. The green dots are the robot's vision guesses of where it is: watch them spread out when the tags are far away.

Simplified: tags are never hidden, and the camera sits at the robot's center (0.5 m up, 12° tilt, 737 px focal length at 1280×800). Dot spread uses chapter 5's rule; the stage strip runs at 4 frames a second with typical, not measured, stage times. 2026 AndyMark tag layout.

A robot that knows its pose (where it is on the field, and which way it faces) can drive itself in auto, aim at a goal and line up in a fraction of a second. Its wheels give it a running guess, called odometry: count how far each wheel turned, take the heading from the gyro, and add it all up. The Swerve course builds that guess step by step.

Wheels slip. Push against another robot, drive over a bump or spin fast, and the wheel counts stop matching how far the robot moved. Each error adds to the last one, so after a minute of hard driving the guess can be off by more than a robot length.

LabWatch vision catch the drift

Press Get bumped!, then turn vision off, and watch the gray guess drift.

Where the robot really isWhat odometry thinksTag sighting

Vision is on.

Position error0 cm
Tag fixes0

Simplified: odometry over-counts distance by 4% and is 2° off in heading; each sighting pulls the guess 70% of the way to the truth. Real robots weight sightings by trust (chapter 5). Tag positions are illustrative.

A landmark a computer can read

The landmark is an AprilTag: a printed square pattern that a program can find in a picture quickly, read an ID from, and locate to a fraction of a pixel at each of its four corners. FRC uses the tag36h11 family. Each tag is a 6.5 in (165.1 mm) black square on a white plate, with a 6 × 6 grid of data squares inside the border.

The tags are mounted on field elements at published positions. FIRST gives every team a field layout file listing each tag's x, y, height and the way it faces, so when the robot sees tag 26 it knows where tag 26 is. The robot code and the vision software have to load the same layout file.

How the detector finds a tag

PhotonVision, Limelight and most FRC vision systems run the same AprilTag library. It turns a camera frame into tag IDs and corners in a few milliseconds, in 8 stages. Step through them below on a real 2026 field frame; every picture is computed from its pixels.

LabStep through the AprilTag detector

Press Next (or the arrow keys), or pick a stage number. Stages 2 to 7 zoom in on tags 29 and 30.

A frame arrives

Loading the camera frame…

Stages 2 to 6 are recomputed here on a 240×150 crop, with the library's threshold rule; the real detector runs them on the whole frame. The final corners, IDs and margins are the AprilTag library's real output for this frame. Frame: PhotonVision test images (GPL-3.0).

How far away can you read it?

A camera doesn't see a tag, it sees pixels. Our camera has 1280 pixels across an 82° view, which calibration (chapter 3) turns into a focal length of 737 pixels. That gives a rule:

pixels across the tag ≈ 737 × 0.1651 m ÷ distance

LabDistance test: tag vs QR code

Drag the distance slider. Find where each code stops being readable.

AprilTag, as the sensor sees it
Same-size QR code (25 × 25)
In the whole 1280 × 800 frame

Tag width—
Tag square—
QR square—

Simplified: the tag is face-on and centered, and every pixel averages the light that lands on it. Real images add lens blur, JPEG blocks and motion blur, which shorten the range. A code reads at about 2 px per square. Focal length from our Thriftiest Cam calibration; a different camera or resolution moves every number.

The detector reads each square from the pixel in its middle, so it needs about 2 pixels per square. A tag's squares are 20.6 mm; a same-size QR code's are 6.6 mm, so the tag reads about 3 times farther away. That's why FRC uses a tag with 36 big squares instead of a QR code.

What does the seeing

Finding tags many times a second is too much work to add on top of the robot code, so a second computer does it: a coprocessor. Most teams run either a Limelight, a camera and computer in one box with its own software, or PhotonVision, free software that runs on a small computer like an Orange Pi with one or more USB cameras. SystemCore, the FRC controller from the 2027 season, integrates Limelight vision technology, so a USB webcam plugged into it can do the same job. The ideas here apply to all of them; where the code differs, we show both.

The one thing to remember Odometry drifts and never recovers by itself. An AprilTag at a known spot gives the robot a fresh fix, as long as the tag covers enough pixels to read.

Next: Inside the camera ↓

Chapter 2 of 7 8 min

Inside the camera

Global shutter, mono sensors, lenses, and the settings that keep a moving tag sharp.

A camera counts light. Its sensor is a grid of about a million tiny light buckets, 1,024,000 on a 1280 × 800 sensor. While the exposure lasts, light fills each bucket with electrons. Then the camera measures every bucket, turns it into a number from 0 (black) to 255 (white), and sends the grid to the coprocessor. A fast camera does this 120 times a second.

Global shutter vs rolling shutter

Most phone cameras and webcams have a rolling shutter: they read the image one row at a time, top to bottom, over most of a frame. At 30 frames per second (fps) that scan takes about 30 ms. If the tag moves during the scan, the top of the tag is photographed in one place and the bottom somewhere else, so the square leans over and the pose comes out wrong.

LabTwo ways to take a picture

Raise the speed or lower the frame rate and watch the rolling photo lean.

Real scene
Global shutter
Rolling shutter

Photo every—
Moves between—
Rolling lean—

Slow-motion illustration. A rolling sensor reads its rows over about 90% of a frame, so a higher frame rate scans faster and leans less; a global shutter catches every row at once. A typical webcam runs 30 fps; an OV9281 at full size, 120.

Why black and white

A pixel only counts photons; it can't tell their color. A color sensor puts a red, green or blue filter over each pixel in a Bayer filter pattern, and each filter throws away about two-thirds of the light. The camera then guesses the two missing colors from the neighbors, which smears edges. A tag is black and white, so a mono sensor gets about 3 times the light and keeps the edges sharp.

LabMono vs color sensor

Drag Light level down. Compare the noise and the edge colors.

Mono sensor
Color, raw
Color, demosaiced

Loading a real frame…

A real tag corner from our camera, 32 × 32 sensor pixels; raw color shows one color per pixel. Same exposure for both sensors; the color image is brightened 3× to match, which triples its noise too. Photon counts are illustrative (400 for a white pixel at full light).

The lens decides how wide a slice of the field those pixels cover. A wide lens sees more of the field but puts fewer pixels on each tag, so it reads tags less far.

The settings

Every vision system gives you the same few knobs, on Limelight's and PhotonVision's Input tab:

  • Exposure: how long each pixel collects light. Longer is brighter, but anything moving smears.
  • Gain: amplifies the signal after it's collected. Brighter, but the noise is amplified too.
  • Black level / brightness: shifts every pixel lighter or darker by the same amount.

Exposure matters most. A robot turning at 3 rad/s smears a tag 11 px at 5 ms and 44 px at 20 ms, while a tag square 4 m away is only 4 px wide. So keep exposure short and win the light back with gain or black level. Too dark, though, and the tag's decision margin drops: the detector is less sure it read the squares right, and below a cutoff it throws the tag away.

LabTune the camera

Turn all four goals green. Try raising Exposure while the robot spins.

Loading a real frame…

  • Found—
  • Sharp—
  • Corners—
  • FPS—
More controls, and why
Motion blur—
Clipped pixels—

Simplified camera model on a real field frame (a PhotonVision test image of the 2024 speaker, tags 3 and 4), taken as shot at 5 ms, so 5 ms with no spin shows it as shot. Each tag's margin is computed on these pixels and scaled to match the AprilTag library's reading on the real frame; the goals follow the weaker tag. The cutoff of 15 is ours; set your own in the AprilTag pipeline. Exposure is how long the shutter is open (brighter but blurrier); gain amplifies signal and noise alike; 2 rad/s of spin is 115°/s.

Lights at events flicker 120 times a second. If the stream pulses in brightness, set the exposure to 8.3 ms or 16.7 ms, a whole number of flicker cycles.

The one thing to remember Use a short exposure on a robot, around 5 ms, and win back the light with gain or black level. Tune it with the robot turning.

Next: Focus and calibration ↓

Chapter 3 of 7 7 min

Focus and calibration

Focus and lock the lens, then calibrate it with board snapshots that reach every corner of the picture.

Say the camera sees a tag's corner at pixel 900 across. Which direction is that? It depends on the lens, and no two lenses are the same: the glass sits a little off-center and bends straight lines, most near the sides of the picture. So every camera, Limelights included, gets two jobs before it goes on a robot: focus the lens and lock it, then calibrate it.

Focus, then lock it

Most FRC cameras use an M12 lens that screws into its holder. Turning it moves the glass toward or away from the sensor, which changes what distance is sharp. A soft image blurs the tag's edges and makes the corners jitter. Focus at the distance you care about most, usually 2–4 m, then lock the lens with glue or thread locker. A robot's vibration will turn a loose lens, and every turn spoils the calibration.

What calibration measures

Vision math starts from a pinhole camera, where straight lines stay straight. Four numbers, all in pixels, describe one: the focal lengths fx, fy (ours are about 737 px) and the principal pointcx, cy, the pixel directly behind the lens, which is rarely the exact middle. A real lens also bends the picture, so calibration adds a few distortion coefficients. All of them together are the camera's intrinsics.

To measure them you show the camera a flat board whose pattern you know exactly. Limelight and PhotonVision both calibrate from their web UI with a chessboard or a ChArUco board, a chessboard with a small marker in each white square so every corner can be named. You take snapshots of the board all over the picture, and the software finds the lens numbers and board positions that best explain every corner it saw, by least squares. The average miss left over is the reprojection error.

LabCalibrate a camera

Press Auto ×10 until the grid is straight. Then Clear; try Center only.

SnapshotsCorners seenGrid via lensTrue grid

  • Reach all 4 corners—
  • Reproj. < 1 px—
  • Lens error < 2 px—
Lens, and the fitted numbers
Lens
Taken—
fx—
cx, cy—

Simplified: the camera and board (12 × 9 squares of 30 mm) are simulated, with 0.7 px of corner noise. The lens is one of our cameras' real calibrations, fitted with the same 12-number solve Limelight and PhotonVision run.

A low reprojection error only means the fit matches the snapshots you took. Limelight's docs say to take at least 50 and call under 1 px good. Spread them to every edge and corner, close and far, tilted and square-on.

The one thing to remember Focus the lens and lock it, then calibrate with snapshots that reach every corner of the image. A low reprojection error from center-only snapshots can still be badly wrong at the edges.

Next: From corners to pose ↓

Chapter 4 of 7 7 min

From corners to pose

PnP turns four corners into a pose; one tag can flip, and more tags or the gyro fix it.

The detector hands over four corner pixels per tag. From those, the camera's calibration (chapter 3) and the field layout, the robot works out where it is standing. Since 2024 every FRC tag is the same size, a 165.1 mm (6.5 in) tag36h11 black square (2023 used 6 in tag16h5 tags), so its size in the picture gives distance: twice as far, half as many pixels.

distance = fx × 0.1651 m ÷ tag size in pixels

LabSimilar triangles

Drag the tag nearer or further. The pink band is how far off the distance would be with 1 px of corner error.

Tag in the image—
If corners are 1 px off—

Not to scale: the real sensor sits about 2 mm behind the pinhole. fy = 737.7 px from our TopLeft camera's calibration.

The error grows with the square of the distance: 1 px of corner error is about 3 cm at 2 m and 30 cm at 6 m.

LabDetected corners

Pick a frame. Green is the tag's outline, orange dots are the four corners in the order the detector reports them.

Loading frames…

Real field frames: PhotonVision test images (GPL-3.0). Corners from the AprilTag library.

PnP: all the clues at once

PnP (Perspective-n-Point) asks: where must the camera be, and which way must it face, so the tag's four real corners land on these four pixels? The answer is 6 numbers: x, y and z for position, and roll, pitch and yaw for which way it's tilted and turned. Three links chain together, and the lab below draws each one: the layout says where the tag is on the field,PnP says where the camera is from the tag, and the camera's mount, measured on the robot, says where the robot is from the camera. Every camera needs its mount measured and entered.

The catch is ambiguity. A small, far tag seen nearly straight on has almost no perspective: turned 20° left or 20° right, its near edge is only about 1% taller than its far edge. Two poses, on opposite sides of the tag's center line, both fit the corners. PhotonVision reports the ratio of the two fits' errors; its docs suggest rejecting single-tag poses with ambiguity above 0.2.

LabWhich side of the tag are you on?

Compare the two camera views. Drag Distance down to 1 m and watch them come apart. Then put Distance back at 4 m and raise Camera below the tag to 0.62 m, a camera 0.5 m up looking at a hub tag: watch where the flipped answer goes in the 3D view.

The camera
The flipped answer
Both outlines on the same pixels

Drag to turn, right-drag or Shift-drag to move, Ctrl + scroll or pinch to zoom (two fingers on a phone).

Where the camera really isThe flipped answer

Tag across—
Biggest corner shift—
Flipped robot off the floor—

Rendered at our camera's real pixel size (f = 737.8 px), a 165.1 mm tag, no blur. The flipped answer is the real camera mirrored through the tag's center line, the second of the two answers a single-tag solve (IPPE) gives. The tag is 1.124 m up, as on the 2026 hub face; 0.5 px is the PnP lab's starting corner noise.

Standing there, you'd know which side you're on from the carpet, the walls and your own feet. The solver gets four corner points and nothing else.

A camera looking up at the tag helps a little with the picture and a lot with the answer. At 4 m and 20°, mounting the camera 0.62 m below the tag only grows the corner shift from 0.29 px to 0.40 px, still under the jitter. But the two answers mirror each other through the tag's center line up and down as well as left and right, so the flipped answer is 0.62 m above the tag and the robot with it, 1.24 m off the carpet. Our floor rule (chapter 5) throws that out, and PhotonLib's estimateClosestToCameraHeightPose() picks the answer whose camera height matches the mount. A camera level with the tag gets neither help.

LabSolve the camera's pose

Drag the robot to move it; drag anywhere else to turn the view (two fingers on a phone). The camera takes 80 frames from where the robot sits. Each has a little noise on its corners, and each dot is where that one frame says the robot is. Blue dots are close to right; pink dots flipped to the wrong side. Drag the robot back and to one side until pink appears, then try Two tags and One tag + gyro.

Scroll here to load the 3D scene.
zoom: rings every 10 cm
Distance—
Average miss—
Flips—
Ambiguity—
Where the robot really isOne frame's answerOne frame's answer, flippedCorner

More controls

Simplified: the camera mount and calibration are perfect, so only corner noise moves the dots. Tags 3 and 4 are placed as on the 2026 AndyMark red hub face (0.3556 m apart, 1.124 m up); the camera mount (0.3 m forward, 0.5 m up, 8° up) and the 1° gyro error are illustrative. Average miss leaves out flipped dots.

Two fixes: more tags, or the gyro

With two tags there are 8 corners, and the field layout says where each one sits relative to the others. Turning the camera to fit one tag's flip would throw the other tag's corners far off, so only one answer fits. A tag mounted a few centimeters off its place in the layout drags every multi-tag answer with it, so leave a badly mounted tag out of the solve.

With one tag, the weakest clue is the angle, and a small angle error goes a long way: the robot swings around the tag like a door on a hinge, so 1° of error at 5 m puts it 9 cm to the side. The robot's gyro already knows its heading to a fraction of a degree. If the solve takes heading from the gyro, and height, roll and pitch from the camera mount, only x and y are left, and the flip can't happen, because the flipped answer needs the robot to face a different way. With only 2 unknowns, even two corners are enough. Limelight calls this MegaTag2; PhotonVision calls it a constrained solve. Solving all the corners of two or more tags at once is MultiTag in PhotonVision and MegaTag1 in Limelight.

The heading has to be the one from the moment the frame was captured, which is what the timestamp in chapter 5 is for.

The one thing to remember One tag can fit two poses meters apart. Use two or more tags, or take the heading from the gyro, and filter single-tag poses with ambiguity above 0.2.

Next: Getting it into the robot ↓

Chapter 5 of 7 8 min

Getting it into the robot

addVisionMeasurement with a timestamp and a trust level, in PhotonLib and LimelightHelpers.

A vision pose is one guess and odometry is another. WPILib's pose estimators blend them: robot code callsaddVisionMeasurement(pose, timestampSeconds, stdDevs) with each vision answer, and the estimator nudges the odometry toward it.

Trust: standard deviations

The last argument is a standard deviation for x, y and heading, in meters and radians: small means "trust this a lot". Trust should fall off with distance and rise with more tags. The labs here use team 6328's 2023 rule, 0.01 × distance² ÷ tags meters: 9 cm for one tag at 3 m, 4.5 cm for two. Our own 2026 robot uses a table instead (below).

LabBlending two guesses

Drag the sliders and watch the green blend lean toward whichever guess is surer.

OdometryVisionBlended

Odometry spread—
Vision spread—
Weight on vision—

One dimension only. The vision spread is 6328's 0.01 × d² ÷ n rule; the odometry drift is illustrative. The real estimator does this for x, y and heading at once.

Don't trust a single tag's heading at all: pass a huge number (Limelight's MegaTag2 example uses 9,999,999) for the third value and let the gyro own heading.

What we throw out

A bad vision pose looks like any other pose once it's in the estimator, so the filter in front of it matters as much as the standard deviations. Our 2026 robot rejects an estimate when:

  • It's older than 1.0 s. The estimator silently drops anything past its 1.5 s buffer; rejecting it ourselves puts a camera with a bad clock in the log.
  • The robot turned faster than 1.6 rad/s in the last 0.3 s. A spin turns timestamp error into heading error, and MegaTag2 turns that into position error. We check the peak over the window because a frame that arrives just after a spin was taken during it. Team 254 uses the same 0.3 s.
  • Any tag's ambiguity is over 0.9 (MegaTag1). That's the single-tag flip from chapter 4.
  • The pose is off the field.
  • The tags cover 0.025% of the image or less. They're too far away to trust.
  • The solve tilts the robot more than 5° in roll or pitch. The robot sits flat on the carpet, so a tilt means a knocked camera or a bad solve.
  • The solve puts the robot more than 0.75 m off the floor. That's a bad solve or a wrong camera mount; 0.75 m is the AdvantageKit template's default.

What gets through is trusted by tier, from how many tags there are and how much of the image they fill:

What the camera seesx, y std dev
Two or more tags over 2% of the image, or the robot stopped (≤ 0.2 m/s) with tags over 4%0.1 m
Two or more tags over 0.2%0.25 m
One tag over 2%, within 0.5 m of where odometry thinks we are0.5 m
One tag over 1%, within 0.25 m1.0 m
Anything else at 0.03% or more (MegaTag1 also needs ambiguity under 0.25)1.5 m

Heading always gets 999,999, so the gyro owns it while enabled. Vision sets heading once, while the robot is disabled before the match: a MegaTag1 solve seeds the whole pose, and once that seed has held steady the cameras switch to MegaTag2.

Time: the timestamp

Every answer describes the past. Exposure, transfer, detection and the network add up to about 15 ms on a well-tuned coprocessor, and a robot at 4 m/s moves 6 cm in that time. Both libraries hand you the capture time: PhotonLib's timestampSeconds and LimelightHelpers' PoseEstimate.timestampSeconds already have the latency subtracted, on the robot's clock. Pass that, not Timer.getFPGATimestamp(), and the estimator applies the fix at that moment in the past and replays the odometry since.

LabWhy frame rate matters too

Pick a frame rate and watch how old the newest answer gets between frames. Then switch to Pretend it's now.

Average age—
Position error—
Frame gap—
New frames per loop—
More controls

Shown 6.7 times slower than real time. Average age = latency + half a frame period. The error without the timestamp is speed × that age, driving straight; spinning adds a heading error on top.

At 120 fps each 20 ms robot loop gets 2 or 3 new frames per camera, so read all of them, not just the latest.

Which way is positive

Both libraries report poses in WPILib's field frame. The origin is always the blue corner, on both alliances, so vision poses never need flipping on red. The same goes for the heading you send MegaTag2 with SetRobotOrientation: it must be the field heading from the blue origin, 0° facing the red wall, even on red (Limelight docs). If the gyro was zeroed facing the wrong way, or never seeded, every MegaTag2 pose is wrong. The one number you must get right yourself is where each camera sits on the robot, measured from the robot's center on the floor.

With the mount measured, the whole loop is short in both libraries:

// PhotonLib: one estimator per camera
PhotonCamera camera = new PhotonCamera("front");
PhotonPoseEstimator estimator = new PhotonPoseEstimator(
    AprilTagFieldLayout.loadField(AprilTagFields.kDefaultField), robotToCamera);

// every loop
for (var result : camera.getAllUnreadResults()) {
  var est = estimator.estimateCoprocMultiTagPose(result);
  if (est.isEmpty()) est = estimator.estimateLowestAmbiguityPose(result);
  est.ifPresent(e -> {
    int n = e.targetsUsed.size();
    double d = e.targetsUsed.stream()
        .mapToDouble(t -> t.getBestCameraToTarget().getTranslation().getNorm())
        .average().orElse(10);
    double xy = 0.01 * d * d / n;
    poseEstimator.addVisionMeasurement(
        e.estimatedPose.toPose2d(), e.timestampSeconds, VecBuilder.fill(xy, xy, 9999999));
  });
}

Load the field layout your event uses (welded or AndyMark) in both the robot code and the coprocessor.

Aiming without a pose

The simplest use needs no pose: turn until the tag is centered. PhotonLib's getYaw() and Limelight'sgetTX() are positive when the tag is to the right, and a positive WPILib turn is to the left, so the turn is −kP × yaw.

The one thing to remember Pass every vision pose with its own timestamp and a standard deviation that grows with distance².

Next: Match day ↓

Chapter 6 of 7 8 min

Match day

Pit checks for Limelight and PhotonVision, a clean lens, and the course quiz.

Vision problems at an event almost always turn out to be one of four things: you can't reach the camera, it doesn't see tags, it sees them badly, or the pose it reports is wrong. Between matches there are only a few minutes, so the order you check things in matters more than knowing every setting.

Finding the camera

Every device on the robot network has an address starting with your team number: 10.TE.AM.x, so team 3847 is 10.38.47.x. Give each camera a static IP so it's always in the same place. Limelight's docs suggest 10.TE.AM.11 (its page is on port 5801); PhotonVision's suggest anything from .6 to .19 (port 5800). The nameslimelight.local and photonvision.local work too, most of the time. When they don't, use the IP.

LabSymptom finder

Pick what's wrong, then answer the questions. Each answer narrows it down to a likely cause and the fix, quickest step first.

Covers the common cases for Limelight and PhotonVision, not every one. Ported from the pit chapter of our Jetson course, where it's longer and specific to our robot. Addresses from the Limelight 4 quick-start and PhotonVision networking docs (2026).

Keep the lens clean

A fingerprint or field dust blurs the tag's edges, so corners land in the wrong place and far tags drop out first. Wipe each lens with a microfiber cloth before every match (not a shirt or glass cleaner), and replace a scratched one and recalibrate.

A pit checklist

  1. Every camera's page loads, and its stream is at the expected fps.
  2. Lenses wiped; each camera boxes a tag held up in front of it.
  3. Field layout on the cameras matches the one in robot code and the event's field.
  4. Robot code logs a vision pose, and parked on a known spot it's within a few cm.

The one thing to remember Work from the outside in: reach the camera, see a box on a tag, then check the pose. And wipe the lens, because a smudge fails quietly.

Check yourself

Nine questions from the whole course, chapters 1 to 6.

QuizVision & AprilTags
  1. 1. Why can’t a robot rely on odometry alone for a whole match?

  2. 2. Each tag square covers about 1.5 pixels in the image. What happens?

  3. 3. The robot sees tags parked but loses them while driving fast. Which camera setting do you change first?

  4. 4. Why do AprilTag cameras use a global shutter?

  5. 5. You refocused a camera’s lens after calibrating it. What now?

  6. 6. Why is a pose from two or more tags in view better than one from a single tag?

  7. 7. What does using the gyro heading in the pose solve (MegaTag2, or a constrained solve) fix?

  8. 8. You pass vision poses to addVisionMeasurement. What else must go with each one?

  9. 9. Tags flicker in and out even close up, with the robot still. What do you check first?

Next: Advanced: going faster ↓

Chapter 7 of 7 5 min

Advanced: going faster

Fewer pixels, GPUs, accelerators, and searching only where the tags were.

Every frame the camera takes has to be exposed, sent to the computer, searched for tags and solved for a pose. Faster vision means more fixes per second and less delay. Farther vision means more pixels on a far tag. Those pull against each other: a 1280 × 800 frame has 1,024,000 pixels to search, and a 640 × 400 frame has a quarter as many but gives each tag half the pixels across.

Fewer pixels, more frames

The cheapest speedup is a setting. Turning up decimation tells the detector to look for tag outlines in a half-size copy of the image, then refine the corners at full size: about 4 times less searching, with most of the corner accuracy kept. The cost is range, because a far tag's squares shrink below 2 pixels sooner (chapter 1). A Limelight 4 runs 1280 × 800 at 120 fps or 640 × 400 at 240 fps, per its docs.

Getting frames off the camera

USB cameras on one computer share its USB bandwidth, which caps how many can run at full speed. A Limelight avoids that by putting the camera and computer in one box and sending results over Ethernet.

Many simple workers: the GPU

A detector asks the same question of every pixel: is it brighter than its neighbors? A CPU has a few fast cores. A GPU has hundreds or thousands of slow ones, and it wins when the work splits into pieces that don't depend on each other.

LabThe painting race

Watch both walls get painted, then switch the job to Each tile needs the last.

CPU, 6 fast cores—
GPU, 1 core per tile—

Illustrative: a GPU painter here is 5× slower per tile than a CPU painter, and there are 6 CPU painters. Ported from our Jetson course, whose GPU has 1,024 cores.

Finding tag outlines splits well. Decoding the few tags found doesn't, so it stays on the CPU.

Accelerators and machine learning

An accelerator is a chip built for neural networks. The Limelight 4 takes an optional Hailo-8 module, which its docs list at 80 fps for finding game pieces with a neural network. Networks are how cameras find game pieces, whose shape and color vary too much for a hand-written rule. For tags, the classic detector is fast and exact, so an accelerator mostly frees the CPU for everything else.

Search where the tags were: EagleEye

Between two frames at 120 fps a tag can't move far. EagleEye, an open-source vision system from Scythe Engineering, uses that: it searches only padded windows around where it found tags in the last frame, and goes back to a full-frame search when a tag isn't where it expected (Chief Delphi post).

The one thing to remember Measure first, then cut the work: fewer pixels, a smaller search, or hardware built for it. Every speedup trades against range or cost somewhere.

That's the Vision & AprilTags course.

Next up: Paths & Autonomous, or head back to all courses.