Expedition weather forecasts via satellite using a Markov rANS codec
2
stars
680
commits
TypeScript
primary language
Sep 13, 2026
updated
Going Blue is an expedition weather app designed specifically for satellite messengers. It was built for a Denali ski expedition with one goal: to get you all the weather information you would have at home, wherever you are. Going Blue uses a custom codec to pack hundreds of forecast data points into a single message that can be sent over SMS, Garmin inReach, ZOLEO, or iPhone satellite messaging. Going Blue is deployed at going.blue and is available on the App Store.
There are a few components to the system:
The service is written in TypeScript. There are four packages:
packages/protocol: shared TypeScript binary encoding/decoding used by both the server and the mobile app.packages/server — Hono/Node.js server that receives inbound SMS messages, fetches forecasts from Open-Meteo, encodes messages using codec services, and sends replies. Also hosts the website.packages/codec-server — Codec server for encoding messages. Each codec version is deployed as a separate container so that old codecs can be frozen and maintained for clients on older versions.packages/mobile — Expo React Native app for building requests and visualizing forecasts.Going Blue uses a Markov model of weather combined with a range Asymmetric Numeral Systems (rANS) entropy coder. This is a similar entropy coder to what is used in modern compression codecs like zstd and JPEG-XL.
First, we need to define an important concept: entropy. Entropy is a measure of the average level of uncertainty in a system. It is defined as the sum of the probability of each symbol multiplied by the logarithm of its probability:
$$H(X) := -\sum_{x \in \mathcal{X}} p(x) \log_2 p(x)$$
Here we use a base-2 logarithm to measure entropy in bits (Shannon entropy).
Entropy depends on the probability distribution of symbols. For example, if we have two symbols that are equally likely (e.g. flipping a coin), then entropy is 1 bit:
$$H(\text{coin}) = -\left(\tfrac{1}{2} \log_2 \tfrac{1}{2} + \tfrac{1}{2} \log_2 \tfrac{1}{2}\right) = 1 \text{ bit}$$
If we have a coin that has two heads, the entropy is 0 since there is no uncertainty.
$$H(\text{two-headed coin}) = -1 \log_2 1 = 0 \text{ bits}$$
Now, let's apply this to weather data with an example of encoding the weathercode, which is a general summary of weather conditions in a single symbol. There are 28 different weathercodes, so if weathercodes were uniformly distributed the entropy would be $\log_2 28 \approx 4.807 \text{ bits}$. Fortunately, weathercodes are not uniformly distributed:
| Weathercode | Probability |
|---|---|
| ☁️ overcast | 32.25% |
| ☀️ clear sky | 31.13% |
| 🌤️ mainly clear | 9.52% |
| ⛅ partly cloudy | 7.73% |
| 🌦️ light drizzle | 7.38% |
| … everything else | 12.00% combined |
The actual entropy of the weathercode is about 2.69 bits, far below the uniform distribution. To decrease the entropy even more, we can take advantage of the fact that the current weather is a good predictor of future weather. We can model weather as a series of state transitions with different probabilities, i.e. a Markov chain ☀️ -> ☀️ -> ⛅ -> ⛅ -> 🌦️. If it is currently sunny, this is the probability distribution of the next hour's weather:
| Next hour | Probability |
|---|---|
| ☀️ clear sky | 85.40% |
| 🌤️ mainly clear | 8.68% |
| ⛅ partly cloudy | 2.67% |
| ☁️ overcast | 2.58% |
| 🌦️ light drizzle | 0.458% |
| … everything else | 0.192% combined |
The entropy conditioned on the previous code is only 0.83 bits/symbol, 5.8x smaller than the uniform distribution!
Now that we know the probability distribution and the entropy of the data, we can encode it. To start, we can use Huffman coding, which assigns codes to symbols based on their probability. More likely symbols get shorter codes, less likely symbols get longer ones, and the expected length of the code approaches the entropy of the data.
| Weathercode | P | Bits | Huffman Code |
|---|---|---|---|
| ☀️ clear sky | 85.40% | 1 | 0 |
| 🌤️ mainly clear | 8.68% | 2 | 10 |
| ⛅ partly cloudy | 2.67% | 3 | 110 |
| ☁️ overcast | 2.58% | 4 | 1110 |
| 🌦️ light drizzle | 0.458% | 5 | 11110 |
| … everything else | 0.192% combined | 6+ | 111110… |
In this example, the clear -> clear transition is very likely so it gets a 1-bit code: 0. The clear -> light drizzle transition is unlikely, so it gets a 5-bit code: 11110. The expected length of the encoded forecast is 1.25 bits/symbol, which is near the 0.83 bits/symbol entropy of the data. This is just the expected length, and the actual length can vary depending on the data. An all-clear forecast takes just 1 bit/symbol to encode while a forecast with rare weathercodes could take even more than 5 bits/symbol.
Huffman coding gets us near the entropy of the data, but it has a major flaw: each code takes at least one bit. Even though the entropy of weathercode is 0.83 bits/symbol, we can only reach 1.25 bits/symbol because of the one-bit floor.
To get around this limitation and get closer to the actual entropy of the data, Going Blue uses rANS.
The basic idea of rANS is to encode the data in a single large integer. Each integer is mapped to a symbol based on its probability distribution. The mapping is set up so that encoding a likely symbol requires a small increase in the number and encoding a rare symbol requires a large increase.
Let's start with a simplified example. Say it is clear (C) 75% of the time and raining (R) 25% of the time. We can assign numbers to each state like this:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ...
C C C R C C C R C C C R C C C R C C C R ...
When encoding a symbol from state x, we find the xth occurrence of that symbol. For example, if we start at state 1 with symbol R, we find the first occurrence of R which is 4. To encode C from state 4, we find the 4th occurrence of C, which is 5. If we encode another R, we find the 5th occurrence of R, which is 20. The message RCR can be represented as the number 20.
To decode, we walk the process in reverse. At state 20, the symbol is R and it is the 5th R, so the previous state was 5. At state 5, the symbol is C and it is the 4th C so the previous state was 4. At 4, the symbol is R and it is the 1st R so the state goes to 1, which is the end of the message.
The reason rANS works is that encoding a symbol multiplies the state by roughly $1/p$. For symbol R, it multiplies the state by $1/(1/4) = 4$, which is about 2 bits. For C, it multiplies the state by $1/(3/4) = 4/3$, which is about 0.4 bits.
rANS gets us very close to the actual entropy of the data. For a more detailed explanation of how rANS works, see this excellent post.
For rANS and other entropy coders, both sides need to know the probability distribution of the data. The distributions are bundled into the app as part of each codec version. When the client makes its request, it sends a version at the beginning of the request and the server uses the corresponding distributions to encode the message.
rANS allows us to transmit data at close to its actual entropy. To further compress the data, we need to reduce its entropy.
There are several ways that we can do this. The first is quantization, or reducing the precision of each variable. Going Blue aims to preserve perceptible differences while not wasting bits on unnecessary precision. For example, temperature is transmitted at 1°C precision because it's unlikely that a difference of <1°C is going to change the decision a person makes based on the forecast. Wind is encoded using the Beaufort scale, which is tuned for noticeable differences in wind speed, e.g. Beaufort 1 "Direction shown by smoke drift but not by wind vanes" versus Beaufort 2 "Wind felt on face; leaves rustle; wind vane moved by wind".
For variables like snow and rain, which are sparse but have large variability, we use a sqrt scale. This provides detail at small amounts while preserving range for larger values. With rain, we might have an hour with 0.1mm rain and a 12 hour period with 100mm of rain. A sqrt scale allows us to represent both extremes on a scale with only 64 values. Rain values range from 0.036mm to 144mm and snow from 0.05cm to 200cm in a single time period.
\begin{aligned}
\mathrm{encode:}\quad c &= \min\left(\left\lfloor 63\sqrt{\frac{v}{v_{\max}}} \right\rceil,\ 63\right) \\
\mathrm{decode:}\quad \hat{v} &= v_{\max}\left(\frac{c}{63}\right)^2
\end{aligned}
| Code | 0 | 1 | 2 | 3 | … | 16 | 32 | 48 | … | 62 | 63 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Rain (mm) | 0 | 0.036 | 0.145 | 0.327 | … | 9.29 | 37.15 | 83.59 | … | 139.47 | 144.00 |
| Step | — | 0.036 | 0.109 | 0.181 | … | 1.13 | 2.29 | 3.45 | … | 4.46 | 4.54 |
For variables with large ranges, like temperature, we encode the starting temperature and then the delta of each forecast point. This avoids having a separate codebook for every possible temperature. It also allows the codec to more easily capture trends. For example, if the temperature rose 2°C in the last hour, it is likely still rising in the next hour. The delta provides more information about the next hour's temperature than the absolute temperature does.
We can also use correlation between variables to reduce entropy. Weathercode is great for this because it is always included and it is a summary of the weather. If the current weathercode is clear, it's probably not raining. We can split weathercodes into buckets (rainy, snowy, dry, etc.) and use that to condition the precipitation variables (snow, rain, precip chance). Variables that have diurnal cycles, like temperature, can be conditioned by the local time of day. If it's solar noon, the temperature is more likely to be rising. Forecast resolution (1h, 3h, 6h, 12h) is also used since the forecast resolution affects the amount of accumulation and magnitude of change in each period.
All of these refinements are chosen to make the distribution of each codebook as skewed as possible and thus decrease the entropy of the data. More refinements could be done, but there is a cost to having more codebooks and a smaller sample size for each codebook.
These are the units and techniques used for each variable:
| Variable | Model | Unit | Codebook keyed by |
|---|---|---|---|
| Weathercode | Value | WMO Code | Previous weathercode |
| Temperature | Delta | 1°C (-100°C to 155°C) | Previous temperature delta, time of day, forecast resolution |
| Precip chance | Value | % in 8 steps | Previous value, weathercode class, forecast resolution |
| Snow | Value | cm, 64 sqrt-companded steps (0-200cm) | Previous value bucket, weathercode class, forecast resolution |
| Rain | Value | mm, 64 sqrt-companded steps (0-144mm) | Previous value bucket, weathercode class, forecast resolution |
| Freezing level | Delta | 1000ft steps (0-31,000ft) | Temperature delta bucket, forecast resolution |
| Cloud band | Value | % in 8 steps | Previous value, pressure level |
| Wind gust | Delta | Extended Beaufort force (0-17) | Forecast resolution |
| Surface wind speed | Delta | Extended Beaufort force (0-17) | Wind gust delta, forecast resolution |
| Pressure-level wind speed | Delta | Extended Beaufort force (0-17) | Pressure level, forecast resolution |
| Wind direction | Value | 8 cardinal directions | Previous direction, forecast resolution |
| AQI, Ozone, NO₂ (diurnal cycle) | Delta | Air quality index (US: 0-500, EU: 0-100), 25 bands | Previous delta, time of day, forecast resolution |
| PM2.5, PM10, SO₂ | Delta | Air quality index (US: 0-500, EU: 0-100), 25 bands | Previous delta, forecast resolution |
| Dominant pollutant | Value | Pollutant (PM2.5, PM10, Ozone, SO₂, NO₂) | Previous dominant pollutant |
| Model agreement | Value | 4 levels, strong disagreement to strong agreement | Previous agreement, lead time |
| Dewpoint | Delta | 1°C | Temperature delta, dewpoint depression, forecast resolution |
| Relative humidity | Derived | % | Derived from temperature and dewpoint |
| Feels like temperature | Derived | 1°C | Derived from temperature, wind, and relative humidity |
Since Going Blue uses an entropy coder, the forecast length is not predictable. It depends on the entropy of the forecast, with stable (low-entropy) conditions taking few bits to encode and variable (high-entropy) conditions taking many bits to encode. In practice, forecasts with the default variable set range between 40 and 225 time periods, with an average near 100.
We can't promise a 3 day hourly forecast or a 10 day forecast at 3h resolution. At the minimum of 40 data points, we can choose between almost 2 days of hourly data or a 10 day forecast at 6h resolution. The app allows the user to select a fill priority: detail, auto, or range. The server fetches the forecast and then tries to fit as much data into the message as possible. At each step, it can either extend the range of the forecast (up to 13 days) or increase the detail (12h/6h/3h/1h resolution). The fill priority determines which it tries to do. These fill ladders are pre-defined and shared between the server and client. The server sends back a sequence number so that the client can derive the resolution of each forecast point. The server does a binary search of the sequence number to find the largest forecast that can fit within the character budget.
To keep the message small, the server never sends the client information that it already has. When the client creates a request, it stores request metadata like forecast location, model, variables, priority mode, UTC offset, and request time in a local cache. The client sends a request index to the server and the server sends that index back in the response. The client can then recover all of the forecast metadata from that index. Using this, the entire header can be packed into just 5 characters:
| Field | Bits | Meaning |
|---|---|---|
version | 7 | base-85 index = protocol version; read before anything else |
index | 7 | message index the client stores its request context under |
seq | 8 | fill sequence number to derive forecast length and layout |
elev | 7 | elevation in 100 m steps |
SMS is the transport layer for Going Blue. Satellite messengers like Garmin inReach, iPhone, and ZOLEO can all send and receive messages via SMS. To reach Going Blue, a message travels through several intermediaries. For example, the path an inReach message takes looks something like this:
inReach -Iridium Short Burst Data (SBD)-> Garmin -SMS-> Twilio -HTTP-> Going Blue server
Each part of the chain has a different character set and message length limit. In the case of Garmin, these are:
The alphabet that Going Blue can use is the intersection of all of these: 160 characters of GSM-7 basic ∩ printable ASCII (minus space), or base-85. This provides log₂(85) ≈ 6.409 bits per character and approximately 1025 bits per message.
Each device has a different character set and message length. Going Blue chooses how to encode a message based on the device:
| Device | Network | Alphabet1 | Message length | Bits per message |
|---|---|---|---|---|
| SMS | Cellular | base-124 (GSM-7 basic) | 160 chars | ~1110 |
| Garmin inReach | Iridium Short Burst Data | base-85 (GSM-7 basic ∩ printable ASCII) | 160 chars | ~1025 |
| ZOLEO | Iridium Short Burst Data | base-85 (GSM-7 basic ∩ printable ASCII) | 240 chars | ~1538 |
| iPhone satellite messaging | Globalstar | base-32768 | 50 chars2 | ~707 |
| Internet | Internet | base-94 (printable ASCII) | — | — |
Entropy coding requires having accurate statistics about the distribution of each symbol since sequences that aren't represented in the training data will be very expensive to encode. For example, if we trained the codebooks only on tropical weather forecasts, the encoder would assign very long symbols to snow and a forecast in the arctic would be very expensive.
The encoder is trained on over 100k historical forecasts collected from the Open-Meteo Historical Forecast API. These forecasts are sampled from 10,000 locations across the world. Forecast locations are not uniformly sampled across the globe since that would bias the forecasts strongly towards the ocean. Instead, the forecast points are allocated based on 30 Köppen climate classes in proportion to the square-root of the area of the climate class. This ensures that rare climate classes have enough training data while still allocating more share to more common climate types.
Ocean locations are not included in Köppen but they are included in the training data with an 85/15 land/ocean split. This gives the ocean a similar weight to a high-level Köppen climate class (tropical, arid, temperate, continental, polar). Ocean locations are sampled from 6 30° latitude bands with the same sqrt(area) allocation as climate classes.
Training data is pulled from the two year window July 2024 - July 2026. 12 14-day forecasts are collected for each location for an average of 1 forecast every 2 months. This ensures coverage of all seasons while also reducing forecast duplication.
1,500 forecast locations are held out for evaluation and never used to train the encoder. 137 of my Windy favorites are also used as an evaluation set since these are the places I actually want weather forecasts for. There is also a small set of 150 peaks used in evaluation to make sure that Going Blue works well in the mountains. There is a custom page for exploring the evaluation results at going.blue/benchmark.
Fill percentage is the main codec performance metric. 100% represents a forecast filled to maximum range and resolution (13 days, hourly data). Encoding improvements should increase this percentage.
Some interesting findings from the evaluation:
Going Blue uses Open-Meteo for weather data with some transformations that are explained below.
Open-Meteo accepts an elevation parameter for forecasts and adjusts temperature from the model's grid cell elevation using temperature lapse rate. It does not adjust other variables like precipitation type. This can lead to contradictory forecasts in the mountains. For example, a forecast for the summit of Denali may show very low temperatures and rain if it is raining at the grid cell elevation (~3000m for GFS).
To fix this, rain is remapped to snow if the forecast elevation is above the freezing level. It is also remapped to snow if the temperature is below -2°C to handle inversions and forecast centers that do not support freezing level (GEM, ECMWF). Snow is never remapped to rain. Rain is translated to snow at a 7:1 SWE ratio for parity with Open-Meteo. More accurate snow:liquid mapping may be added in the future.
The following weathercodes are remapped:
Freezing drizzle (56/57) and freezing rain (66/67) are not transformed.
Open-Meteo provides cloud cover at various pressure levels. This is calculated based on the relative humidity compared to the critical relative humidity at each pressure level using Sundqvist's formula. The pressure-level cloud data drives the detailed cloud view in the meteogram, which shows clouds at 10 different levels in the atmosphere. This information can help determine what type of clouds are forecast: high cirrus overcast, a lenticular on the summit, or valley fog?
There is a subtle problem with using clouds at each pressure level directly: the pressure-level variable only reports clouds that are exactly at that band. If there is a cloud at 20k but we only pull the 18k and 24k bands, we will miss that cloud entirely. This can lead to inconsistent forecasts where we report "cloudy" in the weathercode but the meteogram shows no clouds.
To fix this, Going Blue attributes low (<3km), mid (3-8km), and high (>8km) cloud cover to their respective pressure levels. The low, mid, and high cloud cover variables are derived from the tens to hundreds of pressure levels within each model, so there are no gaps.
First, each pressure level is associated with a band using geopotential heights. For example:
If the band reports clouds but none of its member levels do, the member levels are assigned clouds based on their relative humidity. Clouds from the low/mid/high band are split between the levels in the band whose humidity is furthest above critical relative humidity.
Open-Meteo is an hourly weather API but Going Blue forecast periods range from 1h to 12h. Going Blue summarizes the hourly weathercodes of a period in a single weathercode for the period. Showery codes are used to represent mixed conditions. For example, if it snows 3 hours in a 12h period and is sunny the remaining 9 hours, a "snow showers" code will be used.
Open-Meteo does not emit mixed rain/snow weathercodes. Going Blue uses a mixed code if the water equivalent of the lesser type of precipitation exceeds 25% of the total precipitation. For example, in a period with 1" of snow (~0.14" water equivalent) and 0.1" of rain, rain accounts for 42% of the precip so it gets a mixed code. With 1" of snow and 0.01" of rain, rain is just a trace at 7% of total precip and the snow code is used.
Air quality is sourced from the CAMS model which has 11km resolution in Europe and 44km resolution in the rest of the world. The US and Europe have separate air quality scales that have different weights and health thresholds for each pollutant. Both scales calculate the index of each constituent pollutant and then take the maximum index as the headline AQI. The constituent pollutants are:
In practice, PM2.5 and ozone drive the headline AQI with PM10 a distant third. The other pollutants are rarely the main concern. Dominant pollutant frequency by scale:
Because of this, the headline AQI can be derived from other pollutants if they are already present in the message. If at least PM2.5 and ozone are present, just the residual between the estimated AQI and the actual AQI is sent. The residual is almost nothing (~0.036 bits/period) if PM2.5, ozone, and PM10 are already in the message. With PM2.5 and ozone, the headline AQI only costs 0.275 bits/period on the American scale and 0.653 bits/period on the European. This is significantly cheaper than encoding headline AQI without the constituent variables, which costs roughly 1 bit/period.
Going Blue reports the headline AQI in addition to the dominant pollutant. It can also report the index of any individual pollutant with the exception of Carbon Monoxide, which is US-only and rarely a problem.
Going Blue computes a model agreement score that indicates how well the current forecast agrees with the American, Canadian, European, and German centers. This is useful for judging forecast confidence.
An agreement score is calculated for each forecast center. The score has 4 levels from 0 (strong disagreement) to 3 (strong agreement). Agreement takes into account temperature, precipitation, and wind. For each variable, the agreement is calculated as a score between 0 (disagreement) and 1 (agreement) like this:
a and b are scored like sqrt(min(a, b) / max(a, b)) so that equal amounts are scored as 1 and large differences approach 0. If one model reports wet and one reports dry, the agreement score ranges from 0.55 (one model dry, one model at trace precip) to 0 (one model dry, one model with significant precip).The components are combined using a weighted soft min with precip at 60%, wind at 30%, and temperature at 10%.
The combined agreement score (0 to 1) is then mapped to an agreement level: strong disagreement, weak disagreement, weak agreement, strong agreement. The thresholds are chosen to roughly align to quartiles of actual agreement scores calculated from live forecasts.
The nice thing about these variables is that they can be derived from one another. Only dewpoint is actually sent in the message. Relative humidity can be derived from temperature and dewpoint using the Magnus formula. Apparent temperature can be derived from temperature, relative humidity, and wind speed. Wet bulb temperature and cloud base could also be derived from existing variables. By sending some indicator of humidity, we get a lot for free!
Apparent or "feels like" temperature is calculated from temperature, sustained wind, and relative humidity. It does not take solar radiation into account. There are three ways apparent temperature is calculated depending on conditions:
Development dependencies:
brew install node@26npm install -g pnpm@9.15.4brew install --cask dockerbrew install tmuxnpm install -g eas-cli, then run eas login.brew install cocoapodsbrew install ngrok, then run ngrok config add-authtoken.Everything needed to run locally is bundled into a tmux session for easy setup:
./dev.sh install: Install dependencies and fetch the global basemap that is bundled in the app../dev.sh start: Start the development environment../dev.sh stop: Stop the development environment../dev.sh reset: Stop the development environment and clear the development database. Note that this will invalidate all existing tokens, so reset the account on your dev device afterwards../dev.sh tunnel: Start the development environment with an ngrok tunnel for the gateway and Expo server. This is useful for testing app changes on a physical device while on a public network.The services run on the following ports by default:
Run the mobile app in development mode from packages/mobile:
eas build --platform ios --profile developmenteas build --platform android --profile developmentThis will produce a QR code which you can scan to install the development app on your device.
The app can also be run in the simulator:
npx expo run:iosnpx expo run:androidFor a preview build that uses the deployed server:
eas build --platform ios --profile previeweas build --platform android --profile previewGoing Blue has a large set of unit tests that can be run like this:
pnpm test
benchmark: Regenerate the public encoding benchmark at going.blue/benchmarkcodec-version: Freeze the codec and bump the codec version.database: Read the CloudSQL database.generate-codebook: Generate a codebook.logs: Read gcloud logs.trace: Trace a request through Twilio, service logs, and the database.twilio: Read Twilio logs and alerts.screenshots: Take App Store screenshots.The Going Blue codec relies on the client and server having identical codebooks. Since clients may be out of service and unable to update for long periods of time, the service maintains support for old versions. Each forecast request starts with a version number e.g. v1. The gateway server routes each message to the appropriate codec service. Each version of the codec is a separate container running from main tagged at a specific version. Golden messages are kept for each codec version so that changes to the codec service can be made (for example patching security vulnerabilities) while ensuring that the message format does not change. This approach allows the codec to evolve quickly without sacrificing support for older clients in the field.
Copyright 2025-2026 Lane Aasen
Licensed under the Apache License, Version 2.0. You may use, modify, and distribute this software, including commercially, provided you retain the copyright and license notices and state any significant changes you make. The license includes an express patent grant. See NOTICE for the required attribution notice.
Forecast data comes from Open-Meteo via its public API and is subject to Open-Meteo's own terms; no Open-Meteo source code is included here.
680 commits
TypeScript
94.5%
Python
2.8%
HTML
1.3%
Expedition weather forecasts via satellite using a Markov rANS codec
2
stars
680
commits
TypeScript
primary language
Sep 13, 2026
updated
Going Blue is an expedition weather app designed specifically for satellite messengers. It was built for a Denali ski expedition with one goal: to get you all the weather information you would have at home, wherever you are. Going Blue uses a custom codec to pack hundreds of forecast data points into a single message that can be sent over SMS, Garmin inReach, ZOLEO, or iPhone satellite messaging. Going Blue is deployed at going.blue and is available on the App Store.
There are a few components to the system:
The service is written in TypeScript. There are four packages:
packages/protocol: shared TypeScript binary encoding/decoding used by both the server and the mobile app.packages/server — Hono/Node.js server that receives inbound SMS messages, fetches forecasts from Open-Meteo, encodes messages using codec services, and sends replies. Also hosts the website.packages/codec-server — Codec server for encoding messages. Each codec version is deployed as a separate container so that old codecs can be frozen and maintained for clients on older versions.packages/mobile — Expo React Native app for building requests and visualizing forecasts.Going Blue uses a Markov model of weather combined with a range Asymmetric Numeral Systems (rANS) entropy coder. This is a similar entropy coder to what is used in modern compression codecs like zstd and JPEG-XL.
First, we need to define an important concept: entropy. Entropy is a measure of the average level of uncertainty in a system. It is defined as the sum of the probability of each symbol multiplied by the logarithm of its probability:
$$H(X) := -\sum_{x \in \mathcal{X}} p(x) \log_2 p(x)$$
Here we use a base-2 logarithm to measure entropy in bits (Shannon entropy).
Entropy depends on the probability distribution of symbols. For example, if we have two symbols that are equally likely (e.g. flipping a coin), then entropy is 1 bit:
$$H(\text{coin}) = -\left(\tfrac{1}{2} \log_2 \tfrac{1}{2} + \tfrac{1}{2} \log_2 \tfrac{1}{2}\right) = 1 \text{ bit}$$
If we have a coin that has two heads, the entropy is 0 since there is no uncertainty.
$$H(\text{two-headed coin}) = -1 \log_2 1 = 0 \text{ bits}$$
Now, let's apply this to weather data with an example of encoding the weathercode, which is a general summary of weather conditions in a single symbol. There are 28 different weathercodes, so if weathercodes were uniformly distributed the entropy would be $\log_2 28 \approx 4.807 \text{ bits}$. Fortunately, weathercodes are not uniformly distributed:
| Weathercode | Probability |
|---|---|
| ☁️ overcast | 32.25% |
| ☀️ clear sky | 31.13% |
| 🌤️ mainly clear | 9.52% |
| ⛅ partly cloudy | 7.73% |
| 🌦️ light drizzle | 7.38% |
| … everything else | 12.00% combined |
The actual entropy of the weathercode is about 2.69 bits, far below the uniform distribution. To decrease the entropy even more, we can take advantage of the fact that the current weather is a good predictor of future weather. We can model weather as a series of state transitions with different probabilities, i.e. a Markov chain ☀️ -> ☀️ -> ⛅ -> ⛅ -> 🌦️. If it is currently sunny, this is the probability distribution of the next hour's weather:
| Next hour | Probability |
|---|---|
| ☀️ clear sky | 85.40% |
| 🌤️ mainly clear | 8.68% |
| ⛅ partly cloudy | 2.67% |
| ☁️ overcast | 2.58% |
| 🌦️ light drizzle | 0.458% |
| … everything else | 0.192% combined |
The entropy conditioned on the previous code is only 0.83 bits/symbol, 5.8x smaller than the uniform distribution!
Now that we know the probability distribution and the entropy of the data, we can encode it. To start, we can use Huffman coding, which assigns codes to symbols based on their probability. More likely symbols get shorter codes, less likely symbols get longer ones, and the expected length of the code approaches the entropy of the data.
| Weathercode | P | Bits | Huffman Code |
|---|---|---|---|
| ☀️ clear sky | 85.40% | 1 | 0 |
| 🌤️ mainly clear | 8.68% | 2 | 10 |
| ⛅ partly cloudy | 2.67% | 3 | 110 |
| ☁️ overcast | 2.58% | 4 | 1110 |
| 🌦️ light drizzle | 0.458% | 5 | 11110 |
| … everything else | 0.192% combined | 6+ | 111110… |
In this example, the clear -> clear transition is very likely so it gets a 1-bit code: 0. The clear -> light drizzle transition is unlikely, so it gets a 5-bit code: 11110. The expected length of the encoded forecast is 1.25 bits/symbol, which is near the 0.83 bits/symbol entropy of the data. This is just the expected length, and the actual length can vary depending on the data. An all-clear forecast takes just 1 bit/symbol to encode while a forecast with rare weathercodes could take even more than 5 bits/symbol.
Huffman coding gets us near the entropy of the data, but it has a major flaw: each code takes at least one bit. Even though the entropy of weathercode is 0.83 bits/symbol, we can only reach 1.25 bits/symbol because of the one-bit floor.
To get around this limitation and get closer to the actual entropy of the data, Going Blue uses rANS.
The basic idea of rANS is to encode the data in a single large integer. Each integer is mapped to a symbol based on its probability distribution. The mapping is set up so that encoding a likely symbol requires a small increase in the number and encoding a rare symbol requires a large increase.
Let's start with a simplified example. Say it is clear (C) 75% of the time and raining (R) 25% of the time. We can assign numbers to each state like this:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ...
C C C R C C C R C C C R C C C R C C C R ...
When encoding a symbol from state x, we find the xth occurrence of that symbol. For example, if we start at state 1 with symbol R, we find the first occurrence of R which is 4. To encode C from state 4, we find the 4th occurrence of C, which is 5. If we encode another R, we find the 5th occurrence of R, which is 20. The message RCR can be represented as the number 20.
To decode, we walk the process in reverse. At state 20, the symbol is R and it is the 5th R, so the previous state was 5. At state 5, the symbol is C and it is the 4th C so the previous state was 4. At 4, the symbol is R and it is the 1st R so the state goes to 1, which is the end of the message.
The reason rANS works is that encoding a symbol multiplies the state by roughly $1/p$. For symbol R, it multiplies the state by $1/(1/4) = 4$, which is about 2 bits. For C, it multiplies the state by $1/(3/4) = 4/3$, which is about 0.4 bits.
rANS gets us very close to the actual entropy of the data. For a more detailed explanation of how rANS works, see this excellent post.
For rANS and other entropy coders, both sides need to know the probability distribution of the data. The distributions are bundled into the app as part of each codec version. When the client makes its request, it sends a version at the beginning of the request and the server uses the corresponding distributions to encode the message.
rANS allows us to transmit data at close to its actual entropy. To further compress the data, we need to reduce its entropy.
There are several ways that we can do this. The first is quantization, or reducing the precision of each variable. Going Blue aims to preserve perceptible differences while not wasting bits on unnecessary precision. For example, temperature is transmitted at 1°C precision because it's unlikely that a difference of <1°C is going to change the decision a person makes based on the forecast. Wind is encoded using the Beaufort scale, which is tuned for noticeable differences in wind speed, e.g. Beaufort 1 "Direction shown by smoke drift but not by wind vanes" versus Beaufort 2 "Wind felt on face; leaves rustle; wind vane moved by wind".
For variables like snow and rain, which are sparse but have large variability, we use a sqrt scale. This provides detail at small amounts while preserving range for larger values. With rain, we might have an hour with 0.1mm rain and a 12 hour period with 100mm of rain. A sqrt scale allows us to represent both extremes on a scale with only 64 values. Rain values range from 0.036mm to 144mm and snow from 0.05cm to 200cm in a single time period.
\begin{aligned}
\mathrm{encode:}\quad c &= \min\left(\left\lfloor 63\sqrt{\frac{v}{v_{\max}}} \right\rceil,\ 63\right) \\
\mathrm{decode:}\quad \hat{v} &= v_{\max}\left(\frac{c}{63}\right)^2
\end{aligned}
| Code | 0 | 1 | 2 | 3 | … | 16 | 32 | 48 | … | 62 | 63 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Rain (mm) | 0 | 0.036 | 0.145 | 0.327 | … | 9.29 | 37.15 | 83.59 | … | 139.47 | 144.00 |
| Step | — | 0.036 | 0.109 | 0.181 | … | 1.13 | 2.29 | 3.45 | … | 4.46 | 4.54 |
For variables with large ranges, like temperature, we encode the starting temperature and then the delta of each forecast point. This avoids having a separate codebook for every possible temperature. It also allows the codec to more easily capture trends. For example, if the temperature rose 2°C in the last hour, it is likely still rising in the next hour. The delta provides more information about the next hour's temperature than the absolute temperature does.
We can also use correlation between variables to reduce entropy. Weathercode is great for this because it is always included and it is a summary of the weather. If the current weathercode is clear, it's probably not raining. We can split weathercodes into buckets (rainy, snowy, dry, etc.) and use that to condition the precipitation variables (snow, rain, precip chance). Variables that have diurnal cycles, like temperature, can be conditioned by the local time of day. If it's solar noon, the temperature is more likely to be rising. Forecast resolution (1h, 3h, 6h, 12h) is also used since the forecast resolution affects the amount of accumulation and magnitude of change in each period.
All of these refinements are chosen to make the distribution of each codebook as skewed as possible and thus decrease the entropy of the data. More refinements could be done, but there is a cost to having more codebooks and a smaller sample size for each codebook.
These are the units and techniques used for each variable:
| Variable | Model | Unit | Codebook keyed by |
|---|---|---|---|
| Weathercode | Value | WMO Code | Previous weathercode |
| Temperature | Delta | 1°C (-100°C to 155°C) | Previous temperature delta, time of day, forecast resolution |
| Precip chance | Value | % in 8 steps | Previous value, weathercode class, forecast resolution |
| Snow | Value | cm, 64 sqrt-companded steps (0-200cm) | Previous value bucket, weathercode class, forecast resolution |
| Rain | Value | mm, 64 sqrt-companded steps (0-144mm) | Previous value bucket, weathercode class, forecast resolution |
| Freezing level | Delta | 1000ft steps (0-31,000ft) | Temperature delta bucket, forecast resolution |
| Cloud band | Value | % in 8 steps | Previous value, pressure level |
| Wind gust | Delta | Extended Beaufort force (0-17) | Forecast resolution |
| Surface wind speed | Delta | Extended Beaufort force (0-17) | Wind gust delta, forecast resolution |
| Pressure-level wind speed | Delta | Extended Beaufort force (0-17) | Pressure level, forecast resolution |
| Wind direction | Value | 8 cardinal directions | Previous direction, forecast resolution |
| AQI, Ozone, NO₂ (diurnal cycle) | Delta | Air quality index (US: 0-500, EU: 0-100), 25 bands | Previous delta, time of day, forecast resolution |
| PM2.5, PM10, SO₂ | Delta | Air quality index (US: 0-500, EU: 0-100), 25 bands | Previous delta, forecast resolution |
| Dominant pollutant | Value | Pollutant (PM2.5, PM10, Ozone, SO₂, NO₂) | Previous dominant pollutant |
| Model agreement | Value | 4 levels, strong disagreement to strong agreement | Previous agreement, lead time |
| Dewpoint | Delta | 1°C | Temperature delta, dewpoint depression, forecast resolution |
| Relative humidity | Derived | % | Derived from temperature and dewpoint |
| Feels like temperature | Derived | 1°C | Derived from temperature, wind, and relative humidity |
Since Going Blue uses an entropy coder, the forecast length is not predictable. It depends on the entropy of the forecast, with stable (low-entropy) conditions taking few bits to encode and variable (high-entropy) conditions taking many bits to encode. In practice, forecasts with the default variable set range between 40 and 225 time periods, with an average near 100.
We can't promise a 3 day hourly forecast or a 10 day forecast at 3h resolution. At the minimum of 40 data points, we can choose between almost 2 days of hourly data or a 10 day forecast at 6h resolution. The app allows the user to select a fill priority: detail, auto, or range. The server fetches the forecast and then tries to fit as much data into the message as possible. At each step, it can either extend the range of the forecast (up to 13 days) or increase the detail (12h/6h/3h/1h resolution). The fill priority determines which it tries to do. These fill ladders are pre-defined and shared between the server and client. The server sends back a sequence number so that the client can derive the resolution of each forecast point. The server does a binary search of the sequence number to find the largest forecast that can fit within the character budget.
To keep the message small, the server never sends the client information that it already has. When the client creates a request, it stores request metadata like forecast location, model, variables, priority mode, UTC offset, and request time in a local cache. The client sends a request index to the server and the server sends that index back in the response. The client can then recover all of the forecast metadata from that index. Using this, the entire header can be packed into just 5 characters:
| Field | Bits | Meaning |
|---|---|---|
version | 7 | base-85 index = protocol version; read before anything else |
index | 7 | message index the client stores its request context under |
seq | 8 | fill sequence number to derive forecast length and layout |
elev | 7 | elevation in 100 m steps |
SMS is the transport layer for Going Blue. Satellite messengers like Garmin inReach, iPhone, and ZOLEO can all send and receive messages via SMS. To reach Going Blue, a message travels through several intermediaries. For example, the path an inReach message takes looks something like this:
inReach -Iridium Short Burst Data (SBD)-> Garmin -SMS-> Twilio -HTTP-> Going Blue server
Each part of the chain has a different character set and message length limit. In the case of Garmin, these are:
The alphabet that Going Blue can use is the intersection of all of these: 160 characters of GSM-7 basic ∩ printable ASCII (minus space), or base-85. This provides log₂(85) ≈ 6.409 bits per character and approximately 1025 bits per message.
Each device has a different character set and message length. Going Blue chooses how to encode a message based on the device:
| Device | Network | Alphabet1 | Message length | Bits per message |
|---|---|---|---|---|
| SMS | Cellular | base-124 (GSM-7 basic) | 160 chars | ~1110 |
| Garmin inReach | Iridium Short Burst Data | base-85 (GSM-7 basic ∩ printable ASCII) | 160 chars | ~1025 |
| ZOLEO | Iridium Short Burst Data | base-85 (GSM-7 basic ∩ printable ASCII) | 240 chars | ~1538 |
| iPhone satellite messaging | Globalstar | base-32768 | 50 chars2 | ~707 |
| Internet | Internet | base-94 (printable ASCII) | — | — |
Entropy coding requires having accurate statistics about the distribution of each symbol since sequences that aren't represented in the training data will be very expensive to encode. For example, if we trained the codebooks only on tropical weather forecasts, the encoder would assign very long symbols to snow and a forecast in the arctic would be very expensive.
The encoder is trained on over 100k historical forecasts collected from the Open-Meteo Historical Forecast API. These forecasts are sampled from 10,000 locations across the world. Forecast locations are not uniformly sampled across the globe since that would bias the forecasts strongly towards the ocean. Instead, the forecast points are allocated based on 30 Köppen climate classes in proportion to the square-root of the area of the climate class. This ensures that rare climate classes have enough training data while still allocating more share to more common climate types.
Ocean locations are not included in Köppen but they are included in the training data with an 85/15 land/ocean split. This gives the ocean a similar weight to a high-level Köppen climate class (tropical, arid, temperate, continental, polar). Ocean locations are sampled from 6 30° latitude bands with the same sqrt(area) allocation as climate classes.
Training data is pulled from the two year window July 2024 - July 2026. 12 14-day forecasts are collected for each location for an average of 1 forecast every 2 months. This ensures coverage of all seasons while also reducing forecast duplication.
1,500 forecast locations are held out for evaluation and never used to train the encoder. 137 of my Windy favorites are also used as an evaluation set since these are the places I actually want weather forecasts for. There is also a small set of 150 peaks used in evaluation to make sure that Going Blue works well in the mountains. There is a custom page for exploring the evaluation results at going.blue/benchmark.
Fill percentage is the main codec performance metric. 100% represents a forecast filled to maximum range and resolution (13 days, hourly data). Encoding improvements should increase this percentage.
Some interesting findings from the evaluation:
Going Blue uses Open-Meteo for weather data with some transformations that are explained below.
Open-Meteo accepts an elevation parameter for forecasts and adjusts temperature from the model's grid cell elevation using temperature lapse rate. It does not adjust other variables like precipitation type. This can lead to contradictory forecasts in the mountains. For example, a forecast for the summit of Denali may show very low temperatures and rain if it is raining at the grid cell elevation (~3000m for GFS).
To fix this, rain is remapped to snow if the forecast elevation is above the freezing level. It is also remapped to snow if the temperature is below -2°C to handle inversions and forecast centers that do not support freezing level (GEM, ECMWF). Snow is never remapped to rain. Rain is translated to snow at a 7:1 SWE ratio for parity with Open-Meteo. More accurate snow:liquid mapping may be added in the future.
The following weathercodes are remapped:
Freezing drizzle (56/57) and freezing rain (66/67) are not transformed.
Open-Meteo provides cloud cover at various pressure levels. This is calculated based on the relative humidity compared to the critical relative humidity at each pressure level using Sundqvist's formula. The pressure-level cloud data drives the detailed cloud view in the meteogram, which shows clouds at 10 different levels in the atmosphere. This information can help determine what type of clouds are forecast: high cirrus overcast, a lenticular on the summit, or valley fog?
There is a subtle problem with using clouds at each pressure level directly: the pressure-level variable only reports clouds that are exactly at that band. If there is a cloud at 20k but we only pull the 18k and 24k bands, we will miss that cloud entirely. This can lead to inconsistent forecasts where we report "cloudy" in the weathercode but the meteogram shows no clouds.
To fix this, Going Blue attributes low (<3km), mid (3-8km), and high (>8km) cloud cover to their respective pressure levels. The low, mid, and high cloud cover variables are derived from the tens to hundreds of pressure levels within each model, so there are no gaps.
First, each pressure level is associated with a band using geopotential heights. For example:
If the band reports clouds but none of its member levels do, the member levels are assigned clouds based on their relative humidity. Clouds from the low/mid/high band are split between the levels in the band whose humidity is furthest above critical relative humidity.
Open-Meteo is an hourly weather API but Going Blue forecast periods range from 1h to 12h. Going Blue summarizes the hourly weathercodes of a period in a single weathercode for the period. Showery codes are used to represent mixed conditions. For example, if it snows 3 hours in a 12h period and is sunny the remaining 9 hours, a "snow showers" code will be used.
Open-Meteo does not emit mixed rain/snow weathercodes. Going Blue uses a mixed code if the water equivalent of the lesser type of precipitation exceeds 25% of the total precipitation. For example, in a period with 1" of snow (~0.14" water equivalent) and 0.1" of rain, rain accounts for 42% of the precip so it gets a mixed code. With 1" of snow and 0.01" of rain, rain is just a trace at 7% of total precip and the snow code is used.
Air quality is sourced from the CAMS model which has 11km resolution in Europe and 44km resolution in the rest of the world. The US and Europe have separate air quality scales that have different weights and health thresholds for each pollutant. Both scales calculate the index of each constituent pollutant and then take the maximum index as the headline AQI. The constituent pollutants are:
In practice, PM2.5 and ozone drive the headline AQI with PM10 a distant third. The other pollutants are rarely the main concern. Dominant pollutant frequency by scale:
Because of this, the headline AQI can be derived from other pollutants if they are already present in the message. If at least PM2.5 and ozone are present, just the residual between the estimated AQI and the actual AQI is sent. The residual is almost nothing (~0.036 bits/period) if PM2.5, ozone, and PM10 are already in the message. With PM2.5 and ozone, the headline AQI only costs 0.275 bits/period on the American scale and 0.653 bits/period on the European. This is significantly cheaper than encoding headline AQI without the constituent variables, which costs roughly 1 bit/period.
Going Blue reports the headline AQI in addition to the dominant pollutant. It can also report the index of any individual pollutant with the exception of Carbon Monoxide, which is US-only and rarely a problem.
Going Blue computes a model agreement score that indicates how well the current forecast agrees with the American, Canadian, European, and German centers. This is useful for judging forecast confidence.
An agreement score is calculated for each forecast center. The score has 4 levels from 0 (strong disagreement) to 3 (strong agreement). Agreement takes into account temperature, precipitation, and wind. For each variable, the agreement is calculated as a score between 0 (disagreement) and 1 (agreement) like this:
a and b are scored like sqrt(min(a, b) / max(a, b)) so that equal amounts are scored as 1 and large differences approach 0. If one model reports wet and one reports dry, the agreement score ranges from 0.55 (one model dry, one model at trace precip) to 0 (one model dry, one model with significant precip).The components are combined using a weighted soft min with precip at 60%, wind at 30%, and temperature at 10%.
The combined agreement score (0 to 1) is then mapped to an agreement level: strong disagreement, weak disagreement, weak agreement, strong agreement. The thresholds are chosen to roughly align to quartiles of actual agreement scores calculated from live forecasts.
The nice thing about these variables is that they can be derived from one another. Only dewpoint is actually sent in the message. Relative humidity can be derived from temperature and dewpoint using the Magnus formula. Apparent temperature can be derived from temperature, relative humidity, and wind speed. Wet bulb temperature and cloud base could also be derived from existing variables. By sending some indicator of humidity, we get a lot for free!
Apparent or "feels like" temperature is calculated from temperature, sustained wind, and relative humidity. It does not take solar radiation into account. There are three ways apparent temperature is calculated depending on conditions:
Development dependencies:
brew install node@26npm install -g pnpm@9.15.4brew install --cask dockerbrew install tmuxnpm install -g eas-cli, then run eas login.brew install cocoapodsbrew install ngrok, then run ngrok config add-authtoken.Everything needed to run locally is bundled into a tmux session for easy setup:
./dev.sh install: Install dependencies and fetch the global basemap that is bundled in the app../dev.sh start: Start the development environment../dev.sh stop: Stop the development environment../dev.sh reset: Stop the development environment and clear the development database. Note that this will invalidate all existing tokens, so reset the account on your dev device afterwards../dev.sh tunnel: Start the development environment with an ngrok tunnel for the gateway and Expo server. This is useful for testing app changes on a physical device while on a public network.The services run on the following ports by default:
Run the mobile app in development mode from packages/mobile:
eas build --platform ios --profile developmenteas build --platform android --profile developmentThis will produce a QR code which you can scan to install the development app on your device.
The app can also be run in the simulator:
npx expo run:iosnpx expo run:androidFor a preview build that uses the deployed server:
eas build --platform ios --profile previeweas build --platform android --profile previewGoing Blue has a large set of unit tests that can be run like this:
pnpm test
benchmark: Regenerate the public encoding benchmark at going.blue/benchmarkcodec-version: Freeze the codec and bump the codec version.database: Read the CloudSQL database.generate-codebook: Generate a codebook.logs: Read gcloud logs.trace: Trace a request through Twilio, service logs, and the database.twilio: Read Twilio logs and alerts.screenshots: Take App Store screenshots.The Going Blue codec relies on the client and server having identical codebooks. Since clients may be out of service and unable to update for long periods of time, the service maintains support for old versions. Each forecast request starts with a version number e.g. v1. The gateway server routes each message to the appropriate codec service. Each version of the codec is a separate container running from main tagged at a specific version. Golden messages are kept for each codec version so that changes to the codec service can be made (for example patching security vulnerabilities) while ensuring that the message format does not change. This approach allows the codec to evolve quickly without sacrificing support for older clients in the field.
Copyright 2025-2026 Lane Aasen
Licensed under the Apache License, Version 2.0. You may use, modify, and distribute this software, including commercially, provided you retain the copyright and license notices and state any significant changes you make. The license includes an express patent grant. See NOTICE for the required attribution notice.
Forecast data comes from Open-Meteo via its public API and is subject to Open-Meteo's own terms; no Open-Meteo source code is included here.
680 commits
TypeScript
94.5%
Python
2.8%
HTML
1.3%