Console Login

Building Prometheus from Source with Older Go and Node Versions

The problem

You build Prometheus from source the way the README describes, on a machine whose Go and Node.js are older than the repository asks for. Two things go wrong in practice. First, if Go is not allowed to switch toolchains, the build stops with the message "go.work requires go >= 1.26.10". Second, a binary built with plain go build or go install answers the UI page with an HTTP 500 and "Error opening React index.html". I reproduced both below, and also checked the Node.js version warning that the make build prints.

The environment

This is a clean sandbox container, not a production server. I checked what it is before doing anything else, using one line that prints the kernel, the distribution, the installed Go and Node versions, the CPU count, memory and disk.

uname -a; cat /etc/os-release; go version; node -v; which docker promtool curl make; nproc; free -m; df -h / | tail -1
Linux vm 6.18.44-fc-v338 #1 SMP @0 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04.5 LTS"
go version go1.24.7 linux/amd64
v22.22.0
4

So the sandbox is Ubuntu 24.04.5 with four CPUs, Go 1.24.7 and Node.js 22.22.0. Docker and make are installed, and promtool was not on the PATH. I used the repository prometheus/prometheus (Apache-2.0 license). The latest push-triggered run of its "CI" workflow on main was reported as successful in this workflow run. I did not re-verify that status myself.

What I cloned

cd /tmp && git clone --depth 1 https://github.com/prometheus/prometheus 2>&1 | tail -3; cd prometheus && git log -1 --format='%H %s' && ls
Cloning into 'prometheus'...
bd517f6636ce1b6263dada9dfecb3fe92c84d96f Merge pull request #20400 from prometheus/krajo/otlp-nh-zero-threshold

Why it happens

There are two separate version requirements, and they are enforced very differently. The Go one is a hard requirement written into go.mod and go.work. The Node.js one is only a soft check in the Makefile. Seeing the two side by side explains why one is invisible to you and the other prints a warning.

Go: a hard minimum that the toolchain resolves for you

The repository declares its Go version in two files, go.mod and go.work, and both say the same thing. I printed the lines with grep, because this is the number that the Go toolchain compares with the version installed on the machine before it compiles anything.

cd /tmp/prometheus && grep -n "^go \|^toolchain" go.mod go.work
go.mod:3:go 1.26.10
go.work:1:go 1.26.10

My installed Go is 1.24.7, which is older than the declared minimum. Yet inside the repository the very same go command reports a different version than outside it, so the version you see depends on the directory you are standing in. Here are both answers, first from a directory outside the clone and then from inside it.

cd /tmp && go version
go version go1.24.7 linux/amd64
cd /tmp/prometheus && go version
go version go1.26.10 linux/amd64

Node.js: only a warning

The UI build is a separate step driven by make. The Makefile target assets starts with a check-node-version prerequisite, which runs a small script. That script compares the installed Node with the minimum in web/ui/.nvmrc and prints a yellow warning when it is lower. It does not stop the build.

cd /tmp/prometheus && cat web/ui/.nvmrc; node --version
24.21.0
v22.22.0

Source of truth

The Go toolchain documentation

The Go Toolchains documentation describes the rule that the go line in go.mod is a minimum, and that with the default GOTOOLCHAIN=auto setting an older go command downloads and runs the newer toolchain that the module requires. That explains the two different go version outputs above. It also tells us the download needs network access to a module proxy.

The repository README and Makefile

The README lists Go from go.mod and Node.js from .nvmrc as the requirements, offers go install and make build, and states that a plain go install build reads web assets from web/ui/static in the working directory, while make build compiles them in. The Makefile and scripts/check-node-version.sh define the Node check. Here are the relevant lines as printed in the sandbox.

cd /tmp/prometheus && sed -n 120,135p Makefile; sed -n 36,56p Makefile
.PHONY: assets
ifndef SKIP_UI_BUILD
assets: check-node-version ui-install ui-build
else
assets:
	@echo '>> skipping assets build, pre-built assets provided'
endif

The Makefile also sets SKIP_UI_BUILD when PREBUILT_ASSETS_STATIC_DIR is defined, so the UI build, and with it the Node version check, can be skipped completely if you provide the finished assets yourself. These are the lines in the Makefile that say so.

cd /tmp/prometheus && grep -n -A2 "^ifdef PREBUILT_ASSETS_STATIC_DIR" Makefile
41:ifdef PREBUILT_ASSETS_STATIC_DIR
42-  SKIP_UI_BUILD = true
43-endif

Ways to fix it

If your toolchain is older than the repository expects, there are several ways to deal with it. I only ran the first one fully, plus the plain go build variant as a comparison. The others are listed from the repository's own files and general knowledge, and I label them as untested.

Let Go fetch its own toolchain and accept the Node warning

Run make build as the README says. The default GOTOOLCHAIN=auto pulls Go 1.26.10, and the Node check only warns. This needs no changes to the machine. The cost is a network download of the toolchain, and building the UI with a Node release below the declared minimum.

Install the exact versions yourself

Install Go 1.26.10 system-wide and Node 24.21.0, for example with nvm, which the .nvmrc file is made for. This gives a build identical to the versions the project asks for, with no warnings and no hidden downloads, but it changes the machine, and I did not test this option in the sandbox.

Skip the UI build or use a binary

Provide pre-built assets through PREBUILT_ASSETS_STATIC_DIR, or skip building altogether and use the precompiled binaries or the prom/prometheus container image that the README lists. Docker is present in this sandbox, but I did not run the image, so I make no claims about it.

Plain go build without embedded assets

The README also describes go install. This is fast to type and needs no Node at all, but, as I show below, it produces a binary that cannot serve the web UI from other directories, because the assets are not compiled into it. It is a reasonable choice for working only with the API or with promtool.

Why this fix

I chose the first option: run make build unchanged and let Go switch toolchains. The reason is that it is the path the README documents, it leaves the host untouched, and the only risk is concentrated in one place, namely the Node version used for the UI bundle. That risk can be checked directly: either the finished binary serves the UI and its assets, or it does not.

Step by step

Confirm what happens without the toolchain switch

cd /tmp/prometheus && GOTOOLCHAIN=local go build -o /dev/null ./cmd/promtool; echo exit=$?
exit=1
go: go.work requires go >= 1.26.10 (running go 1.24.7; GOTOOLCHAIN=local)

With the switch disabled, the build stops immediately with a clear message that names the required version and the version that is running. With the default setting, Go had already downloaded the newer toolchain into its module cache the first time I ran a command inside the repository, which I listed like this.

cd /tmp/prometheus && go env GOTOOLCHAIN GOVERSION; ls $(go env GOMODCACHE)/golang.org/ | head
auto
go1.26.10
toolchain@v0.0.1-go1.26.10.linux-amd64
x

Build with make

cd /tmp/prometheus && (time make build) > /tmp/build.log 2>&1; tail -40 /tmp/build.log

The build log holds the shell's time result and the number of packages pnpm installed in its two runs, one for the main workspace and one for the old React app. I read both numbers straight from the log with grep, instead of quoting them from memory, so that they can be checked against the output shown here.

grep "^real" /tmp/build.log | tr '\t' ' '
real 4m21.058s
grep "^Packages:" /tmp/build.log
Packages: +615
Packages: +1385

That is 4 minutes 21 seconds of wall-clock time on the four-CPU sandbox, including the Go module downloads and the 2000 npm packages. Your time will differ with CPU count and network speed. I then pulled the Node warning out of the same log.

grep -o "Node.js version mismatch.*" /tmp/build.log
Node.js version mismatch! Required minimum: 24.21.0 Installed version: 22.22.0

The warning appeared exactly once and the build continued to the end, so it is informational only. The resulting binary reports the newer Go version and has the web assets compiled in, which the builtinassets tag shows in the version output below.

cd /tmp/prometheus && ./prometheus --version
prometheus, version 3.16.0 (branch: main, revision: bd517f6636ce1b6263dada9dfecb3fe92c84d96f)
  build user:       root@vm
  build date:       20261110-10:21:35
  go version:       go1.26.10
  platform:         linux/amd64
  tags:             netgo,builtinassets

Run it with a real configuration

A build that starts proves little, so I wrote a small configuration. It scrapes Prometheus itself and a second target on port 9999 where nothing listens, plus a recording rule and an alert rule for a down target. Here are the files as printed.

cd /tmp/run && cat prometheus.yml rules.yml
global:
  scrape_interval: 5s
  evaluation_interval: 5s
rule_files:
  - rules.yml
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: missing
    static_configs:
      - targets: ["localhost:9999"]
groups:
  - name: demo
    rules:
      - record: job:up:count
        expr: count by (job) (up)
      - alert: TargetDown
        expr: up == 0
        for: 10s
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} is down"

First I validated both files with the promtool binary that was built in the same step. It parses the main configuration, follows the rule_files entry, and checks the rules too, so a typo in either file would have been caught here before the server was ever started.

cd /tmp/run && /tmp/prometheus/promtool check config prometheus.yml
Checking prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax
Checking rules.yml
  SUCCESS: 2 rules found

Then I started the server from the root directory /, deliberately not from the repository, so that the web UI can only work if the assets are really compiled into the binary. The process was started in the background, and its log went to a file in the run directory.

cd / && nohup /tmp/prometheus/prometheus --config.file=/tmp/run/prometheus.yml --storage.tsdb.path=/tmp/run/data4 --web.listen-address=127.0.0.1:9090 > /tmp/run/prom4.log 2>&1 &

Verify with real requests

curl -s localhost:9090/-/ready
Prometheus Server is Ready.
curl -s -D - -o /dev/null localhost:9090/
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: /query
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" localhost:9090/query
200 text/html; charset=utf-8

The root redirects to the new UI and the UI page returns 200 from a process whose working directory is /, which proves that the assets are inside the binary and not read from the source tree. The Node 22 build also left a real JavaScript bundle on disk, which is what the page loads, and its size is shown here. I did not open a browser, so I make no claim about how the UI renders.

cd /tmp/prometheus && ls -l web/ui/static/mantine-ui/assets | grep "index-.*js$"
-rw-r--r-- 1 root root 1984743 Nov 10 10:21 index-B2YC7pDG.js
curl -s 'localhost:9090/api/v1/query?query=up'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1794306720.783,"1"]},{"metric":{"__name__":"up","instance":"localhost:9999","job":"missing"},"value":[1794306720.783,"0"]}]}}

The healthy self-scrape returns 1 and the unreachable target on port 9999 returns 0, which is what the up metric means. After the ten-second pending period that I configured with the for clause, the alert for the missing target was in the firing state, which I read from the alerts endpoint.

curl -s localhost:9090/api/v1/alerts
{"status":"success","data":{"alerts":[{"labels":{"alertname":"TargetDown","instance":"localhost:9999","job":"missing","severity":"page"},"annotations":{"summary":"missing is down"},"state":"firing","activeAt":"2026-11-10T10:31:42.506686049Z","value":"0e+00"}]}}

I also tested the alert rule without a running server, with a unit test file for promtool test rules. I wrote that file (rules_test.yml) in the sandbox. It feeds six zero samples for the up series of the missing target and expects the alert to fire at 30 seconds.

cd /tmp/run && /tmp/prometheus/promtool test rules rules_test.yml
Processing test file: rules_test.yml
  SUCCESS

The comparison: plain go build

To check the README's warning about go install, I built the server without the make step, which means without the builtinassets tag, and started it from /. The version output shows the tag as unknown. Then I asked the running process for the UI page and looked at the response code.

cd /tmp/prometheus && time go build -o /tmp/run/prom-nobuiltin ./cmd/prometheus; ls -la /tmp/run/prom-nobuiltin; /tmp/run/prom-nobuiltin --version | tail -2
  platform:         linux/amd64
  tags:             unknown
cd / && nohup /tmp/run/prom-nobuiltin --config.file=/tmp/run/prometheus.yml --storage.tsdb.path=/tmp/run/data5 --web.listen-address=127.0.0.1:9091 > /tmp/run/prom5.log 2>&1 &
curl -s -w "\n%{http_code}\n" localhost:9091/query
Error opening React index.html: open static/mantine-ui/index.html: no such file or directory
500

The API of that binary worked, but the UI route returned 500 because it looked for a relative static directory. I then started the same binary from the repository root, where my earlier make step had already produced web/ui/static, and asked for the same page.

cd /tmp/prometheus && nohup /tmp/run/prom-nobuiltin --config.file=/tmp/run/prometheus.yml --storage.tsdb.path=/tmp/run/data6 --web.listen-address=127.0.0.1:9092 > /tmp/run/prom6.log 2>&1 &
curl -s -o /dev/null -w "%{http_code}\n" localhost:9092/query
200

That matches the README's description exactly: the UI only works from the root of a clone that has built assets, so a plain go install binary is fine for the API and promtool, but it is the wrong choice if you want to copy one binary to another place and use the web UI.

Short review

What worked

The documented path, make build, worked on Go 1.24.7 and Node 22.22.0 without any change to the machine. The binary reported version 3.16.0 built with Go 1.26.10, loaded my configuration, scraped two targets, evaluated a recording rule, fired an alert and served the UI from a directory unrelated to the source tree. I also ran a subset of the unit tests, go test -short on the packages config, rules, web, promql/parser and model/labels. I counted the result lines in the log, first those starting with ok and then any containing FAIL.

cd /tmp/prometheus && (time go test -short ./config/... ./rules/... ./web/... ./promql/parser/... ./model/labels/...) > /tmp/test.log 2>&1; tail -20 /tmp/test.log
grep -c "^ok" /tmp/test.log; grep -c "FAIL" /tmp/test.log
6
0

Six packages reported ok and nothing failed. That is only a subset of the repository, not the whole suite, which I did not run, so read it as a smoke test of configuration loading, rules, web handlers and the query parser.

The limits

The Node warning is real: the project asks for 24.21.0 and I built the UI with 22.22.0. I verified the HTML page and the JavaScript bundle over HTTP, but I did not run the UI test suite or use a browser, so I cannot promise that the UI is identical to one built with the declared version. The build also needs internet access for the Go toolchain, modules and npm packages, and it took several minutes on four CPUs. I made all of these tests on one machine and one commit.

When to choose another method

If the build host has no internet access or sets GOTOOLCHAIN=local, install the required Go version yourself, as the failure above shows. If you need a UI bundle that matches the project's CI, install the Node version from .nvmrc. If you only need to run Prometheus and not modify it, a released binary or the container image from the README avoids the whole toolchain question, though I did not try either here.

Recent Searches