Compare commits

...

38 Commits

Author SHA1 Message Date
lneto ac57ed9709 Release 2024.12.2 2024-12-19 15:28:18 +00:00
Gonçalo Garcia c6901551e7 TUN-8822: Prevent concurrent usage of ICMPDecoder
## Summary
Some description...

Closes TUN-8822
2024-12-19 07:19:36 -08:00
Luis Neto 9bc6cbd06d TUN-8818: update changes document to reflect newly added diag subcommand
Closes TUN-8818
2024-12-18 04:26:38 -08:00
Devin Carr bc9c5d2e6e TUN-8817: Increase close session channel by one since there are two writers
When closing a session, there are two possible signals that will occur,
one from the outside, indicating that the session is idle and needs to
be closed, and the internal error condition that will be unblocked
with a net.ErrClosed when the connection underneath is closed. Both of
these routines write to the session's closeChan.

Once the reader for the closeChan reads one value, it will immediately
return. This means that the channel is a one-shot and one of the two
writers will get stuck unless the size of the channel is increased to
accomodate for the second write to the channel.

With the channel size increased to two, the second writer (whichever
loses the race to write) will now be unblocked to end their go routine
and return.

Closes TUN-8817
2024-12-17 14:55:09 -08:00
Luis Neto 1859d742a8 TUN-8724: Add CLI command for diagnostic procedure
## Summary
Adds a new CLI subcommand, under the tunnel command, the `diag`. This command has as function the automatic collection of different data points, such as, logs, metrics, network information, system information, tunnel state, and runtime information which will be written to a single zip file.

Closes TUN-8724
2024-12-13 10:07:56 -08:00
Luis Neto 8ed19222b9 TUN-8797: update CHANGES.md with note about semi-deterministic approach used to bind metrics server
Closes TUN-8797
2024-12-13 01:53:11 -08:00
Luis Neto 02e7ffd5b7 TUN-8792: Make diag/system endpoint always return a JSON
## Summary
Change the system information collector and respective http handler so that it always returns a JSON.

Closes [TUN-8792](https://jira.cfdata.org/browse/TUN-8792)
2024-12-11 02:48:41 -08:00
Luis Neto ba9f28ef43 TUN-8786: calculate cli flags once for the diagnostic procedure
## Summary

The flags were always being computed when their value is static.

 Closes TUN-8786
2024-12-11 01:29:20 -08:00
Luis Neto 77b99cf5fe TUN-8784: Set JSON encoder options to print formatted JSON when writing diag files
## Summary

The initial implementation produced correct JSON however it was not formatted which would make it harder to read the file by an user.

 Closes TUN-8784
2024-12-10 13:01:24 -08:00
Luis Neto d74ca97b51 TUN-8785: include the icmp sources in the diag's tunnel state
Closes TUN-8785
2024-12-10 10:42:33 -08:00
Luis Neto 29f0cf354c TUN-8783: fix log collectors for the diagnostic procedure
## Summary

* The host log collector now verifies if the OS is linux and has systemd if so it will use journalctl to get the logs
* In linux systems docker will write the output of the command logs to the stderr therefore the function that handles the execution of the process will copy both the contents of stdout and stderr; this also affect the k8s collector

Closes TUN-8783
2024-12-10 09:53:13 -08:00
lneto e7dcb6edca Release 2024.12.1 2024-12-10 16:07:33 +00:00
Luis Neto 14cf0eff1d TUN-8795: update createrepo to createrepo_c to fix the release_pkgs.py script
## Summary
The default-flavour of cfsetup changed from bullseye to bookworm and in the latter the createrepo package was renamed to createrepo_c.

Closes TUN-8795
2024-12-10 08:07:02 -08:00
lneto a00c80f9e1 Release 2024.12.0 2024-12-09 16:55:31 +00:00
Luis Neto 12d878531c TUN-8789: make python package installation consistent
## Summary

The previous changes regarding python's distribution which broke CI the installation of python packages.

Python packages in cfsetup are now installed via virtual environment. The dependency python3-venv was added as builddep to allow the creation of venv and the python packages installation was moved to the post-cache resulting in the removal of 

* anchor build_release_pre_cache
* anchor component_test_pre_cache

Closes TUN-8789
2024-12-09 08:52:48 -08:00
Devin Carr 588ab7ebaa TUN-8640: Add ICMP support for datagram V3
Closes TUN-8640
2024-12-09 07:23:11 -08:00
Joe Groocock dfbccd917c TUN-8781: Add Trixie, drop Buster. Default to Bookworm
Closes TUN-8781
2024-12-06 05:04:16 -08:00
Devin Carr 37010529bc TUN-8775: Make sure the session Close can only be called once
The previous capture of the sync.OnceValue was re-initialized for each
call to `Close`. This needed to be initialized during the creation of
the session to ensure that the sync.OnceValue reference was held for
the session's lifetime.

Closes TUN-8775
2024-12-05 14:12:53 -08:00
Luis Neto f07d04d129 TUN-8768: add job report to diagnostic zipfile
## Summary

Add a new job that write to a file the result of all of the other tasks along with possible errors. This file is also added to the root of the diagnostic zip file.

 Closes TUN-8768
2024-12-04 10:32:49 -08:00
Luis Neto f12036c2da TUN-8770: add cli configuration and tunnel configuration to diagnostic zipfile
## Summary

Adds two new jobs which will export the cli configuration and tunnel configuration in separate files. These files will also be added to the zipfile's root.

 Closes TUN-8770
2024-12-04 05:26:51 -08:00
Luis Neto 520e266411 TUN-8767: include raw output from network collector in diagnostic zipfile
## Summary

Export raw format of traceroute is widely known and useful for debugging. This raw output is written to the zipfile's root at the end of the diagnostic.

Closes TUN-8767
2024-12-04 04:40:51 -08:00
Luis Neto 7bd86762a7 TUN-8725: implement diagnostic procedure
## Summary
Add a function that orchestrates the diagnostic jobs producing a zip file at the end.

Closes TUN-8725
2024-12-04 03:37:57 -08:00
lneto 451f98e1d1 TUN-8727: extend client to include function to get cli configuration and tunnel configuration 2024-12-03 21:39:03 +00:00
Luis Neto 60fe4a0800 TUN-8769: fix k8s log collector arguments
## Summary

The equal signs were making the exec command to fail removing them fixes the issue.

Closes TUN-8769
2024-12-03 06:24:14 -08:00
Luis Neto 1ef109c042 TUN-8762: fix argument order when invoking tracert and modify network info output parsing.
## Summary

The windows code path has three bugs:

* the -4 and -6 option cannot be passed in the last position
* since there are some lines in the command output that are not parsable the collection fails to parse any kind of output
* the timeout hop is not correctly parsed

This PR also guards the parsing code against empty domains

Closes TUN-8762
2024-12-03 04:56:28 -08:00
Luis Neto 65786597cc TUN-8732: implement port selection algorithm
## Summary
Implements the discovery of the metrics server given an array of addresses (known addresses), tunnelID, and connectorID.

Closes TUN-8732
2024-12-03 04:07:55 -08:00
Luis Neto f884b29d0d TUN-8726: implement compression routine to be used in diagnostic procedure
## Summary
Basic routine to compress the diagnostic files into the root of a zipfile.

Closes TUN-8726
2024-12-03 03:27:04 -08:00
Luis Neto b3304bf05b TUN-8727: implement metrics, runtime, system, and tunnelstate in diagnostic http client
## Summary
The diagnostic procedure needs to extract information available in the metrics server via HTTP calls. 
These changes add to the diagnostic client the remaining endpoints.

Closes TUN-8727
2024-11-29 09:08:42 -08:00
Luis Neto 28796c659e TUN-8729: implement network collection for diagnostic procedure
## Summary
This PR adds implementation for windows & unix that collect the tracert.exe & traceroute output in the form of hops.

Closes TUN-8729
2024-11-29 07:43:36 -08:00
Devin Carr 9da15b5d96 TUN-8640: Refactor ICMPRouter to support new ICMPResponders
A new ICMPResponder interface is introduced to provide different
implementations of how the ICMP flows should return to the QUIC
connection muxer.

Improves usages of netip.AddrPort to leverage the embedded zone
field for IPv6 addresses.

Closes TUN-8640
2024-11-27 12:46:08 -08:00
Luis Neto 46dc6316f9 TUN-8734: add log collection for kubernetes
## Summary
Adds the log collector for K8s based deployments.

Closes TUN-8734
2024-11-27 03:15:15 -08:00
Luis Neto 16e65c70ad TUN-8733: add log collection for docker
## Summary
Adds the log collector for docker based deployments

Closes TUN-8733
2024-11-27 01:08:54 -08:00
Luis Neto a6f9e68739 TUN-8735: add managed/local log collection
## Summary
Adds a log collector for the managed/local runtimes.


Closes TUN-8735 TUN-8736
2024-11-26 10:30:44 -08:00
Luis Neto f85c0f1cc0 TUN-8730: implement diag/configuration
Implements the endpoint that retrieves the configuration of a running instance.

The configuration consists in a map of cli flag to the provided value along with the uid that of the user that started the process
2024-11-25 11:24:51 -08:00
Luis Neto 4b0b6dc8c6 TUN-8728: implement diag/tunnel endpoint
## Summary
The new endpoint returns the current information to be used when calling the diagnostic procedure.
This also adds:
- add indexed connection info and method to extract active connections from connTracker
- add edge address to Event struct and conn tracker
- remove unnecessary event send
- add tunnel configuration handler
- adjust cmd and metrics to create diagnostic server

Closes TUN-8728
2024-11-25 10:43:32 -08:00
Luis Neto aab5364252 TUN-8731: Implement diag/system endpoint
## Summary
This PR will add a new endpoint, "diag/system" to the metrics server that collects system information from different operating systems.

Closes TUN-8731
2024-11-22 08:10:05 -08:00
Luis Neto e2c2b012f1 TUN-8737: update metrics server port selection
## Summary
Update how metrics server binds to a listener by using a known set of ports whenever the default address is used with the fallback to a random port in case all address are already in use. The default address changes at compile time in order to bind to a different default address when the final deliverable is a docker image.

Refactor ReadyServer tests.

Closes TUN-8737
2024-11-22 07:23:46 -08:00
Devin Carr d779394748 TUN-8748: Migrated datagram V3 flows to use migrated context
Previously, during local flow migration the current connection context
was not part of the migration and would cause the flow to still be listening
on the connection context of the old connection (before the migration).
This meant that if a flow was migrated from connection 0 to
connection 1, and connection 0 goes away, the flow would be early
terminated incorrectly with the context lifetime of connection 0.

The new connection context is provided during migration of a flow
and will trigger the observe loop for the flow lifetime to be rebound
to this provided context.
Closes TUN-8748
2024-11-21 12:56:47 -08:00
70 changed files with 5689 additions and 545 deletions
+8
View File
@@ -1,3 +1,11 @@
## 2024.12.2
### New Features
- This release introduces the ability to collect troubleshooting information from one instance of cloudflared running on the local machine. The command can be executed as `cloudflared tunnel diag`.
## 2024.12.1
### Notices
- The use of the `--metrics` is still honoured meaning that if this flag is set the metrics server will try to bind it, however, this version includes a change that makes the metrics server bind to a port with a semi-deterministic approach. If the metrics flag is not present the server will bind to the first available port of the range 20241 to 20245. In case of all ports being unavailable then the fallback is to bind to a random port.
## 2024.10.0
### Bug Fixes
- We fixed a bug related to `--grace-period`. Tunnels that use QUIC as transport weren't abiding by this waiting period before forcefully closing the connections to the edge. From now on, both QUIC and HTTP2 tunnels will wait for either the grace period to end (defaults to 30 seconds) or until the last in-flight request is handled. Users that wish to maintain the previous behavior should set `--grace-period` to 0 if `--protocol` is set to `quic`. This will force `cloudflared` to shutdown as soon as either SIGTERM or SIGINT is received.
+3 -1
View File
@@ -5,7 +5,9 @@ FROM golang:1.22.5 as builder
ENV GO111MODULE=on \
CGO_ENABLED=0 \
TARGET_GOOS=${TARGET_GOOS} \
TARGET_GOARCH=${TARGET_GOARCH}
TARGET_GOARCH=${TARGET_GOARCH} \
CONTAINER_BUILD=1
WORKDIR /go/src/github.com/cloudflare/cloudflared/
+4
View File
@@ -30,6 +30,10 @@ ifdef PACKAGE_MANAGER
VERSION_FLAGS := $(VERSION_FLAGS) -X "github.com/cloudflare/cloudflared/cmd/cloudflared/updater.BuiltForPackageManager=$(PACKAGE_MANAGER)"
endif
ifdef CONTAINER_BUILD
VERSION_FLAGS := $(VERSION_FLAGS) -X "github.com/cloudflare/cloudflared/metrics.Runtime=virtual"
endif
LINK_FLAGS :=
ifeq ($(FIPS), true)
LINK_FLAGS := -linkmode=external -extldflags=-static $(LINK_FLAGS)
+41
View File
@@ -1,3 +1,44 @@
2024.12.2
- 2024-12-19 TUN-8822: Prevent concurrent usage of ICMPDecoder
- 2024-12-18 TUN-8818: update changes document to reflect newly added diag subcommand
- 2024-12-17 TUN-8817: Increase close session channel by one since there are two writers
- 2024-12-13 TUN-8797: update CHANGES.md with note about semi-deterministic approach used to bind metrics server
- 2024-12-13 TUN-8724: Add CLI command for diagnostic procedure
- 2024-12-11 TUN-8786: calculate cli flags once for the diagnostic procedure
- 2024-12-11 TUN-8792: Make diag/system endpoint always return a JSON
- 2024-12-10 TUN-8783: fix log collectors for the diagnostic procedure
- 2024-12-10 TUN-8785: include the icmp sources in the diag's tunnel state
- 2024-12-10 TUN-8784: Set JSON encoder options to print formatted JSON when writing diag files
2024.12.1
- 2024-12-10 TUN-8795: update createrepo to createrepo_c to fix the release_pkgs.py script
2024.12.0
- 2024-12-09 TUN-8640: Add ICMP support for datagram V3
- 2024-12-09 TUN-8789: make python package installation consistent
- 2024-12-06 TUN-8781: Add Trixie, drop Buster. Default to Bookworm
- 2024-12-05 TUN-8775: Make sure the session Close can only be called once
- 2024-12-04 TUN-8725: implement diagnostic procedure
- 2024-12-04 TUN-8767: include raw output from network collector in diagnostic zipfile
- 2024-12-04 TUN-8770: add cli configuration and tunnel configuration to diagnostic zipfile
- 2024-12-04 TUN-8768: add job report to diagnostic zipfile
- 2024-12-03 TUN-8726: implement compression routine to be used in diagnostic procedure
- 2024-12-03 TUN-8732: implement port selection algorithm
- 2024-12-03 TUN-8762: fix argument order when invoking tracert and modify network info output parsing.
- 2024-12-03 TUN-8769: fix k8s log collector arguments
- 2024-12-03 TUN-8727: extend client to include function to get cli configuration and tunnel configuration
- 2024-11-29 TUN-8729: implement network collection for diagnostic procedure
- 2024-11-29 TUN-8727: implement metrics, runtime, system, and tunnelstate in diagnostic http client
- 2024-11-27 TUN-8733: add log collection for docker
- 2024-11-27 TUN-8734: add log collection for kubernetes
- 2024-11-27 TUN-8640: Refactor ICMPRouter to support new ICMPResponders
- 2024-11-26 TUN-8735: add managed/local log collection
- 2024-11-25 TUN-8728: implement diag/tunnel endpoint
- 2024-11-25 TUN-8730: implement diag/configuration
- 2024-11-22 TUN-8737: update metrics server port selection
- 2024-11-22 TUN-8731: Implement diag/system endpoint
- 2024-11-21 TUN-8748: Migrated datagram V3 flows to use migrated context
2024.11.1
- 2024-11-18 Add cloudflared tunnel ready command
- 2024-11-14 Make metrics a requirement for tunnel ready command
+39 -34
View File
@@ -1,8 +1,9 @@
pinned_go: &pinned_go go-boring=1.22.5-1
build_dir: &build_dir /cfsetup_build
default-flavor: bullseye
buster: &buster
default-flavor: bookworm
bullseye: &bullseye
build-linux:
build_dir: *build_dir
builddeps: &build_deps
@@ -31,8 +32,8 @@ buster: &buster
builddeps: *build_deps
pre-cache: *build_pre_cache
post-cache:
- make cover
# except FIPS and macos
- make cover
# except FIPS and macos
build-linux-release:
build_dir: *build_dir
builddeps: &build_deps_release
@@ -46,19 +47,17 @@ buster: &buster
- python3-pip
- python3-setuptools
- wget
pre-cache: &build_release_pre_cache
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- pip3 install boto3==1.22.9
- pip3 install python-gnupg==0.4.9
- python3-venv
post-cache:
- python3 -m venv env
- . /cfsetup_build/env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55 boto3==1.22.9 python-gnupg==0.4.9
# build all packages (except macos and FIPS) and move them to /cfsetup/built_artifacts
- ./build-packages.sh
# handle FIPS separately so that we built with gofips compiler
build-linux-fips-release:
build_dir: *build_dir
builddeps: *build_deps_release
pre-cache: *build_release_pre_cache
post-cache:
# same logic as above, but for FIPS packages only
- ./build-packages-fips.sh
@@ -110,7 +109,7 @@ buster: &buster
- export GOOS=linux
- export GOARCH=arm64
- export NIGHTLY=true
#- export FIPS=true # TUN-7595
# - export FIPS=true # TUN-7595
- export ORIGINAL_NAME=true
- make cloudflared-deb
build-deb-arm64:
@@ -133,12 +132,14 @@ buster: &buster
# libmsi and libgcab are libraries the wixl binary depends on.
- libmsi-dev
- libgcab-dev
- python3-venv
pre-cache:
- wget https://github.com/sudarshan-reddy/msitools/releases/download/v0.101b/wixl -P /usr/local/bin
- chmod a+x /usr/local/bin/wixl
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- .teamcity/package-windows.sh
test:
build_dir: *build_dir
@@ -172,18 +173,22 @@ buster: &buster
build_dir: *build_dir
builddeps: &build_deps_component_test
- *pinned_go
- python3.7
- python3
- python3-pip
- python3-setuptools
# procps installs the ps command which is needed in test_sysv_service because the init script
# uses ps pid to determine if the agent is running
# procps installs the ps command which is needed in test_sysv_service
# because the init script uses ps pid to determine if the agent is
# running
- procps
- python3-venv
pre-cache-copy-paths:
- component-tests/requirements.txt
pre-cache: &component_test_pre_cache
- sudo pip3 install --upgrade -r component-tests/requirements.txt
post-cache: &component_test_post_cache
# Creates and routes a Named Tunnel for this build. Also constructs config file from env vars.
- python3 -m venv env
- . env/bin/activate
- pip install --upgrade -r component-tests/requirements.txt
# Creates and routes a Named Tunnel for this build. Also constructs
# config file from env vars.
- python3 component-tests/setup.py --type create
- pytest component-tests -o log_cli=true --log-cli-level=INFO
# The Named Tunnel is deleted and its route unprovisioned here.
@@ -193,7 +198,6 @@ buster: &buster
builddeps: *build_deps_component_test
pre-cache-copy-paths:
- component-tests/requirements.txt
pre-cache: *component_test_pre_cache
post-cache: *component_test_post_cache
github-release-dryrun:
build_dir: *build_dir
@@ -204,10 +208,11 @@ buster: &buster
- libffi-dev
- python3-setuptools
- python3-pip
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- make github-release-dryrun
github-release:
build_dir: *build_dir
@@ -218,10 +223,11 @@ buster: &buster
- libffi-dev
- python3-setuptools
- python3-pip
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- make github-release
r2-linux-release:
build_dir: *build_dir
@@ -237,14 +243,13 @@ buster: &buster
- python3-setuptools
- python3-pip
- reprepro
- createrepo
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- pip3 install boto3==1.22.9
- pip3 install python-gnupg==0.4.9
- createrepo-c
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55 boto3==1.22.9 python-gnupg==0.4.9
- make r2-linux-release
bullseye: *buster
bookworm: *buster
bookworm: *bullseye
trixie: *bullseye
+171 -7
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"net/url"
"os"
"path/filepath"
"runtime/trace"
"strings"
"sync"
@@ -28,6 +29,7 @@ import (
"github.com/cloudflare/cloudflared/config"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/credentials"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/edgediscovery"
"github.com/cloudflare/cloudflared/features"
"github.com/cloudflare/cloudflared/ingress"
@@ -39,6 +41,7 @@ import (
"github.com/cloudflare/cloudflared/supervisor"
"github.com/cloudflare/cloudflared/tlsconfig"
"github.com/cloudflare/cloudflared/tunneldns"
"github.com/cloudflare/cloudflared/tunnelstate"
"github.com/cloudflare/cloudflared/validation"
)
@@ -125,6 +128,94 @@ var (
"most likely you already have a conflicting record there. You can also rerun this command with --%s to overwrite "+
"any existing DNS records for this hostname.", overwriteDNSFlag)
deprecatedClassicTunnelErr = fmt.Errorf("Classic tunnels have been deprecated, please use Named Tunnels. (https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/)")
// TODO: TUN-8756 the list below denotes the flags that do not possess any kind of sensitive information
// however this approach is not maintainble in the long-term.
nonSecretFlagsList = []string{
"config",
"autoupdate-freq",
"no-autoupdate",
"metrics",
"pidfile",
"url",
"hello-world",
"socks5",
"proxy-connect-timeout",
"proxy-tls-timeout",
"proxy-tcp-keepalive",
"proxy-no-happy-eyeballs",
"proxy-keepalive-connections",
"proxy-keepalive-timeout",
"proxy-connection-timeout",
"proxy-expect-continue-timeout",
"http-host-header",
"origin-server-name",
"unix-socket",
"origin-ca-pool",
"no-tls-verify",
"no-chunked-encoding",
"http2-origin",
"management-hostname",
"service-op-ip",
"local-ssh-port",
"ssh-idle-timeout",
"ssh-max-timeout",
"bucket-name",
"region-name",
"s3-url-host",
"host-key-path",
"ssh-server",
"bastion",
"proxy-address",
"proxy-port",
"loglevel",
"transport-loglevel",
"logfile",
"log-directory",
"trace-output",
"proxy-dns",
"proxy-dns-port",
"proxy-dns-address",
"proxy-dns-upstream",
"proxy-dns-max-upstream-conns",
"proxy-dns-bootstrap",
"is-autoupdated",
"edge",
"region",
"edge-ip-version",
"edge-bind-address",
"cacert",
"hostname",
"id",
"lb-pool",
"api-url",
"metrics-update-freq",
"tag",
"heartbeat-interval",
"heartbeat-count",
"max-edge-addr-retries",
"retries",
"ha-connections",
"rpc-timeout",
"write-stream-timeout",
"quic-disable-pmtu-discovery",
"quic-connection-level-flow-control-limit",
"quic-stream-level-flow-control-limit",
"label",
"grace-period",
"compression-quality",
"use-reconnect-token",
"dial-edge-timeout",
"stdin-control",
"name",
"ui",
"quick-service",
"max-fetch-size",
"post-quantum",
"management-diagnostics",
"protocol",
"overwrite-dns",
"help",
}
)
func Flags() []cli.Flag {
@@ -145,6 +236,7 @@ func Commands() []*cli.Command {
buildDeleteCommand(),
buildCleanupCommand(),
buildTokenCommand(),
buildDiagCommand(),
// for compatibility, allow following as tunnel subcommands
proxydns.Command(true),
cliutil.RemovedCommand("db-connect"),
@@ -420,7 +512,7 @@ func StartServer(
// Disable ICMP packet routing for quick tunnels
if quickTunnelURL != "" {
tunnelConfig.PacketConfig = nil
tunnelConfig.ICMPRouterServer = nil
}
internalRules := []ingress.Rule{}
@@ -448,19 +540,42 @@ func StartServer(
return err
}
metricsListener, err := listeners.Listen("tcp", c.String("metrics"))
metricsListener, err := metrics.CreateMetricsListener(&listeners, c.String("metrics"))
if err != nil {
log.Err(err).Msg("Error opening metrics server listener")
return errors.Wrap(err, "Error opening metrics server listener")
}
defer metricsListener.Close()
wg.Add(1)
go func() {
defer wg.Done()
readinessServer := metrics.NewReadyServer(log, clientID)
observer.RegisterSink(readinessServer)
tracker := tunnelstate.NewConnTracker(log)
observer.RegisterSink(tracker)
ipv4, ipv6, err := determineICMPSources(c, log)
sources := make([]string, 0)
if err == nil {
sources = append(sources, ipv4.String())
sources = append(sources, ipv6.String())
}
readinessServer := metrics.NewReadyServer(clientID, tracker)
cliFlags := nonSecretCliFlags(log, c, nonSecretFlagsList)
diagnosticHandler := diagnostic.NewDiagnosticHandler(
log,
0,
diagnostic.NewSystemCollectorImpl(buildInfo.CloudflaredVersion),
tunnelConfig.NamedTunnel.Credentials.TunnelID,
clientID,
tracker,
cliFlags,
sources,
)
metricsConfig := metrics.Config{
ReadyServer: readinessServer,
DiagnosticHandler: diagnosticHandler,
QuickTunnelHostname: quickTunnelURL,
Orchestrator: orchestrator,
}
@@ -857,9 +972,15 @@ func configureCloudflaredFlags(shouldHide bool) []cli.Flag {
Hidden: shouldHide,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "metrics",
Value: "localhost:",
Usage: "Listen address for metrics reporting.",
Name: "metrics",
Value: metrics.GetMetricsDefaultAddress(metrics.Runtime),
Usage: fmt.Sprintf(
`Listen address for metrics reporting. If no address is passed cloudflared will try to bind to %v.
If all are unavailable, a random port will be used. Note that when running cloudflared from an virtual
environment the default address binds to all interfaces, hence, it is important to isolate the host
and virtualized host network stacks from each other`,
metrics.GetMetricsKnownAddresses(metrics.Runtime),
),
EnvVars: []string{"TUNNEL_METRICS"},
Hidden: shouldHide,
}),
@@ -1190,3 +1311,46 @@ reconnect [delay]
}
}
}
func nonSecretCliFlags(log *zerolog.Logger, cli *cli.Context, flagInclusionList []string) map[string]string {
flagsNames := cli.FlagNames()
flags := make(map[string]string, len(flagsNames))
for _, flag := range flagsNames {
value := cli.String(flag)
if value == "" {
continue
}
isIncluded := isFlagIncluded(flagInclusionList, flag)
if !isIncluded {
continue
}
switch flag {
case logger.LogDirectoryFlag, logger.LogFileFlag:
{
absolute, err := filepath.Abs(value)
if err != nil {
log.Error().Err(err).Msgf("could not convert %s path to absolute", flag)
} else {
flags[flag] = absolute
}
}
default:
flags[flag] = value
}
}
return flags
}
func isFlagIncluded(flagInclusionList []string, flag string) bool {
for _, include := range flagInclusionList {
if include == flag {
return true
}
}
return false
}
+23 -18
View File
@@ -252,11 +252,11 @@ func prepareTunnelConfig(
QUICConnectionLevelFlowControlLimit: c.Uint64(quicConnLevelFlowControlLimit),
QUICStreamLevelFlowControlLimit: c.Uint64(quicStreamLevelFlowControlLimit),
}
packetConfig, err := newPacketConfig(c, log)
icmpRouter, err := newICMPRouter(c, log)
if err != nil {
log.Warn().Err(err).Msg("ICMP proxy feature is disabled")
} else {
tunnelConfig.PacketConfig = packetConfig
tunnelConfig.ICMPRouterServer = icmpRouter
}
orchestratorConfig := &orchestration.Config{
Ingress: &ingressRules,
@@ -351,33 +351,39 @@ func adjustIPVersionByBindAddress(ipVersion allregions.ConfigIPVersion, ip net.I
}
}
func newPacketConfig(c *cli.Context, logger *zerolog.Logger) (*ingress.GlobalRouterConfig, error) {
func newICMPRouter(c *cli.Context, logger *zerolog.Logger) (ingress.ICMPRouterServer, error) {
ipv4Src, ipv6Src, err := determineICMPSources(c, logger)
if err != nil {
return nil, err
}
icmpRouter, err := ingress.NewICMPRouter(ipv4Src, ipv6Src, logger, icmpFunnelTimeout)
if err != nil {
return nil, err
}
return icmpRouter, nil
}
func determineICMPSources(c *cli.Context, logger *zerolog.Logger) (netip.Addr, netip.Addr, error) {
ipv4Src, err := determineICMPv4Src(c.String("icmpv4-src"), logger)
if err != nil {
return nil, errors.Wrap(err, "failed to determine IPv4 source address for ICMP proxy")
return netip.Addr{}, netip.Addr{}, errors.Wrap(err, "failed to determine IPv4 source address for ICMP proxy")
}
logger.Info().Msgf("ICMP proxy will use %s as source for IPv4", ipv4Src)
ipv6Src, zone, err := determineICMPv6Src(c.String("icmpv6-src"), logger, ipv4Src)
if err != nil {
return nil, errors.Wrap(err, "failed to determine IPv6 source address for ICMP proxy")
return netip.Addr{}, netip.Addr{}, errors.Wrap(err, "failed to determine IPv6 source address for ICMP proxy")
}
if zone != "" {
logger.Info().Msgf("ICMP proxy will use %s in zone %s as source for IPv6", ipv6Src, zone)
} else {
logger.Info().Msgf("ICMP proxy will use %s as source for IPv6", ipv6Src)
}
icmpRouter, err := ingress.NewICMPRouter(ipv4Src, ipv6Src, zone, logger, icmpFunnelTimeout)
if err != nil {
return nil, err
}
return &ingress.GlobalRouterConfig{
ICMPRouter: icmpRouter,
IPv4Src: ipv4Src,
IPv6Src: ipv6Src,
Zone: zone,
}, nil
return ipv4Src, ipv6Src, nil
}
func determineICMPv4Src(userDefinedSrc string, logger *zerolog.Logger) (netip.Addr, error) {
@@ -407,13 +413,12 @@ type interfaceIP struct {
func determineICMPv6Src(userDefinedSrc string, logger *zerolog.Logger, ipv4Src netip.Addr) (addr netip.Addr, zone string, err error) {
if userDefinedSrc != "" {
userDefinedIP, zone, _ := strings.Cut(userDefinedSrc, "%")
addr, err := netip.ParseAddr(userDefinedIP)
addr, err := netip.ParseAddr(userDefinedSrc)
if err != nil {
return netip.Addr{}, "", err
}
if addr.Is6() {
return addr, zone, nil
return addr, addr.Zone(), nil
}
return netip.Addr{}, "", fmt.Errorf("expect IPv6, but %s is IPv4", userDefinedSrc)
}
+136 -10
View File
@@ -28,16 +28,26 @@ import (
"github.com/cloudflare/cloudflared/cmd/cloudflared/updater"
"github.com/cloudflare/cloudflared/config"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/metrics"
)
const (
allSortByOptions = "name, id, createdAt, deletedAt, numConnections"
connsSortByOptions = "id, startedAt, numConnections, version"
CredFileFlagAlias = "cred-file"
CredFileFlag = "credentials-file"
CredContentsFlag = "credentials-contents"
TunnelTokenFlag = "token"
overwriteDNSFlagName = "overwrite-dns"
allSortByOptions = "name, id, createdAt, deletedAt, numConnections"
connsSortByOptions = "id, startedAt, numConnections, version"
CredFileFlagAlias = "cred-file"
CredFileFlag = "credentials-file"
CredContentsFlag = "credentials-contents"
TunnelTokenFlag = "token"
overwriteDNSFlagName = "overwrite-dns"
noDiagLogsFlagName = "no-diag-logs"
noDiagMetricsFlagName = "no-diag-metrics"
noDiagSystemFlagName = "no-diag-system"
noDiagRuntimeFlagName = "no-diag-runtime"
noDiagNetworkFlagName = "no-diag-network"
diagContainerIDFlagName = "diag-container-id"
diagPodFlagName = "diag-pod-id"
metricsFlagName = "metrics"
LogFieldTunnelID = "tunnelID"
)
@@ -179,6 +189,46 @@ var (
Usage: "Source address and the interface name to send/receive ICMPv6 messages. If not provided cloudflared will dial a local address to determine the source IP or fallback to ::.",
EnvVars: []string{"TUNNEL_ICMPV6_SRC"},
}
metricsFlag = &cli.StringFlag{
Name: metricsFlagName,
Usage: "The metrics server address i.e.: 127.0.0.1:12345. If your instance is running in a Docker/Kubernetes environment you need to setup port forwarding for your application.",
Value: "",
}
diagContainerFlag = &cli.StringFlag{
Name: diagContainerIDFlagName,
Usage: "Container ID or Name to collect logs from",
Value: "",
}
diagPodFlag = &cli.StringFlag{
Name: diagPodFlagName,
Usage: "Kubernetes POD to collect logs from",
Value: "",
}
noDiagLogsFlag = &cli.BoolFlag{
Name: noDiagLogsFlagName,
Usage: "Log collection will not be performed",
Value: false,
}
noDiagMetricsFlag = &cli.BoolFlag{
Name: noDiagMetricsFlagName,
Usage: "Metric collection will not be performed",
Value: false,
}
noDiagSystemFlag = &cli.BoolFlag{
Name: noDiagSystemFlagName,
Usage: "System information collection will not be performed",
Value: false,
}
noDiagRuntimeFlag = &cli.BoolFlag{
Name: noDiagRuntimeFlagName,
Usage: "Runtime information collection will not be performed",
Value: false,
}
noDiagNetworkFlag = &cli.BoolFlag{
Name: noDiagNetworkFlagName,
Usage: "Network diagnostics won't be performed",
Value: false,
}
)
func buildCreateCommand() *cli.Command {
@@ -375,7 +425,6 @@ func formatAndPrintTunnelList(tunnels []*cfapi.Tunnel, showRecentlyDisconnected
}
func fmtConnections(connections []cfapi.Connection, showRecentlyDisconnected bool) string {
// Count connections per colo
numConnsPerColo := make(map[string]uint, len(connections))
for _, connection := range connections {
@@ -897,8 +946,10 @@ func lbRouteFromArg(c *cli.Context) (cfapi.HostnameRoute, error) {
return cfapi.NewLBRoute(lbName, lbPool), nil
}
var nameRegex = regexp.MustCompile("^[_a-zA-Z0-9][-_.a-zA-Z0-9]*$")
var hostNameRegex = regexp.MustCompile("^[*_a-zA-Z0-9][-_.a-zA-Z0-9]*$")
var (
nameRegex = regexp.MustCompile("^[_a-zA-Z0-9][-_.a-zA-Z0-9]*$")
hostNameRegex = regexp.MustCompile("^[*_a-zA-Z0-9][-_.a-zA-Z0-9]*$")
)
func validateName(s string, allowWildcardSubdomain bool) bool {
if allowWildcardSubdomain {
@@ -986,3 +1037,78 @@ SUBCOMMAND OPTIONS:
`
return fmt.Sprintf(template, parentFlagsHelp)
}
func buildDiagCommand() *cli.Command {
return &cli.Command{
Name: "diag",
Action: cliutil.ConfiguredAction(diagCommand),
Usage: "Creates a diagnostic report from a local cloudflared instance",
UsageText: "cloudflared tunnel [tunnel command options] diag [subcommand options]",
Description: "cloudflared tunnel diag will create a diagnostic report of a local cloudflared instance. The diagnostic procedure collects: logs, metrics, system information, traceroute to Cloudflare Edge, and runtime information. Since there may be multiple instances of cloudflared running the --metrics option may be provided to target a specific instance.",
Flags: []cli.Flag{
metricsFlag,
diagContainerFlag,
diagPodFlag,
noDiagLogsFlag,
noDiagMetricsFlag,
noDiagSystemFlag,
noDiagRuntimeFlag,
noDiagNetworkFlag,
},
CustomHelpTemplate: commandHelpTemplate(),
}
}
func diagCommand(ctx *cli.Context) error {
sctx, err := newSubcommandContext(ctx)
if err != nil {
return err
}
log := sctx.log
options := diagnostic.Options{
KnownAddresses: metrics.GetMetricsKnownAddresses(metrics.Runtime),
Address: sctx.c.String(metricsFlagName),
ContainerID: sctx.c.String(diagContainerIDFlagName),
PodID: sctx.c.String(diagPodFlagName),
Toggles: diagnostic.Toggles{
NoDiagLogs: sctx.c.Bool(noDiagLogsFlagName),
NoDiagMetrics: sctx.c.Bool(noDiagMetricsFlagName),
NoDiagSystem: sctx.c.Bool(noDiagSystemFlagName),
NoDiagRuntime: sctx.c.Bool(noDiagRuntimeFlagName),
NoDiagNetwork: sctx.c.Bool(noDiagNetworkFlagName),
},
}
if options.Address == "" {
log.Info().Msg("If your instance is running in a Docker/Kubernetes environment you need to setup port forwarding for your application.")
}
states, err := diagnostic.RunDiagnostic(log, options)
if errors.Is(err, diagnostic.ErrMetricsServerNotFound) {
log.Warn().Msg("No instances found")
return nil
}
if errors.Is(err, diagnostic.ErrMultipleMetricsServerFound) {
if states != nil {
log.Info().Msgf("Found multiple instances running:")
for _, state := range states {
log.Info().Msgf("Instance: tunnel-id=%s connector-id=%s metrics-address=%s", state.TunnelID, state.ConnectorID, state.URL.String())
}
log.Info().Msgf("To select one instance use the option --metrics")
}
return nil
}
if errors.Is(err, diagnostic.ErrLogConfigurationIsInvalid) {
log.Info().Msg("Couldn't extract logs from the instance. If the instance is running in a containerized environment use the option --diag-container-id or --diag-pod-id. If there is no logging configuration use --no-diag-logs.")
}
if err != nil {
log.Warn().Msg("Diagnostic completed with one or more errors")
} else {
log.Info().Msg("Diagnostic completed")
}
return nil
}
+1 -1
View File
@@ -102,7 +102,7 @@ func (c *controlStream) ServeControlStream(
c.observer.metrics.regSuccess.WithLabelValues("registerConnection").Inc()
c.observer.logConnected(registrationDetails.UUID, c.connIndex, registrationDetails.Location, c.edgeAddress, c.protocol)
c.observer.sendConnectedEvent(c.connIndex, c.protocol, registrationDetails.Location)
c.observer.sendConnectedEvent(c.connIndex, c.protocol, registrationDetails.Location, c.edgeAddress)
c.connectedFuse.Connected()
// if conn index is 0 and tunnel is not remotely managed, then send local ingress rules configuration
+8 -5
View File
@@ -1,12 +1,15 @@
package connection
import "net"
// Event is something that happened to a connection, e.g. disconnection or registration.
type Event struct {
Index uint8
EventType Status
Location string
Protocol Protocol
URL string
Index uint8
EventType Status
Location string
Protocol Protocol
URL string
EdgeAddress net.IP
}
// Status is the status of a connection.
+2 -3
View File
@@ -47,7 +47,6 @@ func (o *Observer) RegisterSink(sink EventSink) {
}
func (o *Observer) logConnected(connectionID uuid.UUID, connIndex uint8, location string, address net.IP, protocol Protocol) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Location: location})
o.log.Info().
Int(management.EventTypeKey, int(management.Cloudflared)).
Str(LogFieldConnectionID, connectionID.String()).
@@ -63,8 +62,8 @@ func (o *Observer) sendRegisteringEvent(connIndex uint8) {
o.sendEvent(Event{Index: connIndex, EventType: RegisteringTunnel})
}
func (o *Observer) sendConnectedEvent(connIndex uint8, protocol Protocol, location string) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Protocol: protocol, Location: location})
func (o *Observer) sendConnectedEvent(connIndex uint8, protocol Protocol, location string, edgeAddress net.IP) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Protocol: protocol, Location: location, EdgeAddress: edgeAddress})
}
func (o *Observer) SendURL(url string) {
-27
View File
@@ -7,7 +7,6 @@ import (
"io"
"net"
"net/http"
"net/netip"
"strconv"
"strings"
"sync/atomic"
@@ -18,7 +17,6 @@ import (
"github.com/rs/zerolog"
"golang.org/x/sync/errgroup"
"github.com/cloudflare/cloudflared/packet"
cfdquic "github.com/cloudflare/cloudflared/quic"
"github.com/cloudflare/cloudflared/tracing"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
@@ -417,28 +415,3 @@ func (np *nopCloserReadWriter) Close() error {
return nil
}
// muxerWrapper wraps DatagramMuxerV2 to satisfy the packet.FunnelUniPipe interface
type muxerWrapper struct {
muxer *cfdquic.DatagramMuxerV2
}
func (rp *muxerWrapper) SendPacket(dst netip.Addr, pk packet.RawPacket) error {
return rp.muxer.SendPacket(cfdquic.RawPacket(pk))
}
func (rp *muxerWrapper) ReceivePacket(ctx context.Context) (packet.RawPacket, error) {
pk, err := rp.muxer.ReceivePacket(ctx)
if err != nil {
return packet.RawPacket{}, err
}
rawPacket, ok := pk.(cfdquic.RawPacket)
if ok {
return packet.RawPacket(rawPacket), nil
}
return packet.RawPacket{}, fmt.Errorf("unexpected packet type %+v", pk)
}
func (rp *muxerWrapper) Close() error {
return nil
}
+2 -1
View File
@@ -752,7 +752,8 @@ func testTunnelConnection(t *testing.T, serverAddr netip.AddrPort, index uint8)
sessionDemuxChan := make(chan *packet.Session, 4)
datagramMuxer := cfdquic.NewDatagramMuxerV2(conn, &log, sessionDemuxChan)
sessionManager := datagramsession.NewManager(&log, datagramMuxer.SendToSession, sessionDemuxChan)
packetRouter := ingress.NewPacketRouter(nil, datagramMuxer, &log)
var connIndex uint8 = 0
packetRouter := ingress.NewPacketRouter(nil, datagramMuxer, connIndex, &log)
datagramConn := &datagramV2Connection{
conn,
+3 -2
View File
@@ -54,7 +54,8 @@ type datagramV2Connection struct {
func NewDatagramV2Connection(ctx context.Context,
conn quic.Connection,
packetConfig *ingress.GlobalRouterConfig,
icmpRouter ingress.ICMPRouter,
index uint8,
rpcTimeout time.Duration,
streamWriteTimeout time.Duration,
logger *zerolog.Logger,
@@ -62,7 +63,7 @@ func NewDatagramV2Connection(ctx context.Context,
sessionDemuxChan := make(chan *packet.Session, demuxChanCapacity)
datagramMuxer := cfdquic.NewDatagramMuxerV2(conn, logger, sessionDemuxChan)
sessionManager := datagramsession.NewManager(logger, datagramMuxer.SendToSession, sessionDemuxChan)
packetRouter := ingress.NewPacketRouter(packetConfig, datagramMuxer, logger)
packetRouter := ingress.NewPacketRouter(icmpRouter, datagramMuxer, index, logger)
return &datagramV2Connection{
conn,
+3 -1
View File
@@ -10,6 +10,7 @@ import (
"github.com/quic-go/quic-go"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/management"
cfdquic "github.com/cloudflare/cloudflared/quic/v3"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
@@ -25,6 +26,7 @@ type datagramV3Connection struct {
func NewDatagramV3Connection(ctx context.Context,
conn quic.Connection,
sessionManager cfdquic.SessionManager,
icmpRouter ingress.ICMPRouter,
index uint8,
metrics cfdquic.Metrics,
logger *zerolog.Logger,
@@ -34,7 +36,7 @@ func NewDatagramV3Connection(ctx context.Context,
Int(management.EventTypeKey, int(management.UDP)).
Uint8(LogFieldConnIndex, index).
Logger()
datagramMuxer := cfdquic.NewDatagramConn(conn, sessionManager, index, metrics, &log)
datagramMuxer := cfdquic.NewDatagramConn(conn, sessionManager, icmpRouter, index, metrics, &log)
return &datagramV3Connection{
conn,
+216
View File
@@ -0,0 +1,216 @@
package diagnostic
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"github.com/cloudflare/cloudflared/logger"
)
type httpClient struct {
http.Client
baseURL *url.URL
}
func NewHTTPClient() *httpClient {
httpTransport := http.Transport{
TLSHandshakeTimeout: defaultTimeout,
ResponseHeaderTimeout: defaultTimeout,
}
return &httpClient{
http.Client{
Transport: &httpTransport,
Timeout: defaultTimeout,
},
nil,
}
}
func (client *httpClient) SetBaseURL(baseURL *url.URL) {
client.baseURL = baseURL
}
func (client *httpClient) GET(ctx context.Context, endpoint string) (*http.Response, error) {
if client.baseURL == nil {
return nil, ErrNoBaseURL
}
url := client.baseURL.JoinPath(endpoint)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url.String(), nil)
if err != nil {
return nil, fmt.Errorf("error creating GET request: %w", err)
}
req.Header.Add("Accept", "application/json;version=1")
response, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("error GET request: %w", err)
}
return response, nil
}
type LogConfiguration struct {
logFile string
logDirectory string
uid int // the uid of the user that started cloudflared
}
func (client *httpClient) GetLogConfiguration(ctx context.Context) (*LogConfiguration, error) {
response, err := client.GET(ctx, cliConfigurationEndpoint)
if err != nil {
return nil, err
}
defer response.Body.Close()
var data map[string]string
if err := json.NewDecoder(response.Body).Decode(&data); err != nil {
return nil, fmt.Errorf("failed to decode body: %w", err)
}
uidStr, exists := data[configurationKeyUID]
if !exists {
return nil, ErrKeyNotFound
}
uid, err := strconv.Atoi(uidStr)
if err != nil {
return nil, fmt.Errorf("error convertin pid to int: %w", err)
}
logFile, exists := data[logger.LogFileFlag]
if exists {
return &LogConfiguration{logFile, "", uid}, nil
}
logDirectory, exists := data[logger.LogDirectoryFlag]
if exists {
return &LogConfiguration{"", logDirectory, uid}, nil
}
// No log configured may happen when cloudflared is executed as a managed service or
// when containerized
return &LogConfiguration{"", "", uid}, nil
}
func (client *httpClient) GetMemoryDump(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, memoryDumpEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetGoroutineDump(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, goroutineDumpEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetTunnelState(ctx context.Context) (*TunnelState, error) {
response, err := client.GET(ctx, tunnelStateEndpoint)
if err != nil {
return nil, err
}
defer response.Body.Close()
var state TunnelState
if err := json.NewDecoder(response.Body).Decode(&state); err != nil {
return nil, fmt.Errorf("failed to decode body: %w", err)
}
return &state, nil
}
func (client *httpClient) GetSystemInformation(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, systemInformationEndpoint)
if err != nil {
return err
}
return copyJSONToWriter(response, writer)
}
func (client *httpClient) GetMetrics(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, metricsEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetTunnelConfiguration(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, tunnelConfigurationEndpoint)
if err != nil {
return err
}
return copyJSONToWriter(response, writer)
}
func (client *httpClient) GetCliConfiguration(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, cliConfigurationEndpoint)
if err != nil {
return err
}
return copyJSONToWriter(response, writer)
}
func copyToWriter(response *http.Response, writer io.Writer) error {
defer response.Body.Close()
_, err := io.Copy(writer, response.Body)
if err != nil {
return fmt.Errorf("error writing response: %w", err)
}
return nil
}
func copyJSONToWriter(response *http.Response, writer io.Writer) error {
defer response.Body.Close()
var data interface{}
decoder := json.NewDecoder(response.Body)
err := decoder.Decode(&data)
if err != nil {
return fmt.Errorf("diagnostic client error whilst reading response: %w", err)
}
encoder := newFormattedEncoder(writer)
err = encoder.Encode(data)
if err != nil {
return fmt.Errorf("diagnostic client error whilst writing json: %w", err)
}
return nil
}
type HTTPClient interface {
GetLogConfiguration(ctx context.Context) (*LogConfiguration, error)
GetMemoryDump(ctx context.Context, writer io.Writer) error
GetGoroutineDump(ctx context.Context, writer io.Writer) error
GetTunnelState(ctx context.Context) (*TunnelState, error)
GetSystemInformation(ctx context.Context, writer io.Writer) error
GetMetrics(ctx context.Context, writer io.Writer) error
GetCliConfiguration(ctx context.Context, writer io.Writer) error
GetTunnelConfiguration(ctx context.Context, writer io.Writer) error
}
+37
View File
@@ -0,0 +1,37 @@
package diagnostic
import "time"
const (
defaultCollectorTimeout = time.Second * 10 // This const define the timeout value of a collector operation.
collectorField = "collector" // used for logging purposes
systemCollectorName = "system" // used for logging purposes
tunnelStateCollectorName = "tunnelState" // used for logging purposes
configurationCollectorName = "configuration" // used for logging purposes
defaultTimeout = 15 * time.Second // timeout for the collectors
twoWeeksOffset = -14 * 24 * time.Hour // maximum offset for the logs
logFilename = "cloudflared_logs.txt" // name of the output log file
configurationKeyUID = "uid" // Key used to set and get the UID value from the configuration map
tailMaxNumberOfLines = "10000" // maximum number of log lines from a virtual runtime (docker or kubernetes)
// Endpoints used by the diagnostic HTTP Client.
cliConfigurationEndpoint = "/diag/configuration"
tunnelStateEndpoint = "/diag/tunnel"
systemInformationEndpoint = "/diag/system"
memoryDumpEndpoint = "debug/pprof/heap"
goroutineDumpEndpoint = "debug/pprof/goroutine"
metricsEndpoint = "metrics"
tunnelConfigurationEndpoint = "/config"
// Base for filenames of the diagnostic procedure
systemInformationBaseName = "systeminformation.json"
metricsBaseName = "metrics.txt"
zipName = "cloudflared-diag"
heapPprofBaseName = "heap.pprof"
goroutinePprofBaseName = "goroutine.pprof"
networkBaseName = "network.json"
rawNetworkBaseName = "raw-network.txt"
tunnelStateBaseName = "tunnelstate.json"
cliConfigurationBaseName = "cli-configuration.json"
configurationBaseName = "configuration.json"
taskResultBaseName = "task-result.json"
)
+561
View File
@@ -0,0 +1,561 @@
package diagnostic
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/rs/zerolog"
network "github.com/cloudflare/cloudflared/diagnostic/network"
)
const (
taskSuccess = "success"
taskFailure = "failure"
jobReportName = "job report"
tunnelStateJobName = "tunnel state"
systemInformationJobName = "system information"
goroutineJobName = "goroutine profile"
heapJobName = "heap profile"
metricsJobName = "metrics"
logInformationJobName = "log information"
rawNetworkInformationJobName = "raw network information"
networkInformationJobName = "network information"
cliConfigurationJobName = "cli configuration"
configurationJobName = "configuration"
)
// Struct used to hold the results of different routines executing the network collection.
type taskResult struct {
Result string `json:"result,omitempty"`
Err error `json:"error,omitempty"`
path string
}
func (result taskResult) MarshalJSON() ([]byte, error) {
s := map[string]string{
"result": result.Result,
}
if result.Err != nil {
s["error"] = result.Err.Error()
}
return json.Marshal(s)
}
// Struct used to hold the results of different routines executing the network collection.
type networkCollectionResult struct {
name string
info []*network.Hop
raw string
err error
}
// This type represents the most common functions from the diagnostic http client
// functions.
type collectToWriterFunc func(ctx context.Context, writer io.Writer) error
// This type represents the common denominator among all the collection procedures.
type collectFunc func(ctx context.Context) (string, error)
// collectJob is an internal struct that denotes holds the information necessary
// to run a collection job.
type collectJob struct {
jobName string
fn collectFunc
bypass bool
}
// The Toggles structure denotes the available toggles for the diagnostic procedure.
// Each toggle enables/disables tasks from the diagnostic.
type Toggles struct {
NoDiagLogs bool
NoDiagMetrics bool
NoDiagSystem bool
NoDiagRuntime bool
NoDiagNetwork bool
}
// The Options structure holds every option necessary for
// the diagnostic procedure to work.
type Options struct {
KnownAddresses []string
Address string
ContainerID string
PodID string
Toggles Toggles
}
func collectLogs(
ctx context.Context,
client HTTPClient,
diagContainer, diagPod string,
) (string, error) {
var collector LogCollector
if diagPod != "" {
collector = NewKubernetesLogCollector(diagContainer, diagPod)
} else if diagContainer != "" {
collector = NewDockerLogCollector(diagContainer)
} else {
collector = NewHostLogCollector(client)
}
logInformation, err := collector.Collect(ctx)
if err != nil {
return "", fmt.Errorf("error collecting logs: %w", err)
}
if logInformation.isDirectory {
return CopyFilesFromDirectory(logInformation.path)
}
if logInformation.wasCreated {
return logInformation.path, nil
}
logHandle, err := os.Open(logInformation.path)
if err != nil {
return "", fmt.Errorf("error opening log file while collecting logs: %w", err)
}
defer logHandle.Close()
outputLogHandle, err := os.Create(filepath.Join(os.TempDir(), logFilename))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer outputLogHandle.Close()
_, err = io.Copy(outputLogHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying logs while collecting logs: %w", err)
}
return outputLogHandle.Name(), err
}
func collectNetworkResultRoutine(
ctx context.Context,
collector network.NetworkCollector,
hostname string,
useIPv4 bool,
results chan networkCollectionResult,
) {
const (
hopsNo = 5
timeout = time.Second * 5
)
name := hostname
if useIPv4 {
name += "-v4"
} else {
name += "-v6"
}
hops, raw, err := collector.Collect(ctx, network.NewTraceOptions(hopsNo, timeout, hostname, useIPv4))
results <- networkCollectionResult{name, hops, raw, err}
}
func gatherNetworkInformation(ctx context.Context) map[string]networkCollectionResult {
networkCollector := network.NetworkCollectorImpl{}
hostAndIPversionPairs := []struct {
host string
useV4 bool
}{
{"region1.v2.argotunnel.com", true},
{"region1.v2.argotunnel.com", false},
{"region2.v2.argotunnel.com", true},
{"region2.v2.argotunnel.com", false},
}
// the number of results is known thus use len to avoid footguns
results := make(chan networkCollectionResult, len(hostAndIPversionPairs))
var wgroup sync.WaitGroup
for _, item := range hostAndIPversionPairs {
wgroup.Add(1)
go func() {
defer wgroup.Done()
collectNetworkResultRoutine(ctx, &networkCollector, item.host, item.useV4, results)
}()
}
// Wait for routines to end.
wgroup.Wait()
resultMap := make(map[string]networkCollectionResult)
for range len(hostAndIPversionPairs) {
result := <-results
resultMap[result.name] = result
}
return resultMap
}
func networkInformationCollectors() (rawNetworkCollector, jsonNetworkCollector collectFunc) {
// The network collector is an operation that takes most of the diagnostic time, thus,
// the sync.Once is used to memoize the result of the collector and then create different
// outputs.
var once sync.Once
var resultMap map[string]networkCollectionResult
rawNetworkCollector = func(ctx context.Context) (string, error) {
once.Do(func() { resultMap = gatherNetworkInformation(ctx) })
return rawNetworkInformationWriter(resultMap)
}
jsonNetworkCollector = func(ctx context.Context) (string, error) {
once.Do(func() { resultMap = gatherNetworkInformation(ctx) })
return jsonNetworkInformationWriter(resultMap)
}
return rawNetworkCollector, jsonNetworkCollector
}
func rawNetworkInformationWriter(resultMap map[string]networkCollectionResult) (string, error) {
networkDumpHandle, err := os.Create(filepath.Join(os.TempDir(), rawNetworkBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer networkDumpHandle.Close()
var exitErr error
for k, v := range resultMap {
if v.err != nil {
if exitErr == nil {
exitErr = v.err
}
_, err := networkDumpHandle.WriteString(k + "\nno content\n")
if err != nil {
return networkDumpHandle.Name(), fmt.Errorf("error writing 'no content' to raw network file: %w", err)
}
} else {
_, err := networkDumpHandle.WriteString(k + "\n" + v.raw + "\n")
if err != nil {
return networkDumpHandle.Name(), fmt.Errorf("error writing raw network information: %w", err)
}
}
}
return networkDumpHandle.Name(), exitErr
}
func jsonNetworkInformationWriter(resultMap map[string]networkCollectionResult) (string, error) {
networkDumpHandle, err := os.Create(filepath.Join(os.TempDir(), networkBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer networkDumpHandle.Close()
encoder := newFormattedEncoder(networkDumpHandle)
var exitErr error
jsonMap := make(map[string][]*network.Hop, len(resultMap))
for k, v := range resultMap {
jsonMap[k] = v.info
if exitErr == nil && v.err != nil {
exitErr = v.err
}
}
err = encoder.Encode(jsonMap)
if err != nil {
return networkDumpHandle.Name(), fmt.Errorf("error encoding network information results: %w", err)
}
return networkDumpHandle.Name(), exitErr
}
func collectFromEndpointAdapter(collect collectToWriterFunc, fileName string) collectFunc {
return func(ctx context.Context) (string, error) {
dumpHandle, err := os.Create(filepath.Join(os.TempDir(), fileName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer dumpHandle.Close()
err = collect(ctx, dumpHandle)
if err != nil {
return dumpHandle.Name(), fmt.Errorf("error running collector: %w", err)
}
return dumpHandle.Name(), nil
}
}
func tunnelStateCollectEndpointAdapter(client HTTPClient, tunnel *TunnelState, fileName string) collectFunc {
endpointFunc := func(ctx context.Context, writer io.Writer) error {
if tunnel == nil {
// When the metrics server is not passed the diagnostic will query all known hosts
// and get the tunnel state, however, when the metrics server is passed that won't
// happen hence the check for nil in this function.
tunnelResponse, err := client.GetTunnelState(ctx)
if err != nil {
return fmt.Errorf("error retrieving tunnel state: %w", err)
}
tunnel = tunnelResponse
}
encoder := newFormattedEncoder(writer)
err := encoder.Encode(tunnel)
if err != nil {
return fmt.Errorf("error encoding tunnel state: %w", err)
}
return nil
}
return collectFromEndpointAdapter(endpointFunc, fileName)
}
// resolveInstanceBaseURL is responsible to
// resolve the base URL of the instance that should be diagnosed.
// To resolve the instance it may be necessary to query the
// /diag/tunnel endpoint of the known instances, thus, if a single
// instance is found its state is also returned; if multiple instances
// are found then their states are returned in an array along with an
// error.
func resolveInstanceBaseURL(
metricsServerAddress string,
log *zerolog.Logger,
client *httpClient,
addresses []string,
) (*url.URL, *TunnelState, []*AddressableTunnelState, error) {
if metricsServerAddress != "" {
if !strings.HasPrefix(metricsServerAddress, "http://") {
metricsServerAddress = "http://" + metricsServerAddress
}
url, err := url.Parse(metricsServerAddress)
if err != nil {
return nil, nil, nil, fmt.Errorf("provided address is not valid: %w", err)
}
return url, nil, nil, nil
}
tunnelState, foundTunnelStates, err := FindMetricsServer(log, client, addresses)
if err != nil {
return nil, nil, foundTunnelStates, err
}
return tunnelState.URL, tunnelState.TunnelState, nil, nil
}
func createJobs(
client *httpClient,
tunnel *TunnelState,
diagContainer string,
diagPod string,
noDiagSystem bool,
noDiagRuntime bool,
noDiagMetrics bool,
noDiagLogs bool,
noDiagNetwork bool,
) []collectJob {
rawNetworkCollectorFunc, jsonNetworkCollectorFunc := networkInformationCollectors()
jobs := []collectJob{
{
jobName: tunnelStateJobName,
fn: tunnelStateCollectEndpointAdapter(client, tunnel, tunnelStateBaseName),
bypass: false,
},
{
jobName: systemInformationJobName,
fn: collectFromEndpointAdapter(client.GetSystemInformation, systemInformationBaseName),
bypass: noDiagSystem,
},
{
jobName: goroutineJobName,
fn: collectFromEndpointAdapter(client.GetGoroutineDump, goroutinePprofBaseName),
bypass: noDiagRuntime,
},
{
jobName: heapJobName,
fn: collectFromEndpointAdapter(client.GetMemoryDump, heapPprofBaseName),
bypass: noDiagRuntime,
},
{
jobName: metricsJobName,
fn: collectFromEndpointAdapter(client.GetMetrics, metricsBaseName),
bypass: noDiagMetrics,
},
{
jobName: logInformationJobName,
fn: func(ctx context.Context) (string, error) {
return collectLogs(ctx, client, diagContainer, diagPod)
},
bypass: noDiagLogs,
},
{
jobName: rawNetworkInformationJobName,
fn: rawNetworkCollectorFunc,
bypass: noDiagNetwork,
},
{
jobName: networkInformationJobName,
fn: jsonNetworkCollectorFunc,
bypass: noDiagNetwork,
},
{
jobName: cliConfigurationJobName,
fn: collectFromEndpointAdapter(client.GetCliConfiguration, cliConfigurationBaseName),
bypass: false,
},
{
jobName: configurationJobName,
fn: collectFromEndpointAdapter(client.GetTunnelConfiguration, configurationBaseName),
bypass: false,
},
}
return jobs
}
func createTaskReport(taskReport map[string]taskResult) (string, error) {
dumpHandle, err := os.Create(filepath.Join(os.TempDir(), taskResultBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer dumpHandle.Close()
encoder := newFormattedEncoder(dumpHandle)
err = encoder.Encode(taskReport)
if err != nil {
return "", fmt.Errorf("error encoding task results: %w", err)
}
return dumpHandle.Name(), nil
}
func runJobs(ctx context.Context, jobs []collectJob, log *zerolog.Logger) map[string]taskResult {
jobReport := make(map[string]taskResult, len(jobs))
for _, job := range jobs {
if job.bypass {
continue
}
log.Info().Msgf("Collecting %s...", job.jobName)
path, err := job.fn(ctx)
var result taskResult
if err != nil {
result = taskResult{Result: taskFailure, Err: err, path: path}
log.Error().Err(err).Msgf("Job: %s finished with error.", job.jobName)
} else {
result = taskResult{Result: taskSuccess, Err: nil, path: path}
log.Info().Msgf("Collected %s.", job.jobName)
}
jobReport[job.jobName] = result
}
taskReportName, err := createTaskReport(jobReport)
var result taskResult
if err != nil {
result = taskResult{
Result: taskFailure,
path: taskReportName,
Err: err,
}
} else {
result = taskResult{
Result: taskSuccess,
path: taskReportName,
Err: nil,
}
}
jobReport[jobReportName] = result
return jobReport
}
func RunDiagnostic(
log *zerolog.Logger,
options Options,
) ([]*AddressableTunnelState, error) {
client := NewHTTPClient()
baseURL, tunnel, foundTunnels, err := resolveInstanceBaseURL(options.Address, log, client, options.KnownAddresses)
if err != nil {
return foundTunnels, err
}
log.Info().Msgf("Selected server %s starting diagnostic...", baseURL.String())
client.SetBaseURL(baseURL)
const timeout = 45 * time.Second
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
jobs := createJobs(
client,
tunnel,
options.ContainerID,
options.PodID,
options.Toggles.NoDiagSystem,
options.Toggles.NoDiagRuntime,
options.Toggles.NoDiagMetrics,
options.Toggles.NoDiagLogs,
options.Toggles.NoDiagNetwork,
)
jobsReport := runJobs(ctx, jobs, log)
paths := make([]string, 0)
var gerr error
for _, v := range jobsReport {
paths = append(paths, v.path)
if gerr == nil && v.Err != nil {
gerr = v.Err
}
defer func() {
if !errors.Is(v.Err, ErrCreatingTemporaryFile) {
os.Remove(v.path)
}
}()
}
zipfile, err := CreateDiagnosticZipFile(zipName, paths)
if err != nil {
return nil, err
}
log.Info().Msgf("Diagnostic file written: %v", zipfile)
return nil, gerr
}
+148
View File
@@ -0,0 +1,148 @@
package diagnostic
import (
"archive/zip"
"context"
"encoding/json"
"fmt"
"io"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"github.com/google/uuid"
"github.com/rs/zerolog"
)
// CreateDiagnosticZipFile create a zip file with the contents from the all
// files paths. The files will be written in the root of the zip file.
// In case of an error occurs after whilst writing to the zip file
// this will be removed.
func CreateDiagnosticZipFile(base string, paths []string) (zipFileName string, err error) {
// Create a zip file with all files from paths added to the root
suffix := time.Now().Format(time.RFC3339)
zipFileName = base + "-" + suffix + ".zip"
zipFileName = strings.ReplaceAll(zipFileName, ":", "-")
archive, cerr := os.Create(zipFileName)
if cerr != nil {
return "", fmt.Errorf("error creating file %s: %w", zipFileName, cerr)
}
archiveWriter := zip.NewWriter(archive)
defer func() {
archiveWriter.Close()
archive.Close()
if err != nil {
os.Remove(zipFileName)
}
}()
for _, file := range paths {
if file == "" {
continue
}
var handle *os.File
handle, err = os.Open(file)
if err != nil {
return "", fmt.Errorf("error opening file %s: %w", zipFileName, err)
}
defer handle.Close()
// Keep the base only to not create sub directories in the
// zip file.
var writer io.Writer
writer, err = archiveWriter.Create(filepath.Base(file))
if err != nil {
return "", fmt.Errorf("error creating archive writer from %s: %w", file, err)
}
if _, err = io.Copy(writer, handle); err != nil {
return "", fmt.Errorf("error copying file %s: %w", file, err)
}
}
zipFileName = archive.Name()
return zipFileName, nil
}
type AddressableTunnelState struct {
*TunnelState
URL *url.URL
}
func findMetricsServerPredicate(tunnelID, connectorID uuid.UUID) func(state *TunnelState) bool {
if tunnelID != uuid.Nil && connectorID != uuid.Nil {
return func(state *TunnelState) bool {
return state.ConnectorID == connectorID && state.TunnelID == tunnelID
}
} else if tunnelID == uuid.Nil && connectorID != uuid.Nil {
return func(state *TunnelState) bool {
return state.ConnectorID == connectorID
}
} else if tunnelID != uuid.Nil && connectorID == uuid.Nil {
return func(state *TunnelState) bool {
return state.TunnelID == tunnelID
}
}
return func(*TunnelState) bool {
return true
}
}
// The FindMetricsServer will try to find the metrics server url.
// There are two possible error scenarios:
// 1. No instance is found which will only return ErrMetricsServerNotFound
// 2. Multiple instances are found which will return an array of state and ErrMultipleMetricsServerFound
// In case of success, only the state for the instance is returned.
func FindMetricsServer(
log *zerolog.Logger,
client *httpClient,
addresses []string,
) (*AddressableTunnelState, []*AddressableTunnelState, error) {
instances := make([]*AddressableTunnelState, 0)
for _, address := range addresses {
url, err := url.Parse("http://" + address)
if err != nil {
log.Debug().Err(err).Msgf("error parsing address %s", address)
continue
}
client.SetBaseURL(url)
state, err := client.GetTunnelState(context.Background())
if err == nil {
instances = append(instances, &AddressableTunnelState{state, url})
} else {
log.Debug().Err(err).Msgf("error getting tunnel state from address %s", address)
}
}
if len(instances) == 0 {
return nil, nil, ErrMetricsServerNotFound
}
if len(instances) == 1 {
return instances[0], nil, nil
}
return nil, instances, ErrMultipleMetricsServerFound
}
// newFormattedEncoder return a JSON encoder with identation
func newFormattedEncoder(w io.Writer) *json.Encoder {
encoder := json.NewEncoder(w)
encoder.SetIndent("", " ")
return encoder
}
+147
View File
@@ -0,0 +1,147 @@
package diagnostic_test
import (
"context"
"net/http"
"net/url"
"sync"
"testing"
"time"
"github.com/facebookgo/grace/gracenet"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/metrics"
"github.com/cloudflare/cloudflared/tunnelstate"
)
func helperCreateServer(t *testing.T, listeners *gracenet.Net, tunnelID uuid.UUID, connectorID uuid.UUID) func() {
t.Helper()
listener, err := metrics.CreateMetricsListener(listeners, "localhost:0")
require.NoError(t, err)
log := zerolog.Nop()
tracker := tunnelstate.NewConnTracker(&log)
handler := diagnostic.NewDiagnosticHandler(&log, 0, nil, tunnelID, connectorID, tracker, map[string]string{}, []string{})
router := http.NewServeMux()
router.HandleFunc("/diag/tunnel", handler.TunnelStateHandler)
server := &http.Server{
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
Handler: router,
}
var wgroup sync.WaitGroup
wgroup.Add(1)
go func() {
defer wgroup.Done()
_ = server.Serve(listener)
}()
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
cleanUp := func() {
_ = server.Shutdown(ctx)
cancel()
wgroup.Wait()
}
return cleanUp
}
func TestFindMetricsServer_WhenSingleServerIsRunning_ReturnState(t *testing.T) {
listeners := gracenet.Net{}
tid1 := uuid.New()
cid1 := uuid.New()
cleanUp := helperCreateServer(t, &listeners, tid1, cid1)
defer cleanUp()
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
url1, err := url.Parse("http://localhost:20241")
require.NoError(t, err)
tunnel1 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid1,
Connections: nil,
},
URL: url1,
}
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
if err != nil {
require.ErrorIs(t, err, diagnostic.ErrMultipleMetricsServerFound)
}
assert.Equal(t, tunnel1, state)
assert.Nil(t, tunnels)
}
func TestFindMetricsServer_WhenMultipleServerAreRunning_ReturnError(t *testing.T) {
listeners := gracenet.Net{}
tid1 := uuid.New()
cid1 := uuid.New()
cid2 := uuid.New()
cleanUp := helperCreateServer(t, &listeners, tid1, cid1)
defer cleanUp()
cleanUp = helperCreateServer(t, &listeners, tid1, cid2)
defer cleanUp()
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
url1, err := url.Parse("http://localhost:20241")
require.NoError(t, err)
url2, err := url.Parse("http://localhost:20242")
require.NoError(t, err)
tunnel1 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid1,
Connections: nil,
},
URL: url1,
}
tunnel2 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid2,
Connections: nil,
},
URL: url2,
}
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
if err != nil {
require.ErrorIs(t, err, diagnostic.ErrMultipleMetricsServerFound)
}
assert.Nil(t, state)
assert.Equal(t, []*diagnostic.AddressableTunnelState{tunnel1, tunnel2}, tunnels)
}
func TestFindMetricsServer_WhenNoInstanceIsRuning_ReturnError(t *testing.T) {
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
require.ErrorIs(t, err, diagnostic.ErrMetricsServerNotFound)
assert.Nil(t, state)
assert.Nil(t, tunnels)
}
+28
View File
@@ -0,0 +1,28 @@
package diagnostic
import (
"errors"
)
var (
// Error used when there is no log directory available.
ErrManagedLogNotFound = errors.New("managed log directory not found")
// Error used when it is not possible to collect logs using the log configuration.
ErrLogConfigurationIsInvalid = errors.New("provided log configuration is invalid")
// Error used when parsing the fields of the output of collector.
ErrInsufficientLines = errors.New("insufficient lines")
// Error used when parsing the lines of the output of collector.
ErrInsuficientFields = errors.New("insufficient fields")
// Error used when given key is not found while parsing KV.
ErrKeyNotFound = errors.New("key not found")
// Error used when there is no disk volume information available.
ErrNoVolumeFound = errors.New("no disk volume information found")
// Error user when the base url of the diagnostic client is not provided.
ErrNoBaseURL = errors.New("no base url")
// Error used when no metrics server is found listening to the known addresses list (check [metrics.GetMetricsKnownAddresses]).
ErrMetricsServerNotFound = errors.New("metrics server not found")
// Error used when multiple metrics server are found listening to the known addresses list (check [metrics.GetMetricsKnownAddresses]).
ErrMultipleMetricsServerFound = errors.New("multiple metrics server found")
// Error used when a temporary file creation fails within the diagnostic procedure
ErrCreatingTemporaryFile = errors.New("temporary file creation failed")
)
+144
View File
@@ -0,0 +1,144 @@
package diagnostic
import (
"context"
"encoding/json"
"net/http"
"os"
"strconv"
"time"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/tunnelstate"
)
type Handler struct {
log *zerolog.Logger
timeout time.Duration
systemCollector SystemCollector
tunnelID uuid.UUID
connectorID uuid.UUID
tracker *tunnelstate.ConnTracker
cliFlags map[string]string
icmpSources []string
}
func NewDiagnosticHandler(
log *zerolog.Logger,
timeout time.Duration,
systemCollector SystemCollector,
tunnelID uuid.UUID,
connectorID uuid.UUID,
tracker *tunnelstate.ConnTracker,
cliFlags map[string]string,
icmpSources []string,
) *Handler {
logger := log.With().Logger()
if timeout == 0 {
timeout = defaultCollectorTimeout
}
cliFlags[configurationKeyUID] = strconv.Itoa(os.Getuid())
return &Handler{
log: &logger,
timeout: timeout,
systemCollector: systemCollector,
tunnelID: tunnelID,
connectorID: connectorID,
tracker: tracker,
cliFlags: cliFlags,
icmpSources: icmpSources,
}
}
func (handler *Handler) InstallEndpoints(router *http.ServeMux) {
router.HandleFunc(cliConfigurationEndpoint, handler.ConfigurationHandler)
router.HandleFunc(tunnelStateEndpoint, handler.TunnelStateHandler)
router.HandleFunc(systemInformationEndpoint, handler.SystemHandler)
}
type SystemInformationResponse struct {
Info *SystemInformation `json:"info"`
Err error `json:"errors"`
}
func (handler *Handler) SystemHandler(writer http.ResponseWriter, request *http.Request) {
logger := handler.log.With().Str(collectorField, systemCollectorName).Logger()
logger.Info().Msg("Collection started")
defer logger.Info().Msg("Collection finished")
ctx, cancel := context.WithTimeout(request.Context(), handler.timeout)
defer cancel()
info, err := handler.systemCollector.Collect(ctx)
response := SystemInformationResponse{
Info: info,
Err: err,
}
encoder := json.NewEncoder(writer)
err = encoder.Encode(response)
if err != nil {
logger.Error().Err(err).Msgf("error occurred whilst serializing information")
writer.WriteHeader(http.StatusInternalServerError)
}
}
type TunnelState struct {
TunnelID uuid.UUID `json:"tunnelID,omitempty"`
ConnectorID uuid.UUID `json:"connectorID,omitempty"`
Connections []tunnelstate.IndexedConnectionInfo `json:"connections,omitempty"`
ICMPSources []string `json:"icmp_sources,omitempty"`
}
func (handler *Handler) TunnelStateHandler(writer http.ResponseWriter, _ *http.Request) {
log := handler.log.With().Str(collectorField, tunnelStateCollectorName).Logger()
log.Info().Msg("Collection started")
defer log.Info().Msg("Collection finished")
body := TunnelState{
handler.tunnelID,
handler.connectorID,
handler.tracker.GetActiveConnections(),
handler.icmpSources,
}
encoder := json.NewEncoder(writer)
err := encoder.Encode(body)
if err != nil {
handler.log.Error().Err(err).Msgf("error occurred whilst serializing information")
writer.WriteHeader(http.StatusInternalServerError)
}
}
func (handler *Handler) ConfigurationHandler(writer http.ResponseWriter, _ *http.Request) {
log := handler.log.With().Str(collectorField, configurationCollectorName).Logger()
log.Info().Msg("Collection started")
defer func() {
log.Info().Msg("Collection finished")
}()
encoder := json.NewEncoder(writer)
err := encoder.Encode(handler.cliFlags)
if err != nil {
handler.log.Error().Err(err).Msgf("error occurred whilst serializing response")
writer.WriteHeader(http.StatusInternalServerError)
}
}
func writeResponse(w http.ResponseWriter, bytes []byte, logger *zerolog.Logger) {
bytesWritten, err := w.Write(bytes)
if err != nil {
logger.Error().Err(err).Msg("error occurred writing response")
} else if bytesWritten != len(bytes) {
logger.Error().Msgf("error incomplete write response %d/%d", bytesWritten, len(bytes))
}
}
+224
View File
@@ -0,0 +1,224 @@
package diagnostic_test
import (
"context"
"encoding/json"
"errors"
"net"
"net/http"
"net/http/httptest"
"runtime"
"testing"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/tunnelstate"
)
type SystemCollectorMock struct {
systemInfo *diagnostic.SystemInformation
err error
}
const (
systemInformationKey = "sikey"
errorKey = "errkey"
)
func newTrackerFromConns(t *testing.T, connections []tunnelstate.IndexedConnectionInfo) *tunnelstate.ConnTracker {
t.Helper()
log := zerolog.Nop()
tracker := tunnelstate.NewConnTracker(&log)
for _, conn := range connections {
tracker.OnTunnelEvent(connection.Event{
Index: conn.Index,
EventType: connection.Connected,
Protocol: conn.Protocol,
EdgeAddress: conn.EdgeAddress,
})
}
return tracker
}
func (collector *SystemCollectorMock) Collect(context.Context) (*diagnostic.SystemInformation, error) {
return collector.systemInfo, collector.err
}
func TestSystemHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
systemInfo *diagnostic.SystemInformation
err error
statusCode int
}{
{
name: "happy path",
systemInfo: diagnostic.NewSystemInformation(
0, 0, 0, 0,
"string", "string", "string", "string",
"string", "string",
runtime.Version(), runtime.GOARCH, nil,
),
err: nil,
statusCode: http.StatusOK,
},
{
name: "on error and no raw info", systemInfo: nil,
err: errors.New("an error"), statusCode: http.StatusOK,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
handler := diagnostic.NewDiagnosticHandler(&log, 0, &SystemCollectorMock{
systemInfo: tCase.systemInfo,
err: tCase.err,
}, uuid.New(), uuid.New(), nil, map[string]string{}, nil)
recorder := httptest.NewRecorder()
ctx := context.Background()
request, err := http.NewRequestWithContext(ctx, http.MethodGet, "/diag/system", nil)
require.NoError(t, err)
handler.SystemHandler(recorder, request)
assert.Equal(t, tCase.statusCode, recorder.Code)
if tCase.statusCode == http.StatusOK && tCase.systemInfo != nil {
var response diagnostic.SystemInformationResponse
decoder := json.NewDecoder(recorder.Body)
err := decoder.Decode(&response)
require.NoError(t, err)
assert.Equal(t, tCase.systemInfo, response.Info)
}
})
}
}
func TestTunnelStateHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
tunnelID uuid.UUID
clientID uuid.UUID
connections []tunnelstate.IndexedConnectionInfo
icmpSources []string
}{
{
name: "case1",
tunnelID: uuid.New(),
clientID: uuid.New(),
},
{
name: "case2",
tunnelID: uuid.New(),
clientID: uuid.New(),
icmpSources: []string{"172.17.0.3", "::1"},
connections: []tunnelstate.IndexedConnectionInfo{{
ConnectionInfo: tunnelstate.ConnectionInfo{
IsConnected: true,
Protocol: connection.QUIC,
EdgeAddress: net.IPv4(100, 100, 100, 100),
},
Index: 0,
}},
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
tracker := newTrackerFromConns(t, tCase.connections)
handler := diagnostic.NewDiagnosticHandler(
&log,
0,
nil,
tCase.tunnelID,
tCase.clientID,
tracker,
map[string]string{},
tCase.icmpSources,
)
recorder := httptest.NewRecorder()
handler.TunnelStateHandler(recorder, nil)
decoder := json.NewDecoder(recorder.Body)
var response diagnostic.TunnelState
err := decoder.Decode(&response)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, recorder.Code)
assert.Equal(t, tCase.tunnelID, response.TunnelID)
assert.Equal(t, tCase.clientID, response.ConnectorID)
assert.Equal(t, tCase.connections, response.Connections)
assert.Equal(t, tCase.icmpSources, response.ICMPSources)
})
}
}
func TestConfigurationHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
flags map[string]string
expected map[string]string
}{
{
name: "empty cli",
flags: make(map[string]string),
expected: map[string]string{
"uid": "0",
},
},
{
name: "cli with flags",
flags: map[string]string{
"b": "a",
"c": "a",
"d": "a",
"uid": "0",
},
expected: map[string]string{
"b": "a",
"c": "a",
"d": "a",
"uid": "0",
},
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
var response map[string]string
handler := diagnostic.NewDiagnosticHandler(&log, 0, nil, uuid.New(), uuid.New(), nil, tCase.flags, nil)
recorder := httptest.NewRecorder()
handler.ConfigurationHandler(recorder, nil)
decoder := json.NewDecoder(recorder.Body)
err := decoder.Decode(&response)
require.NoError(t, err)
_, ok := response["uid"]
assert.True(t, ok)
delete(tCase.expected, "uid")
delete(response, "uid")
assert.Equal(t, http.StatusOK, recorder.Code)
assert.Equal(t, tCase.expected, response)
})
}
}
+34
View File
@@ -0,0 +1,34 @@
package diagnostic
import (
"context"
)
// Represents the path of the log file or log directory.
// This struct is meant to give some ergonimics regarding
// the logging information.
type LogInformation struct {
path string // path to a file or directory
wasCreated bool // denotes if `path` was created
isDirectory bool // denotes if `path` is a directory
}
func NewLogInformation(
path string,
wasCreated bool,
isDirectory bool,
) *LogInformation {
return &LogInformation{
path,
wasCreated,
isDirectory,
}
}
type LogCollector interface {
// This function is responsible for returning a path to a single file
// whose contents are the logs of a cloudflared instance.
// A new file may be create by a LogCollector, thus, its the caller
// responsibility to remove the newly create file.
Collect(ctx context.Context) (*LogInformation, error)
}
+47
View File
@@ -0,0 +1,47 @@
package diagnostic
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
)
type DockerLogCollector struct {
containerID string // This member identifies the container by identifier or name
}
func NewDockerLogCollector(containerID string) *DockerLogCollector {
return &DockerLogCollector{
containerID,
}
}
func (collector *DockerLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
tmp := os.TempDir()
outputHandle, err := os.Create(filepath.Join(tmp, logFilename))
if err != nil {
return nil, fmt.Errorf("error opening output file: %w", err)
}
defer outputHandle.Close()
// Calculate 2 weeks ago
since := time.Now().Add(twoWeeksOffset).Format(time.RFC3339)
command := exec.CommandContext(
ctx,
"docker",
"logs",
"--tail",
tailMaxNumberOfLines,
"--since",
since,
collector.containerID,
)
return PipeCommandOutputToFile(command, outputHandle)
}
+105
View File
@@ -0,0 +1,105 @@
package diagnostic
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
)
const (
linuxManagedLogsPath = "/var/log/cloudflared.err"
darwinManagedLogsPath = "/Library/Logs/com.cloudflare.cloudflared.err.log"
linuxServiceConfigurationPath = "/etc/systemd/system/cloudflared.service"
linuxSystemdPath = "/run/systemd/system"
)
type HostLogCollector struct {
client HTTPClient
}
func NewHostLogCollector(client HTTPClient) *HostLogCollector {
return &HostLogCollector{
client,
}
}
func extractLogsFromJournalCtl(ctx context.Context) (*LogInformation, error) {
tmp := os.TempDir()
outputHandle, err := os.Create(filepath.Join(tmp, logFilename))
if err != nil {
return nil, fmt.Errorf("error opening output file: %w", err)
}
defer outputHandle.Close()
command := exec.CommandContext(
ctx,
"journalctl",
"--since",
"2 weeks ago",
"-u",
"cloudflared.service",
)
return PipeCommandOutputToFile(command, outputHandle)
}
func getServiceLogPath() (string, error) {
switch runtime.GOOS {
case "darwin":
{
path := darwinManagedLogsPath
if _, err := os.Stat(path); err == nil {
return path, nil
}
userHomeDir, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("error getting user home: %w", err)
}
return filepath.Join(userHomeDir, darwinManagedLogsPath), nil
}
case "linux":
{
return linuxManagedLogsPath, nil
}
default:
return "", ErrManagedLogNotFound
}
}
func (collector *HostLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
logConfiguration, err := collector.client.GetLogConfiguration(ctx)
if err != nil {
return nil, fmt.Errorf("error getting log configuration: %w", err)
}
if logConfiguration.uid == 0 {
_, statSystemdErr := os.Stat(linuxServiceConfigurationPath)
_, statServiceConfigurationErr := os.Stat(linuxServiceConfigurationPath)
if statSystemdErr == nil && statServiceConfigurationErr == nil && runtime.GOOS == "linux" {
return extractLogsFromJournalCtl(ctx)
}
path, err := getServiceLogPath()
if err != nil {
return nil, err
}
return NewLogInformation(path, false, false), nil
}
if logConfiguration.logFile != "" {
return NewLogInformation(logConfiguration.logFile, false, false), nil
} else if logConfiguration.logDirectory != "" {
return NewLogInformation(logConfiguration.logDirectory, false, true), nil
}
return nil, ErrLogConfigurationIsInvalid
}
+63
View File
@@ -0,0 +1,63 @@
package diagnostic
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
)
type KubernetesLogCollector struct {
containerID string // This member identifies the container by identifier or name
pod string // This member identifies the pod where the container is deployed
}
func NewKubernetesLogCollector(containerID, pod string) *KubernetesLogCollector {
return &KubernetesLogCollector{
containerID,
pod,
}
}
func (collector *KubernetesLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
tmp := os.TempDir()
outputHandle, err := os.Create(filepath.Join(tmp, logFilename))
if err != nil {
return nil, fmt.Errorf("error opening output file: %w", err)
}
defer outputHandle.Close()
var command *exec.Cmd
// Calculate 2 weeks ago
since := time.Now().Add(twoWeeksOffset).Format(time.RFC3339)
if collector.containerID != "" {
command = exec.CommandContext(
ctx,
"kubectl",
"logs",
collector.pod,
"--since-time",
since,
"--tail",
tailMaxNumberOfLines,
"-c",
collector.containerID,
)
} else {
command = exec.CommandContext(
ctx,
"kubectl",
"logs",
collector.pod,
"--since-time",
since,
"--tail",
tailMaxNumberOfLines,
)
}
return PipeCommandOutputToFile(command, outputHandle)
}
+109
View File
@@ -0,0 +1,109 @@
package diagnostic
import (
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
)
func PipeCommandOutputToFile(command *exec.Cmd, outputHandle *os.File) (*LogInformation, error) {
stdoutReader, err := command.StdoutPipe()
if err != nil {
return nil, fmt.Errorf(
"error retrieving stdout from command '%s': %w",
command.String(),
err,
)
}
stderrReader, err := command.StderrPipe()
if err != nil {
return nil, fmt.Errorf(
"error retrieving stderr from command '%s': %w",
command.String(),
err,
)
}
if err := command.Start(); err != nil {
return nil, fmt.Errorf(
"error running command '%s': %w",
command.String(),
err,
)
}
_, err = io.Copy(outputHandle, stdoutReader)
if err != nil {
return nil, fmt.Errorf(
"error copying stdout from %s to file %s: %w",
command.String(),
outputHandle.Name(),
err,
)
}
_, err = io.Copy(outputHandle, stderrReader)
if err != nil {
return nil, fmt.Errorf(
"error copying stderr from %s to file %s: %w",
command.String(),
outputHandle.Name(),
err,
)
}
if err := command.Wait(); err != nil {
return nil, fmt.Errorf(
"error waiting from command '%s': %w",
command.String(),
err,
)
}
return NewLogInformation(outputHandle.Name(), true, false), nil
}
func CopyFilesFromDirectory(path string) (string, error) {
// rolling logs have as suffix the current date thus
// when iterating the path files they are already in
// chronological order
files, err := os.ReadDir(path)
if err != nil {
return "", fmt.Errorf("error reading directory %s: %w", path, err)
}
outputHandle, err := os.Create(filepath.Join(os.TempDir(), logFilename))
if err != nil {
return "", fmt.Errorf("creating file %s: %w", outputHandle.Name(), err)
}
defer outputHandle.Close()
for _, file := range files {
logHandle, err := os.Open(filepath.Join(path, file.Name()))
if err != nil {
return "", fmt.Errorf("error opening file %s:%w", file.Name(), err)
}
defer logHandle.Close()
_, err = io.Copy(outputHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying file %s:%w", logHandle.Name(), err)
}
}
logHandle, err := os.Open(filepath.Join(path, "cloudflared.log"))
if err != nil {
return "", fmt.Errorf("error opening file %s:%w", logHandle.Name(), err)
}
defer logHandle.Close()
_, err = io.Copy(outputHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying file %s:%w", logHandle.Name(), err)
}
return outputHandle.Name(), nil
}
+77
View File
@@ -0,0 +1,77 @@
package diagnostic
import (
"context"
"errors"
"time"
)
const MicrosecondsFactor = 1000.0
var ErrEmptyDomain = errors.New("domain must not be empty")
// For now only support ICMP is provided.
type IPVersion int
const (
V4 IPVersion = iota
V6 IPVersion = iota
)
type Hop struct {
Hop uint8 `json:"hop,omitempty"` // hop number along the route
Domain string `json:"domain,omitempty"` // domain and/or ip of the hop, this field will be '*' if the hop is a timeout
Rtts []time.Duration `json:"rtts,omitempty"` // RTT measurements in microseconds
}
type TraceOptions struct {
ttl uint64 // number of hops to perform
timeout time.Duration // wait timeout for each response
address string // address to trace
useV4 bool
}
func NewTimeoutHop(
hop uint8,
) *Hop {
// Whenever there is a hop in the format of 'N * * *'
// it means that the hop in the path didn't answer to
// any probe.
return NewHop(
hop,
"*",
nil,
)
}
func NewHop(hop uint8, domain string, rtts []time.Duration) *Hop {
return &Hop{
hop,
domain,
rtts,
}
}
func NewTraceOptions(
ttl uint64,
timeout time.Duration,
address string,
useV4 bool,
) TraceOptions {
return TraceOptions{
ttl,
timeout,
address,
useV4,
}
}
type NetworkCollector interface {
// Performs a trace route operation with the specified options.
// In case the trace fails, it will return a non-nil error and
// it may return a string which represents the raw information
// obtained.
// In case it is successful it will only return an array of Hops
// an empty string and a nil error.
Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error)
}
+78
View File
@@ -0,0 +1,78 @@
//go:build darwin || linux
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
"time"
)
type NetworkCollectorImpl struct{}
func (tracer *NetworkCollectorImpl) Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error) {
args := []string{
"-I",
"-w",
strconv.FormatInt(int64(options.timeout.Seconds()), 10),
"-m",
strconv.FormatUint(options.ttl, 10),
options.address,
}
var command string
switch options.useV4 {
case false:
command = "traceroute6"
default:
command = "traceroute"
}
process := exec.CommandContext(ctx, command, args...)
return decodeNetworkOutputToFile(process, DecodeLine)
}
func DecodeLine(text string) (*Hop, error) {
fields := strings.Fields(text)
parts := []string{}
filter := func(s string) bool { return s != "*" && s != "ms" }
for _, field := range fields {
if filter(field) {
parts = append(parts, field)
}
}
index, err := strconv.ParseUint(parts[0], 10, 8)
if err != nil {
return nil, fmt.Errorf("couldn't parse index from timeout hop: %w", err)
}
if len(parts) == 1 {
return NewTimeoutHop(uint8(index)), nil
}
domain := ""
rtts := []time.Duration{}
for _, part := range parts[1:] {
rtt, err := strconv.ParseFloat(part, 64)
if err != nil {
domain += part + " "
} else {
rtts = append(rtts, time.Duration(rtt*MicrosecondsFactor))
}
}
domain, _ = strings.CutSuffix(domain, " ")
if domain == "" {
return nil, ErrEmptyDomain
}
return NewHop(uint8(index), domain, rtts), nil
}
+173
View File
@@ -0,0 +1,173 @@
//go:build darwin || linux
package diagnostic_test
import (
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
diagnostic "github.com/cloudflare/cloudflared/diagnostic/network"
)
func TestDecode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
text string
expectedHops []*diagnostic.Hop
}{
{
"repeated hop index parse failure",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
someletters * * *
4 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms `,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(4),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"hop index parse failure",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
someletters 8.8.8.8 8.8.8.9 abc ms 0.456 ms 0.789 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"missing rtt",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 * 8.8.8.8 8.8.8.9 0.456 ms 0.789 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"8.8.8.8 8.8.8.9",
[]time.Duration{
time.Duration(456),
time.Duration(789),
},
),
},
},
{
"simple example ipv4",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
3 * * *`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
},
},
{
"simple example ipv6",
` 1 2400:cb00:107:1024::ac44:6550 12.780 ms 9.118 ms 10.046 ms
2 2a09:bac1:: 9.945 ms 10.033 ms 11.562 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"2400:cb00:107:1024::ac44:6550",
[]time.Duration{
time.Duration(12780),
time.Duration(9118),
time.Duration(10046),
},
),
diagnostic.NewHop(
uint8(2),
"2a09:bac1::",
[]time.Duration{
time.Duration(9945),
time.Duration(10033),
time.Duration(11562),
},
),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
t.Parallel()
hops, err := diagnostic.Decode(strings.NewReader(test.text), diagnostic.DecodeLine)
require.NoError(t, err)
assert.Equal(t, test.expectedHops, hops)
})
}
}
+74
View File
@@ -0,0 +1,74 @@
package diagnostic
import (
"bufio"
"bytes"
"fmt"
"io"
"os/exec"
)
type DecodeLineFunc func(text string) (*Hop, error)
func decodeNetworkOutputToFile(command *exec.Cmd, decodeLine DecodeLineFunc) ([]*Hop, string, error) {
stdout, err := command.StdoutPipe()
if err != nil {
return nil, "", fmt.Errorf("error piping traceroute's output: %w", err)
}
if err := command.Start(); err != nil {
return nil, "", fmt.Errorf("error starting traceroute: %w", err)
}
// Tee the output to a string to have the raw information
// in case the decode call fails
// This error is handled only after the Wait call below returns
// otherwise the process can become a zombie
buf := bytes.NewBuffer([]byte{})
tee := io.TeeReader(stdout, buf)
hops, err := Decode(tee, decodeLine)
// regardless of success of the decoding
// consume all output to have available in buf
_, _ = io.ReadAll(tee)
if werr := command.Wait(); werr != nil {
return nil, "", fmt.Errorf("error finishing traceroute: %w", werr)
}
if err != nil {
return nil, buf.String(), err
}
return hops, buf.String(), nil
}
func Decode(reader io.Reader, decodeLine DecodeLineFunc) ([]*Hop, error) {
scanner := bufio.NewScanner(reader)
scanner.Split(bufio.ScanLines)
var hops []*Hop
for scanner.Scan() {
text := scanner.Text()
if text == "" {
continue
}
hop, err := decodeLine(text)
if err != nil {
// This continue is here on the error case because there are lines at the start and end
// that may not be parsable. (check windows tracert output)
// The skip is here because aside from the start and end lines the other lines should
// always be parsable without errors.
continue
}
hops = append(hops, hop)
}
if scanner.Err() != nil {
return nil, fmt.Errorf("scanner reported an error: %w", scanner.Err())
}
return hops, nil
}
+81
View File
@@ -0,0 +1,81 @@
//go:build windows
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
"time"
)
type NetworkCollectorImpl struct{}
func (tracer *NetworkCollectorImpl) Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error) {
ipversion := "-4"
if !options.useV4 {
ipversion = "-6"
}
args := []string{
ipversion,
"-w",
strconv.FormatInt(int64(options.timeout.Seconds()), 10),
"-h",
strconv.FormatUint(options.ttl, 10),
// Do not resolve host names (can add 30+ seconds to run time)
"-d",
options.address,
}
command := exec.CommandContext(ctx, "tracert.exe", args...)
return decodeNetworkOutputToFile(command, DecodeLine)
}
func DecodeLine(text string) (*Hop, error) {
const requestTimedOut = "Request timed out."
fields := strings.Fields(text)
parts := []string{}
filter := func(s string) bool { return s != "*" && s != "ms" }
for _, field := range fields {
if filter(field) {
parts = append(parts, field)
}
}
index, err := strconv.ParseUint(parts[0], 10, 8)
if err != nil {
return nil, fmt.Errorf("couldn't parse index from timeout hop: %w", err)
}
domain := ""
rtts := []time.Duration{}
for _, part := range parts[1:] {
rtt, err := strconv.ParseFloat(strings.TrimLeft(part, "<"), 64)
if err != nil {
domain += part + " "
} else {
rtts = append(rtts, time.Duration(rtt*MicrosecondsFactor))
}
}
domain, _ = strings.CutSuffix(domain, " ")
// If the domain is equal to "Request timed out." then we build a
// timeout hop.
if domain == requestTimedOut {
return NewTimeoutHop(uint8(index)), nil
}
if domain == "" {
return nil, ErrEmptyDomain
}
return NewHop(uint8(index), domain, rtts), nil
}
@@ -0,0 +1,210 @@
//go:build windows
package diagnostic_test
import (
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
diagnostic "github.com/cloudflare/cloudflared/diagnostic/network"
)
func TestDecode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
text string
expectedHops []*diagnostic.Hop
}{
{
"tracert output",
`
Tracing route to region2.v2.argotunnel.com [198.41.200.73]
over a maximum of 5 hops:
1 10 ms <1 ms 1 ms 192.168.64.1
2 27 ms 14 ms 5 ms 192.168.1.254
3 * * * Request timed out.
4 * * * Request timed out.
5 27 ms 5 ms 5 ms 195.8.30.245
Trace complete.
`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"192.168.64.1",
[]time.Duration{
time.Duration(10000),
time.Duration(1000),
time.Duration(1000),
},
),
diagnostic.NewHop(
uint8(2),
"192.168.1.254",
[]time.Duration{
time.Duration(27000),
time.Duration(14000),
time.Duration(5000),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
diagnostic.NewTimeoutHop(uint8(4)),
diagnostic.NewHop(
uint8(5),
"195.8.30.245",
[]time.Duration{
time.Duration(27000),
time.Duration(5000),
time.Duration(5000),
},
),
},
},
{
"repeated hop index parse failure",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
someletters * * *`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"hop index parse failure",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
someletters abc ms 0.456 ms 0.789 ms 8.8.8.8 8.8.8.9`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"missing rtt",
`1 <12.874 ms <15.517 ms <15.311 ms 172.68.101.121 (172.68.101.121)
2 * 0.456 ms 0.789 ms 8.8.8.8 8.8.8.9`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"8.8.8.8 8.8.8.9",
[]time.Duration{
time.Duration(456),
time.Duration(789),
},
),
},
},
{
"simple example ipv4",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
3 * * * Request timed out.`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
},
},
{
"simple example ipv6",
` 1 12.780 ms 9.118 ms 10.046 ms 2400:cb00:107:1024::ac44:6550
2 9.945 ms 10.033 ms 11.562 ms 2a09:bac1::`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"2400:cb00:107:1024::ac44:6550",
[]time.Duration{
time.Duration(12780),
time.Duration(9118),
time.Duration(10046),
},
),
diagnostic.NewHop(
uint8(2),
"2a09:bac1::",
[]time.Duration{
time.Duration(9945),
time.Duration(10033),
time.Duration(11562),
},
),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
t.Parallel()
hops, err := diagnostic.Decode(strings.NewReader(test.text), diagnostic.DecodeLine)
require.NoError(t, err)
assert.Equal(t, test.expectedHops, hops)
})
}
}
+150
View File
@@ -0,0 +1,150 @@
package diagnostic
import (
"context"
"encoding/json"
"errors"
"strings"
)
type SystemInformationError struct {
Err error `json:"error"`
RawInfo string `json:"rawInfo"`
}
func (err SystemInformationError) Error() string {
return err.Err.Error()
}
func (err SystemInformationError) MarshalJSON() ([]byte, error) {
s := map[string]string{
"error": err.Err.Error(),
"rawInfo": err.RawInfo,
}
return json.Marshal(s)
}
type SystemInformationGeneralError struct {
OperatingSystemInformationError error
MemoryInformationError error
FileDescriptorsInformationError error
DiskVolumeInformationError error
}
func (err SystemInformationGeneralError) Error() string {
builder := &strings.Builder{}
builder.WriteString("errors found:")
if err.OperatingSystemInformationError != nil {
builder.WriteString(err.OperatingSystemInformationError.Error() + ", ")
}
if err.MemoryInformationError != nil {
builder.WriteString(err.MemoryInformationError.Error() + ", ")
}
if err.FileDescriptorsInformationError != nil {
builder.WriteString(err.FileDescriptorsInformationError.Error() + ", ")
}
if err.DiskVolumeInformationError != nil {
builder.WriteString(err.DiskVolumeInformationError.Error() + ", ")
}
return builder.String()
}
func (err SystemInformationGeneralError) MarshalJSON() ([]byte, error) {
data := map[string]SystemInformationError{}
var sysErr SystemInformationError
if errors.As(err.OperatingSystemInformationError, &sysErr) {
data["operatingSystemInformationError"] = sysErr
}
if errors.As(err.MemoryInformationError, &sysErr) {
data["memoryInformationError"] = sysErr
}
if errors.As(err.FileDescriptorsInformationError, &sysErr) {
data["fileDescriptorsInformationError"] = sysErr
}
if errors.As(err.DiskVolumeInformationError, &sysErr) {
data["diskVolumeInformationError"] = sysErr
}
return json.Marshal(data)
}
type DiskVolumeInformation struct {
Name string `json:"name"` // represents the filesystem in linux/macos or device name in windows
SizeMaximum uint64 `json:"sizeMaximum"` // represents the maximum size of the disk in kilobytes
SizeCurrent uint64 `json:"sizeCurrent"` // represents the current size of the disk in kilobytes
}
func NewDiskVolumeInformation(name string, maximum, current uint64) *DiskVolumeInformation {
return &DiskVolumeInformation{
name,
maximum,
current,
}
}
type SystemInformation struct {
MemoryMaximum uint64 `json:"memoryMaximum,omitempty"` // represents the maximum memory of the system in kilobytes
MemoryCurrent uint64 `json:"memoryCurrent,omitempty"` // represents the system's memory in use in kilobytes
FileDescriptorMaximum uint64 `json:"fileDescriptorMaximum,omitempty"` // represents the maximum number of file descriptors of the system
FileDescriptorCurrent uint64 `json:"fileDescriptorCurrent,omitempty"` // represents the system's file descriptors in use
OsSystem string `json:"osSystem,omitempty"` // represents the operating system name i.e.: linux, windows, darwin
HostName string `json:"hostName,omitempty"` // represents the system host name
OsVersion string `json:"osVersion,omitempty"` // detailed information about the system's release version level
OsRelease string `json:"osRelease,omitempty"` // detailed information about the system's release
Architecture string `json:"architecture,omitempty"` // represents the system's hardware platform i.e: arm64/amd64
CloudflaredVersion string `json:"cloudflaredVersion,omitempty"` // the runtime version of cloudflared
GoVersion string `json:"goVersion,omitempty"`
GoArch string `json:"goArch,omitempty"`
Disk []*DiskVolumeInformation `json:"disk,omitempty"`
}
func NewSystemInformation(
memoryMaximum,
memoryCurrent,
filesMaximum,
filesCurrent uint64,
osystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
goVersion,
goArchitecture string,
disk []*DiskVolumeInformation,
) *SystemInformation {
return &SystemInformation{
memoryMaximum,
memoryCurrent,
filesMaximum,
filesCurrent,
osystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
goVersion,
goArchitecture,
disk,
}
}
type SystemCollector interface {
// If the collection is successful it will return `SystemInformation` struct,
// and a nil error.
//
// This function expects that the caller sets the context timeout to prevent
// long-lived collectors.
Collect(ctx context.Context) (*SystemInformation, error)
}
+150
View File
@@ -0,0 +1,150 @@
//go:build linux
package diagnostic
import (
"context"
"fmt"
"os/exec"
"runtime"
"strconv"
"strings"
)
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
fdInfo, fdInfoRaw, fdInfoErr := collectFileDescriptorInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformationUnix(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformationUnix(ctx)
var memoryMaximum, memoryCurrent, fileDescriptorMaximum, fileDescriptorCurrent uint64
var osSystem, name, osVersion, osRelease, architecture string
gerror := SystemInformationGeneralError{}
if memoryInfoErr != nil {
gerror.MemoryInformationError = SystemInformationError{
Err: memoryInfoErr,
RawInfo: memoryInfoRaw,
}
} else {
memoryMaximum = memoryInfo.MemoryMaximum
memoryCurrent = memoryInfo.MemoryCurrent
}
if fdInfoErr != nil {
gerror.FileDescriptorsInformationError = SystemInformationError{
Err: fdInfoErr,
RawInfo: fdInfoRaw,
}
} else {
fileDescriptorMaximum = fdInfo.FileDescriptorMaximum
fileDescriptorCurrent = fdInfo.FileDescriptorCurrent
}
if diskErr != nil {
gerror.DiskVolumeInformationError = SystemInformationError{
Err: diskErr,
RawInfo: disksRaw,
}
}
if osInfoErr != nil {
gerror.OperatingSystemInformationError = SystemInformationError{
Err: osInfoErr,
RawInfo: osInfoRaw,
}
} else {
osSystem = osInfo.OsSystem
name = osInfo.Name
osVersion = osInfo.OsVersion
osRelease = osInfo.OsRelease
architecture = osInfo.Architecture
}
cloudflaredVersion := collector.version
info := NewSystemInformation(
memoryMaximum,
memoryCurrent,
fileDescriptorMaximum,
fileDescriptorCurrent,
osSystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
runtime.Version(),
runtime.GOARCH,
disks,
)
return info, gerror
}
func collectMemoryInformation(ctx context.Context) (*MemoryInformation, string, error) {
// This function relies on the output of `cat /proc/meminfo` to retrieve
// memoryMax and memoryCurrent.
// The expected output is in the format of `KEY VALUE UNIT`.
const (
memTotalPrefix = "MemTotal"
memAvailablePrefix = "MemAvailable"
)
command := exec.CommandContext(ctx, "cat", "/proc/meminfo")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
mapper := func(field string) (uint64, error) {
field = strings.TrimRight(field, " kB")
return strconv.ParseUint(field, 10, 64)
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memTotalPrefix, memAvailablePrefix, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
func collectFileDescriptorInformation(ctx context.Context) (*FileDescriptorInformation, string, error) {
// Command retrieved from https://docs.kernel.org/admin-guide/sysctl/fs.html#file-max-file-nr.
// If the sysctl is not available the command with fail.
command := exec.CommandContext(ctx, "sysctl", "-n", "fs.file-nr")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
fileDescriptorInfo, err := ParseSysctlFileDescriptorInformation(output)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return fileDescriptorInfo, output, nil
}
+172
View File
@@ -0,0 +1,172 @@
//go:build darwin
package diagnostic
import (
"context"
"fmt"
"os/exec"
"runtime"
"strconv"
)
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
fdInfo, fdInfoRaw, fdInfoErr := collectFileDescriptorInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformationUnix(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformationUnix(ctx)
var memoryMaximum, memoryCurrent, fileDescriptorMaximum, fileDescriptorCurrent uint64
var osSystem, name, osVersion, osRelease, architecture string
err := SystemInformationGeneralError{
OperatingSystemInformationError: nil,
MemoryInformationError: nil,
FileDescriptorsInformationError: nil,
DiskVolumeInformationError: nil,
}
if memoryInfoErr != nil {
err.MemoryInformationError = SystemInformationError{
Err: memoryInfoErr,
RawInfo: memoryInfoRaw,
}
} else {
memoryMaximum = memoryInfo.MemoryMaximum
memoryCurrent = memoryInfo.MemoryCurrent
}
if fdInfoErr != nil {
err.FileDescriptorsInformationError = SystemInformationError{
Err: fdInfoErr,
RawInfo: fdInfoRaw,
}
} else {
fileDescriptorMaximum = fdInfo.FileDescriptorMaximum
fileDescriptorCurrent = fdInfo.FileDescriptorCurrent
}
if diskErr != nil {
err.DiskVolumeInformationError = SystemInformationError{
Err: diskErr,
RawInfo: disksRaw,
}
}
if osInfoErr != nil {
err.OperatingSystemInformationError = SystemInformationError{
Err: osInfoErr,
RawInfo: osInfoRaw,
}
} else {
osSystem = osInfo.OsSystem
name = osInfo.Name
osVersion = osInfo.OsVersion
osRelease = osInfo.OsRelease
architecture = osInfo.Architecture
}
cloudflaredVersion := collector.version
info := NewSystemInformation(
memoryMaximum,
memoryCurrent,
fileDescriptorMaximum,
fileDescriptorCurrent,
osSystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
runtime.Version(),
runtime.GOARCH,
disks,
)
return info, err
}
func collectFileDescriptorInformation(ctx context.Context) (
*FileDescriptorInformation,
string,
error,
) {
const (
fileDescriptorMaximumKey = "kern.maxfiles"
fileDescriptorCurrentKey = "kern.num_files"
)
command := exec.CommandContext(ctx, "sysctl", fileDescriptorMaximumKey, fileDescriptorCurrentKey)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
fileDescriptorInfo, err := ParseFileDescriptorInformationFromKV(
output,
fileDescriptorMaximumKey,
fileDescriptorCurrentKey,
)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return fileDescriptorInfo, output, nil
}
func collectMemoryInformation(ctx context.Context) (
*MemoryInformation,
string,
error,
) {
const (
memoryMaximumKey = "hw.memsize"
memoryAvailableKey = "hw.memsize_usable"
)
command := exec.CommandContext(
ctx,
"sysctl",
memoryMaximumKey,
memoryAvailableKey,
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
mapper := func(field string) (uint64, error) {
const kiloBytes = 1024
value, err := strconv.ParseUint(field, 10, 64)
return value / kiloBytes, err
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memoryMaximumKey, memoryAvailableKey, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
+466
View File
@@ -0,0 +1,466 @@
package diagnostic_test
import (
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/diagnostic"
)
func TestParseMemoryInformationFromKV(t *testing.T) {
t.Parallel()
mapper := func(field string) (uint64, error) {
value, err := strconv.ParseUint(field, 10, 64)
return value, err
}
linuxMapper := func(field string) (uint64, error) {
field = strings.TrimRight(field, " kB")
return strconv.ParseUint(field, 10, 64)
}
windowsMemoryOutput := `
FreeVirtualMemory : 5350472
TotalVirtualMemorySize : 8903424
`
macosMemoryOutput := `hw.memsize: 38654705664
hw.memsize_usable: 38009012224`
memoryOutputWithMissingKey := `hw.memsize: 38654705664`
linuxMemoryOutput := `MemTotal: 8028860 kB
MemFree: 731396 kB
MemAvailable: 4678844 kB
Buffers: 472632 kB
Cached: 3186492 kB
SwapCached: 4196 kB
Active: 3088988 kB
Inactive: 3468560 kB`
tests := []struct {
name string
output string
memoryMaximumKey string
memoryAvailableKey string
expected *diagnostic.MemoryInformation
expectedErr bool
mapper func(string) (uint64, error)
}{
{
name: "parse linux memory values",
output: linuxMemoryOutput,
memoryMaximumKey: "MemTotal",
memoryAvailableKey: "MemAvailable",
expected: &diagnostic.MemoryInformation{
8028860,
8028860 - 4678844,
},
expectedErr: false,
mapper: linuxMapper,
},
{
name: "parse memory values with missing key",
output: memoryOutputWithMissingKey,
memoryMaximumKey: "hw.memsize",
memoryAvailableKey: "hw.memsize_usable",
expected: nil,
expectedErr: true,
mapper: mapper,
},
{
name: "parse macos memory values",
output: macosMemoryOutput,
memoryMaximumKey: "hw.memsize",
memoryAvailableKey: "hw.memsize_usable",
expected: &diagnostic.MemoryInformation{
38654705664,
38654705664 - 38009012224,
},
expectedErr: false,
mapper: mapper,
},
{
name: "parse windows memory values",
output: windowsMemoryOutput,
memoryMaximumKey: "TotalVirtualMemorySize",
memoryAvailableKey: "FreeVirtualMemory",
expected: &diagnostic.MemoryInformation{
8903424,
8903424 - 5350472,
},
expectedErr: false,
mapper: mapper,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
memoryInfo, err := diagnostic.ParseMemoryInformationFromKV(
tCase.output,
tCase.memoryMaximumKey,
tCase.memoryAvailableKey,
tCase.mapper,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, memoryInfo)
}
})
}
}
func TestParseUnameOutput(t *testing.T) {
t.Parallel()
tests := []struct {
name string
output string
os string
expected *diagnostic.OsInfo
expectedErr bool
}{
{
name: "darwin machine",
output: "Darwin APC 23.6.0 Darwin Kernel Version 99.6.0: Wed Jul 31 20:48:04 PDT 1997; root:xnu-66666.666.6.666.6~1/RELEASE_ARM64_T6666 arm64",
os: "darwin",
expected: &diagnostic.OsInfo{
Architecture: "arm64",
Name: "APC",
OsSystem: "Darwin",
OsRelease: "Darwin Kernel Version 99.6.0: Wed Jul 31 20:48:04 PDT 1997; root:xnu-66666.666.6.666.6~1/RELEASE_ARM64_T6666",
OsVersion: "23.6.0",
},
expectedErr: false,
},
{
name: "linux machine",
output: "Linux dab00d565591 6.6.31-linuxkit #1 SMP Thu May 23 08:36:57 UTC 2024 aarch64 GNU/Linux",
os: "linux",
expected: &diagnostic.OsInfo{
Architecture: "aarch64",
Name: "dab00d565591",
OsSystem: "Linux",
OsRelease: "#1 SMP Thu May 23 08:36:57 UTC 2024",
OsVersion: "6.6.31-linuxkit",
},
expectedErr: false,
},
{
name: "not enough fields",
output: "Linux ",
os: "linux",
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
memoryInfo, err := diagnostic.ParseUnameOutput(
tCase.output,
tCase.os,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, memoryInfo)
}
})
}
}
func TestParseFileDescriptorInformationFromKV(t *testing.T) {
const (
fileDescriptorMaximumKey = "kern.maxfiles"
fileDescriptorCurrentKey = "kern.num_files"
)
t.Parallel()
memoryOutput := `kern.maxfiles: 276480
kern.num_files: 11787`
memoryOutputWithMissingKey := `kern.maxfiles: 276480`
tests := []struct {
name string
output string
expected *diagnostic.FileDescriptorInformation
expectedErr bool
}{
{
name: "parse memory values with missing key",
output: memoryOutputWithMissingKey,
expected: nil,
expectedErr: true,
},
{
name: "parse macos memory values",
output: memoryOutput,
expected: &diagnostic.FileDescriptorInformation{
276480,
11787,
},
expectedErr: false,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
fdInfo, err := diagnostic.ParseFileDescriptorInformationFromKV(
tCase.output,
fileDescriptorMaximumKey,
fileDescriptorCurrentKey,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, fdInfo)
}
})
}
}
func TestParseSysctlFileDescriptorInformation(t *testing.T) {
t.Parallel()
tests := []struct {
name string
output string
expected *diagnostic.FileDescriptorInformation
expectedErr bool
}{
{
name: "expected output",
output: "111 0 1111111",
expected: &diagnostic.FileDescriptorInformation{
FileDescriptorMaximum: 1111111,
FileDescriptorCurrent: 111,
},
expectedErr: false,
},
{
name: "not enough fields",
output: "111 111 ",
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
fdsInfo, err := diagnostic.ParseSysctlFileDescriptorInformation(
tCase.output,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, fdsInfo)
}
})
}
}
func TestParseWinOperatingSystemInfo(t *testing.T) {
const (
architecturePrefix = "OSArchitecture"
osSystemPrefix = "Caption"
osVersionPrefix = "Version"
osReleasePrefix = "BuildNumber"
namePrefix = "CSName"
)
t.Parallel()
windowsIncompleteOsInfo := `
OSArchitecture : ARM 64 bits
Caption : Microsoft Windows 11 Home
Morekeys : 121314
CSName : UTILIZA-QO859QP
`
windowsCompleteOsInfo := `
OSArchitecture : ARM 64 bits
Caption : Microsoft Windows 11 Home
Version : 10.0.22631
BuildNumber : 22631
Morekeys : 121314
CSName : UTILIZA-QO859QP
`
tests := []struct {
name string
output string
expected *diagnostic.OsInfo
expectedErr bool
}{
{
name: "expected output",
output: windowsCompleteOsInfo,
expected: &diagnostic.OsInfo{
Architecture: "ARM 64 bits",
Name: "UTILIZA-QO859QP",
OsSystem: "Microsoft Windows 11 Home",
OsRelease: "22631",
OsVersion: "10.0.22631",
},
expectedErr: false,
},
{
name: "missing keys",
output: windowsIncompleteOsInfo,
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
osInfo, err := diagnostic.ParseWinOperatingSystemInfo(
tCase.output,
architecturePrefix,
osSystemPrefix,
osVersionPrefix,
osReleasePrefix,
namePrefix,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, osInfo)
}
})
}
}
func TestParseDiskVolumeInformationOutput(t *testing.T) {
t.Parallel()
invalidUnixDiskVolumeInfo := `Filesystem Size Used Avail Use% Mounted on
overlay 59G 19G 38G 33% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/run/host_mark/Users 461G 266G 195G 58% /tmp/cloudflared
/dev/vda1 59G 19G 38G 33% /etc/hosts
tmpfs 3.9G 0 3.9G 0% /sys/firmware
`
unixDiskVolumeInfo := `Filesystem Size Used Avail Use% Mounted on
overlay 61202244 18881444 39179476 33% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
/run/host_mark/Users 482797652 278648468 204149184 58% /tmp/cloudflared
/dev/vda1 61202244 18881444 39179476 33% /etc/hosts
tmpfs 4014428 0 4014428 0% /sys/firmware`
missingFields := ` DeviceID Size
-------- ----
C: size
E: 235563008
Z: 67754782720
`
invalidTypeField := ` DeviceID Size FreeSpace
-------- ---- ---------
C: size 31318736896
D:
E: 235563008 0
Z: 67754782720 31318732800
`
windowsDiskVolumeInfo := `
DeviceID Size FreeSpace
-------- ---- ---------
C: 67754782720 31318736896
E: 235563008 0
Z: 67754782720 31318732800`
tests := []struct {
name string
output string
expected []*diagnostic.DiskVolumeInformation
skipLines int
expectedErr bool
}{
{
name: "invalid unix disk volume information (numbers have units)",
output: invalidUnixDiskVolumeInfo,
expected: []*diagnostic.DiskVolumeInformation{},
skipLines: 1,
expectedErr: true,
},
{
name: "unix disk volume information",
output: unixDiskVolumeInfo,
skipLines: 1,
expected: []*diagnostic.DiskVolumeInformation{
diagnostic.NewDiskVolumeInformation("overlay", 61202244, 18881444),
diagnostic.NewDiskVolumeInformation("tmpfs", 65536, 0),
diagnostic.NewDiskVolumeInformation("shm", 65536, 0),
diagnostic.NewDiskVolumeInformation("/run/host_mark/Users", 482797652, 278648468),
diagnostic.NewDiskVolumeInformation("/dev/vda1", 61202244, 18881444),
diagnostic.NewDiskVolumeInformation("tmpfs", 4014428, 0),
},
expectedErr: false,
},
{
name: "windows disk volume information",
output: windowsDiskVolumeInfo,
expected: []*diagnostic.DiskVolumeInformation{
diagnostic.NewDiskVolumeInformation("C:", 67754782720, 31318736896),
diagnostic.NewDiskVolumeInformation("E:", 235563008, 0),
diagnostic.NewDiskVolumeInformation("Z:", 67754782720, 31318732800),
},
skipLines: 4,
expectedErr: false,
},
{
name: "insuficient fields",
output: missingFields,
expected: nil,
skipLines: 2,
expectedErr: true,
},
{
name: "invalid field",
output: invalidTypeField,
expected: nil,
skipLines: 2,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
disks, err := diagnostic.ParseDiskVolumeInformationOutput(tCase.output, tCase.skipLines, 1)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, disks)
}
})
}
}
+377
View File
@@ -0,0 +1,377 @@
package diagnostic
import (
"context"
"fmt"
"os/exec"
"runtime"
"sort"
"strconv"
"strings"
)
func findColonSeparatedPairs[V any](output string, keys []string, mapper func(string) (V, error)) map[string]V {
const (
memoryField = 1
memoryInformationFields = 2
)
lines := strings.Split(output, "\n")
pairs := make(map[string]V, 0)
// sort keys and lines to allow incremental search
sort.Strings(lines)
sort.Strings(keys)
// keeps track of the last key found
lastIndex := 0
for _, line := range lines {
if lastIndex == len(keys) {
// already found all keys no need to continue iterating
// over the other values
break
}
for index, key := range keys[lastIndex:] {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, key) {
fields := strings.Split(line, ":")
if len(fields) < memoryInformationFields {
lastIndex = index + 1
break
}
field, err := mapper(strings.TrimSpace(fields[memoryField]))
if err != nil {
lastIndex = lastIndex + index + 1
break
}
pairs[key] = field
lastIndex = lastIndex + index + 1
break
}
}
}
return pairs
}
func ParseDiskVolumeInformationOutput(output string, skipLines int, scale float64) ([]*DiskVolumeInformation, error) {
const (
diskFieldsMinimum = 3
nameField = 0
sizeMaximumField = 1
sizeCurrentField = 2
)
disksRaw := strings.Split(output, "\n")
disks := make([]*DiskVolumeInformation, 0)
if skipLines > len(disksRaw) || skipLines < 0 {
skipLines = 0
}
for _, disk := range disksRaw[skipLines:] {
if disk == "" {
// skip empty line
continue
}
fields := strings.Fields(disk)
if len(fields) < diskFieldsMinimum {
return nil, fmt.Errorf("expected disk volume to have %d fields got %d: %w",
diskFieldsMinimum, len(fields), ErrInsuficientFields,
)
}
name := fields[nameField]
sizeMaximum, err := strconv.ParseUint(fields[sizeMaximumField], 10, 64)
if err != nil {
continue
}
sizeCurrent, err := strconv.ParseUint(fields[sizeCurrentField], 10, 64)
if err != nil {
continue
}
diskInfo := NewDiskVolumeInformation(
name, uint64(float64(sizeMaximum)*scale), uint64(float64(sizeCurrent)*scale),
)
disks = append(disks, diskInfo)
}
if len(disks) == 0 {
return nil, ErrNoVolumeFound
}
return disks, nil
}
type OsInfo struct {
OsSystem string
Name string
OsVersion string
OsRelease string
Architecture string
}
func ParseUnameOutput(output string, system string) (*OsInfo, error) {
const (
osystemField = 0
nameField = 1
osVersionField = 2
osReleaseStartField = 3
osInformationFieldsMinimum = 6
darwin = "darwin"
)
architectureOffset := 2
if system == darwin {
architectureOffset = 1
}
fields := strings.Fields(output)
if len(fields) < osInformationFieldsMinimum {
return nil, fmt.Errorf("expected system information to have %d fields got %d: %w",
osInformationFieldsMinimum, len(fields), ErrInsuficientFields,
)
}
architectureField := len(fields) - architectureOffset
osystem := fields[osystemField]
name := fields[nameField]
osVersion := fields[osVersionField]
osRelease := strings.Join(fields[osReleaseStartField:architectureField], " ")
architecture := fields[architectureField]
return &OsInfo{
osystem,
name,
osVersion,
osRelease,
architecture,
}, nil
}
func ParseWinOperatingSystemInfo(
output string,
architectureKey string,
osSystemKey string,
osVersionKey string,
osReleaseKey string,
nameKey string,
) (*OsInfo, error) {
identity := func(s string) (string, error) { return s, nil }
keys := []string{architectureKey, osSystemKey, osVersionKey, osReleaseKey, nameKey}
pairs := findColonSeparatedPairs(
output,
keys,
identity,
)
architecture, exists := pairs[architectureKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, architectureKey)
}
osSystem, exists := pairs[osSystemKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osSystemKey)
}
osVersion, exists := pairs[osVersionKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osVersionKey)
}
osRelease, exists := pairs[osReleaseKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osReleaseKey)
}
name, exists := pairs[nameKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, nameKey)
}
return &OsInfo{osSystem, name, osVersion, osRelease, architecture}, nil
}
type FileDescriptorInformation struct {
FileDescriptorMaximum uint64
FileDescriptorCurrent uint64
}
func ParseSysctlFileDescriptorInformation(output string) (*FileDescriptorInformation, error) {
const (
openFilesField = 0
maxFilesField = 2
fileDescriptorLimitsFields = 3
)
fields := strings.Fields(output)
if len(fields) != fileDescriptorLimitsFields {
return nil,
fmt.Errorf(
"expected file descriptor information to have %d fields got %d: %w",
fileDescriptorLimitsFields,
len(fields),
ErrInsuficientFields,
)
}
fileDescriptorCurrent, err := strconv.ParseUint(fields[openFilesField], 10, 64)
if err != nil {
return nil, fmt.Errorf(
"error parsing files current field '%s': %w",
fields[openFilesField],
err,
)
}
fileDescriptorMaximum, err := strconv.ParseUint(fields[maxFilesField], 10, 64)
if err != nil {
return nil, fmt.Errorf("error parsing files max field '%s': %w", fields[maxFilesField], err)
}
return &FileDescriptorInformation{fileDescriptorMaximum, fileDescriptorCurrent}, nil
}
func ParseFileDescriptorInformationFromKV(
output string,
fileDescriptorMaximumKey string,
fileDescriptorCurrentKey string,
) (*FileDescriptorInformation, error) {
mapper := func(field string) (uint64, error) {
return strconv.ParseUint(field, 10, 64)
}
pairs := findColonSeparatedPairs(output, []string{fileDescriptorMaximumKey, fileDescriptorCurrentKey}, mapper)
fileDescriptorMaximum, exists := pairs[fileDescriptorMaximumKey]
if !exists {
return nil, fmt.Errorf(
"parsing file descriptor information: %w, key=%s",
ErrKeyNotFound,
fileDescriptorMaximumKey,
)
}
fileDescriptorCurrent, exists := pairs[fileDescriptorCurrentKey]
if !exists {
return nil, fmt.Errorf(
"parsing file descriptor information: %w, key=%s",
ErrKeyNotFound,
fileDescriptorCurrentKey,
)
}
return &FileDescriptorInformation{fileDescriptorMaximum, fileDescriptorCurrent}, nil
}
type MemoryInformation struct {
MemoryMaximum uint64 // size in KB
MemoryCurrent uint64 // size in KB
}
func ParseMemoryInformationFromKV(
output string,
memoryMaximumKey string,
memoryAvailableKey string,
mapper func(field string) (uint64, error),
) (*MemoryInformation, error) {
pairs := findColonSeparatedPairs(output, []string{memoryMaximumKey, memoryAvailableKey}, mapper)
memoryMaximum, exists := pairs[memoryMaximumKey]
if !exists {
return nil, fmt.Errorf("parsing memory information: %w, key=%s", ErrKeyNotFound, memoryMaximumKey)
}
memoryAvailable, exists := pairs[memoryAvailableKey]
if !exists {
return nil, fmt.Errorf("parsing memory information: %w, key=%s", ErrKeyNotFound, memoryAvailableKey)
}
memoryCurrent := memoryMaximum - memoryAvailable
return &MemoryInformation{memoryMaximum, memoryCurrent}, nil
}
func RawSystemInformation(osInfoRaw string, memoryInfoRaw string, fdInfoRaw string, disksRaw string) string {
var builder strings.Builder
formatInfo := func(info string, builder *strings.Builder) {
if info == "" {
builder.WriteString("No information\n")
} else {
builder.WriteString(info)
builder.WriteString("\n")
}
}
builder.WriteString("---BEGIN Operating system information\n")
formatInfo(osInfoRaw, &builder)
builder.WriteString("---END Operating system information\n")
builder.WriteString("---BEGIN Memory information\n")
formatInfo(memoryInfoRaw, &builder)
builder.WriteString("---END Memory information\n")
builder.WriteString("---BEGIN File descriptors information\n")
formatInfo(fdInfoRaw, &builder)
builder.WriteString("---END File descriptors information\n")
builder.WriteString("---BEGIN Disks information\n")
formatInfo(disksRaw, &builder)
builder.WriteString("---END Disks information\n")
rawInformation := builder.String()
return rawInformation
}
func collectDiskVolumeInformationUnix(ctx context.Context) ([]*DiskVolumeInformation, string, error) {
command := exec.CommandContext(ctx, "df", "-k")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
disks, err := ParseDiskVolumeInformationOutput(output, 1, 1)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return disks, output, nil
}
func collectOSInformationUnix(ctx context.Context) (*OsInfo, string, error) {
command := exec.CommandContext(ctx, "uname", "-a")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
osInfo, err := ParseUnameOutput(output, runtime.GOOS)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return osInfo, output, nil
}
+183
View File
@@ -0,0 +1,183 @@
//go:build windows
package diagnostic
import (
"context"
"fmt"
"os/exec"
"runtime"
"strconv"
)
const kiloBytesScale = 1.0 / 1024
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformation(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformation(ctx)
var memoryMaximum, memoryCurrent, fileDescriptorMaximum, fileDescriptorCurrent uint64
var osSystem, name, osVersion, osRelease, architecture string
err := SystemInformationGeneralError{
OperatingSystemInformationError: nil,
MemoryInformationError: nil,
FileDescriptorsInformationError: nil,
DiskVolumeInformationError: nil,
}
if memoryInfoErr != nil {
err.MemoryInformationError = SystemInformationError{
Err: memoryInfoErr,
RawInfo: memoryInfoRaw,
}
} else {
memoryMaximum = memoryInfo.MemoryMaximum
memoryCurrent = memoryInfo.MemoryCurrent
}
if diskErr != nil {
err.DiskVolumeInformationError = SystemInformationError{
Err: diskErr,
RawInfo: disksRaw,
}
}
if osInfoErr != nil {
err.OperatingSystemInformationError = SystemInformationError{
Err: osInfoErr,
RawInfo: osInfoRaw,
}
} else {
osSystem = osInfo.OsSystem
name = osInfo.Name
osVersion = osInfo.OsVersion
osRelease = osInfo.OsRelease
architecture = osInfo.Architecture
}
cloudflaredVersion := collector.version
info := NewSystemInformation(
memoryMaximum,
memoryCurrent,
fileDescriptorMaximum,
fileDescriptorCurrent,
osSystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
runtime.Version(),
runtime.GOARCH,
disks,
)
return info, err
}
func collectMemoryInformation(ctx context.Context) (*MemoryInformation, string, error) {
const (
memoryTotalPrefix = "TotalVirtualMemorySize"
memoryAvailablePrefix = "FreeVirtualMemory"
)
command := exec.CommandContext(
ctx,
"powershell",
"-Command",
"Get-CimInstance -Class Win32_OperatingSystem | Select-Object FreeVirtualMemory, TotalVirtualMemorySize | Format-List",
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
// the result of the command above will return values in bytes hence
// they need to be converted to kilobytes
mapper := func(field string) (uint64, error) {
value, err := strconv.ParseUint(field, 10, 64)
return uint64(float64(value) * kiloBytesScale), err
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memoryTotalPrefix, memoryAvailablePrefix, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
func collectDiskVolumeInformation(ctx context.Context) ([]*DiskVolumeInformation, string, error) {
command := exec.CommandContext(
ctx,
"powershell", "-Command", "Get-CimInstance -Class Win32_LogicalDisk | Select-Object DeviceID, Size, FreeSpace")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
disks, err := ParseDiskVolumeInformationOutput(output, 2, kiloBytesScale)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return disks, output, nil
}
func collectOSInformation(ctx context.Context) (*OsInfo, string, error) {
const (
architecturePrefix = "OSArchitecture"
osSystemPrefix = "Caption"
osVersionPrefix = "Version"
osReleasePrefix = "BuildNumber"
namePrefix = "CSName"
)
command := exec.CommandContext(
ctx,
"powershell",
"-Command",
"Get-CimInstance -Class Win32_OperatingSystem | Select-Object OSArchitecture, Caption, Version, BuildNumber, CSName | Format-List",
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
osInfo, err := ParseWinOperatingSystemInfo(output, architecturePrefix, osSystemPrefix, osVersionPrefix, osReleasePrefix, namePrefix)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return osInfo, output, nil
}
+11 -14
View File
@@ -28,10 +28,8 @@ type icmpProxy struct {
srcFunnelTracker *packet.FunnelTracker
echoIDTracker *echoIDTracker
conn *icmp.PacketConn
// Response is handled in one-by-one, so encoder can be shared between funnels
encoder *packet.Encoder
logger *zerolog.Logger
idleTimeout time.Duration
logger *zerolog.Logger
idleTimeout time.Duration
}
// echoIDTracker tracks which ID has been assigned. It first loops through assignment from lastAssignment to then end,
@@ -114,8 +112,8 @@ func (snf echoFunnelID) String() string {
return strconv.FormatUint(uint64(snf), 10)
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
conn, err := newICMPConn(listenIP, zone)
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
conn, err := newICMPConn(listenIP)
if err != nil {
return nil, err
}
@@ -123,16 +121,15 @@ func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idle
return &icmpProxy{
srcFunnelTracker: packet.NewFunnelTracker(),
echoIDTracker: newEchoIDTracker(),
encoder: packet.NewEncoder(),
conn: conn,
logger: logger,
idleTimeout: idleTimeout,
}, nil
}
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
_, span := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
_, span := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -154,7 +151,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
}
span.SetAttributes(attribute.Int("assignedEchoID", int(assignedEchoID)))
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder.datagramMuxer, pk, originalEcho.ID)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder, pk, originalEcho.ID)
newFunnelFunc := func() (packet.Funnel, error) {
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -164,7 +161,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
ip.echoIDTracker.release(echoIDTrackerKey, assignedEchoID)
return nil
}
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, ip.conn, responder, int(assignedEchoID), originalEcho.ID, ip.encoder)
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, ip.conn, responder, int(assignedEchoID), originalEcho.ID)
return icmpFlow, nil
}
funnelID := echoFunnelID(assignedEchoID)
@@ -265,8 +262,8 @@ func (ip *icmpProxy) sendReply(ctx context.Context, reply *echoReply) error {
return err
}
_, span := icmpFlow.responder.replySpan(ctx, ip.logger)
defer icmpFlow.responder.exportSpan()
_, span := icmpFlow.responder.ReplySpan(ctx, ip.logger)
defer icmpFlow.responder.ExportSpan()
if err := icmpFlow.returnToSrc(reply); err != nil {
tracing.EndWithErrorStatus(span, err)
+2 -2
View File
@@ -18,7 +18,7 @@ var errICMPProxyNotImplemented = fmt.Errorf("ICMP proxy is not implemented on %s
type icmpProxy struct{}
func (ip icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ip icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
return errICMPProxyNotImplemented
}
@@ -26,6 +26,6 @@ func (ip *icmpProxy) Serve(ctx context.Context) error {
return errICMPProxyNotImplemented
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
return nil, errICMPProxyNotImplemented
}
+12 -14
View File
@@ -37,25 +37,23 @@ var (
type icmpProxy struct {
srcFunnelTracker *packet.FunnelTracker
listenIP netip.Addr
ipv6Zone string
logger *zerolog.Logger
idleTimeout time.Duration
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
if err := testPermission(listenIP, zone, logger); err != nil {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
if err := testPermission(listenIP, logger); err != nil {
return nil, err
}
return &icmpProxy{
srcFunnelTracker: packet.NewFunnelTracker(),
listenIP: listenIP,
ipv6Zone: zone,
logger: logger,
idleTimeout: idleTimeout,
}, nil
}
func testPermission(listenIP netip.Addr, zone string, logger *zerolog.Logger) error {
func testPermission(listenIP netip.Addr, logger *zerolog.Logger) error {
// Opens a non-privileged ICMP socket. On Linux the group ID of the process needs to be in ping_group_range
// Only check ping_group_range once for IPv4
if listenIP.Is4() {
@@ -64,7 +62,7 @@ func testPermission(listenIP netip.Addr, zone string, logger *zerolog.Logger) er
return err
}
}
conn, err := newICMPConn(listenIP, zone)
conn, err := newICMPConn(listenIP)
if err != nil {
return err
}
@@ -98,9 +96,9 @@ func checkInPingGroup() error {
return fmt.Errorf("did not find group range in %s", pingGroupPath)
}
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
ctx, span := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
ctx, span := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -109,9 +107,9 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
}
observeICMPRequest(ip.logger, span, pk.Src.String(), pk.Dst.String(), originalEcho.ID, originalEcho.Seq)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder.datagramMuxer, pk, originalEcho.ID)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder, pk, originalEcho.ID)
newFunnelFunc := func() (packet.Funnel, error) {
conn, err := newICMPConn(ip.listenIP, ip.ipv6Zone)
conn, err := newICMPConn(ip.listenIP)
if err != nil {
tracing.EndWithErrorStatus(span, err)
return nil, errors.Wrap(err, "failed to open ICMP socket")
@@ -127,7 +125,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
span.SetAttributes(attribute.Int("port", localUDPAddr.Port))
echoID := localUDPAddr.Port
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, conn, responder, echoID, originalEcho.ID, packet.NewEncoder())
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, conn, responder, echoID, originalEcho.ID)
return icmpFlow, nil
}
funnelID := flow3Tuple{
@@ -181,8 +179,8 @@ func (ip *icmpProxy) listenResponse(ctx context.Context, flow *icmpEchoFlow) {
// Listens for ICMP response and handles error logging
func (ip *icmpProxy) handleResponse(ctx context.Context, flow *icmpEchoFlow, buf []byte) (done bool) {
_, span := flow.responder.replySpan(ctx, ip.logger)
defer flow.responder.exportSpan()
_, span := flow.responder.ReplySpan(ctx, ip.logger)
defer flow.responder.ExportSpan()
span.SetAttributes(
attribute.Int("originalEchoID", flow.originalEchoID),
+9 -19
View File
@@ -18,15 +18,11 @@ import (
)
// Opens a non-privileged ICMP socket on Linux and Darwin
func newICMPConn(listenIP netip.Addr, zone string) (*icmp.PacketConn, error) {
func newICMPConn(listenIP netip.Addr) (*icmp.PacketConn, error) {
if listenIP.Is4() {
return icmp.ListenPacket("udp4", listenIP.String())
}
listenAddr := listenIP.String()
if zone != "" {
listenAddr = listenAddr + "%" + zone
}
return icmp.ListenPacket("udp6", listenAddr)
return icmp.ListenPacket("udp6", listenIP.String())
}
func netipAddr(addr net.Addr) (netip.Addr, bool) {
@@ -34,7 +30,8 @@ func netipAddr(addr net.Addr) (netip.Addr, bool) {
if !ok {
return netip.Addr{}, false
}
return netip.AddrFromSlice(udpAddr.IP)
return udpAddr.AddrPort().Addr(), true
}
type flow3Tuple struct {
@@ -50,14 +47,12 @@ type icmpEchoFlow struct {
closed *atomic.Bool
src netip.Addr
originConn *icmp.PacketConn
responder *packetResponder
responder ICMPResponder
assignedEchoID int
originalEchoID int
// it's up to the user to ensure respEncoder is not used concurrently
respEncoder *packet.Encoder
}
func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icmp.PacketConn, responder *packetResponder, assignedEchoID, originalEchoID int, respEncoder *packet.Encoder) *icmpEchoFlow {
func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icmp.PacketConn, responder ICMPResponder, assignedEchoID, originalEchoID int) *icmpEchoFlow {
return &icmpEchoFlow{
ActivityTracker: packet.NewActivityTracker(),
closeCallback: closeCallback,
@@ -67,7 +62,6 @@ func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icm
responder: responder,
assignedEchoID: assignedEchoID,
originalEchoID: originalEchoID,
respEncoder: respEncoder,
}
}
@@ -139,11 +133,7 @@ func (ief *icmpEchoFlow) returnToSrc(reply *echoReply) error {
},
Message: reply.msg,
}
serializedPacket, err := ief.respEncoder.Encode(&pk)
if err != nil {
return err
}
return ief.responder.returnPacket(serializedPacket)
return ief.responder.ReturnPacket(&pk)
}
type echoReply struct {
@@ -184,7 +174,7 @@ func toICMPEchoFlow(funnel packet.Funnel) (*icmpEchoFlow, error) {
return icmpFlow, nil
}
func createShouldReplaceFunnelFunc(logger *zerolog.Logger, muxer muxer, pk *packet.ICMP, originalEchoID int) func(packet.Funnel) bool {
func createShouldReplaceFunnelFunc(logger *zerolog.Logger, responder ICMPResponder, pk *packet.ICMP, originalEchoID int) func(packet.Funnel) bool {
return func(existing packet.Funnel) bool {
existingFlow, err := toICMPEchoFlow(existing)
if err != nil {
@@ -199,7 +189,7 @@ func createShouldReplaceFunnelFunc(logger *zerolog.Logger, muxer muxer, pk *pack
// If the existing flow has a different muxer, there's a new quic connection where return packets should be
// routed. Otherwise, return packets will be send to the first observed incoming connection, rather than the
// most recently observed connection.
if existingFlow.responder.datagramMuxer != muxer {
if existingFlow.responder.ConnectionIndex() != responder.ConnectionIndex() {
logger.Debug().
Str("src", pk.Src.String()).
Str("dst", pk.Dst.String()).
+11 -20
View File
@@ -27,7 +27,7 @@ func TestFunnelIdleTimeout(t *testing.T) {
startSeq = 8129
)
logger := zerolog.New(os.Stderr)
proxy, err := newICMPProxy(localhostIP, "", &logger, idleTimeout)
proxy, err := newICMPProxy(localhostIP, &logger, idleTimeout)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -56,24 +56,19 @@ func TestFunnelIdleTimeout(t *testing.T) {
},
}
muxer := newMockMuxer(0)
responder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &responder))
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
// Send second request, should reuse the funnel
require.NoError(t, proxy.Request(ctx, &pk, &packetResponder{
datagramMuxer: muxer,
}))
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
// New muxer on a different connection should use a new flow
time.Sleep(idleTimeout * 2)
newMuxer := newMockMuxer(0)
newResponder := packetResponder{
datagramMuxer: newMuxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &newResponder))
newResponder := newPacketResponder(newMuxer, 1, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, newResponder))
validateEchoFlow(t, <-newMuxer.cfdToEdge, &pk)
time.Sleep(idleTimeout * 2)
@@ -90,7 +85,7 @@ func TestReuseFunnel(t *testing.T) {
startSeq = 8129
)
logger := zerolog.New(os.Stderr)
proxy, err := newICMPProxy(localhostIP, "", &logger, idleTimeout)
proxy, err := newICMPProxy(localhostIP, &logger, idleTimeout)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -124,18 +119,14 @@ func TestReuseFunnel(t *testing.T) {
originalEchoID: echoID,
}
muxer := newMockMuxer(0)
responder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &responder))
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
funnel1, found := getFunnel(t, proxy, tuple)
require.True(t, found)
// Send second request, should reuse the funnel
require.NoError(t, proxy.Request(ctx, &pk, &packetResponder{
datagramMuxer: muxer,
}))
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
funnel2, found := getFunnel(t, proxy, tuple)
require.True(t, found)
+8 -30
View File
@@ -13,7 +13,6 @@ import (
"fmt"
"net/netip"
"runtime/debug"
"sync"
"syscall"
"time"
"unsafe"
@@ -222,11 +221,9 @@ type icmpProxy struct {
// This is a ICMPv6 if srcSocketAddr is not nil
srcSocketAddr *sockAddrIn6
logger *zerolog.Logger
// A pool of reusable *packet.Encoder
encoderPool sync.Pool
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
var (
srcSocketAddr *sockAddrIn6
handle uintptr
@@ -250,11 +247,6 @@ func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idle
handle: handle,
srcSocketAddr: srcSocketAddr,
logger: logger,
encoderPool: sync.Pool{
New: func() any {
return packet.NewEncoder()
},
},
}, nil
}
@@ -267,15 +259,15 @@ func (ip *icmpProxy) Serve(ctx context.Context) error {
// Request sends an ICMP echo request and wait for a reply or timeout.
// The async version of Win32 APIs take a callback whose memory is not garbage collected, so we use the synchronous version.
// It's possible that a slow request will block other requests, so we set the timeout to only 1s.
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
defer func() {
if r := recover(); r != nil {
ip.logger.Error().Interface("error", r).Msgf("Recover panic from sending icmp request/response, error %s", debug.Stack())
}
}()
_, requestSpan := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
_, requestSpan := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
echo, err := getICMPEcho(pk.Message)
if err != nil {
@@ -290,9 +282,9 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
return err
}
tracing.End(requestSpan)
responder.exportSpan()
responder.ExportSpan()
_, replySpan := responder.replySpan(ctx, ip.logger)
_, replySpan := responder.ReplySpan(ctx, ip.logger)
err = ip.handleEchoReply(pk, echo, resp, responder)
if err != nil {
ip.logger.Err(err).Msg("Failed to send ICMP reply")
@@ -308,7 +300,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
return nil
}
func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, resp echoResp, responder *packetResponder) error {
func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, resp echoResp, responder ICMPResponder) error {
var replyType icmp.Type
if request.Dst.Is4() {
replyType = ipv4.ICMPTypeEchoReply
@@ -333,21 +325,7 @@ func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, r
},
},
}
cachedEncoder := ip.encoderPool.Get()
// The encoded packet is a slice to of the encoder, so we shouldn't return the encoder back to the pool until
// the encoded packet is sent.
defer ip.encoderPool.Put(cachedEncoder)
encoder, ok := cachedEncoder.(*packet.Encoder)
if !ok {
return fmt.Errorf("encoderPool returned %T, expect *packet.Encoder", cachedEncoder)
}
serializedPacket, err := encoder.Encode(&pk)
if err != nil {
return err
}
return responder.returnPacket(serializedPacket)
return responder.ReturnPacket(&pk)
}
func (ip *icmpProxy) icmpEchoRoundtrip(dst netip.Addr, echo *icmp.Echo) (echoResp, error) {
+1 -1
View File
@@ -132,7 +132,7 @@ func TestSendEchoErrors(t *testing.T) {
}
func testSendEchoErrors(t *testing.T, listenIP netip.Addr) {
proxy, err := newICMPProxy(listenIP, "", &noopLogger, time.Second)
proxy, err := newICMPProxy(listenIP, &noopLogger, time.Second)
require.NoError(t, err)
echo := icmp.Echo{
+46 -4
View File
@@ -14,6 +14,7 @@ import (
"golang.org/x/net/ipv6"
"github.com/cloudflare/cloudflared/packet"
"github.com/cloudflare/cloudflared/tracing"
)
const (
@@ -26,17 +27,46 @@ var (
errPacketNil = fmt.Errorf("packet is nil")
)
// ICMPRouterServer is a parent interface over-top of ICMPRouter that allows for the operation of the proxy origin listeners.
type ICMPRouterServer interface {
ICMPRouter
// Serve runs the ICMPRouter proxy origin listeners for any of the IPv4 or IPv6 interfaces configured.
Serve(ctx context.Context) error
}
// ICMPRouter manages out-going ICMP requests towards the origin.
type ICMPRouter interface {
// Request will send an ICMP packet towards the origin with an ICMPResponder to attach to the ICMP flow for the
// response to utilize.
Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error
// ConvertToTTLExceeded will take an ICMP packet and create a ICMP TTL Exceeded response origininating from the
// ICMPRouter's IP interface.
ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP
}
// ICMPResponder manages how to handle incoming ICMP messages coming from the origin to the edge.
type ICMPResponder interface {
ConnectionIndex() uint8
ReturnPacket(pk *packet.ICMP) error
AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte)
RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span)
ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span)
ExportSpan()
}
type icmpRouter struct {
ipv4Proxy *icmpProxy
ipv4Src netip.Addr
ipv6Proxy *icmpProxy
ipv6Src netip.Addr
}
// NewICMPRouter doesn't return an error if either ipv4 proxy or ipv6 proxy can be created. The machine might only
// support one of them.
// funnelIdleTimeout controls how long to wait to close a funnel without send/return
func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, ipv6Zone string, logger *zerolog.Logger, funnelIdleTimeout time.Duration) (*icmpRouter, error) {
ipv4Proxy, ipv4Err := newICMPProxy(ipv4Addr, "", logger, funnelIdleTimeout)
ipv6Proxy, ipv6Err := newICMPProxy(ipv6Addr, ipv6Zone, logger, funnelIdleTimeout)
func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, logger *zerolog.Logger, funnelIdleTimeout time.Duration) (ICMPRouterServer, error) {
ipv4Proxy, ipv4Err := newICMPProxy(ipv4Addr, logger, funnelIdleTimeout)
ipv6Proxy, ipv6Err := newICMPProxy(ipv6Addr, logger, funnelIdleTimeout)
if ipv4Err != nil && ipv6Err != nil {
err := fmt.Errorf("cannot create ICMPv4 proxy: %v nor ICMPv6 proxy: %v", ipv4Err, ipv6Err)
logger.Debug().Err(err).Msg("ICMP proxy feature is disabled")
@@ -52,7 +82,9 @@ func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, ipv6Zone string, logger *zerol
}
return &icmpRouter{
ipv4Proxy: ipv4Proxy,
ipv4Src: ipv4Addr,
ipv6Proxy: ipv6Proxy,
ipv6Src: ipv6Addr,
}, nil
}
@@ -76,7 +108,7 @@ func (ir *icmpRouter) Serve(ctx context.Context) error {
return fmt.Errorf("ICMPv4 proxy and ICMPv6 proxy are both nil")
}
func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
if pk == nil {
return errPacketNil
}
@@ -92,6 +124,16 @@ func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder *p
return fmt.Errorf("ICMPv6 proxy was not instantiated")
}
func (ir *icmpRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
var srcIP netip.Addr
if pk.Dst.Is4() {
srcIP = ir.ipv4Src
} else {
srcIP = ir.ipv6Src
}
return packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, srcIP)
}
func getICMPEcho(msg *icmp.Message) (*icmp.Echo, error) {
echo, ok := msg.Body.(*icmp.Echo)
if !ok {
+17 -30
View File
@@ -50,7 +50,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
endSeq = 20
)
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -61,9 +61,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
}()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
protocol := layers.IPProtocolICMPv6
if sendIPv4 {
@@ -98,7 +96,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
},
},
}
require.NoError(t, router.Request(ctx, &pk, &responder))
require.NoError(t, router.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
}
}
@@ -114,7 +112,7 @@ func TestTraceICMPRouterEcho(t *testing.T) {
tracingCtx := "ec31ad8a01fde11fdcabe2efdce36873:52726f6cabc144f5:0:1"
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -131,11 +129,8 @@ func TestTraceICMPRouterEcho(t *testing.T) {
serializedIdentity, err := tracingIdentity.MarshalBinary()
require.NoError(t, err)
responder := packetResponder{
datagramMuxer: muxer,
tracedCtx: tracing.NewTracedContext(ctx, tracingIdentity.String(), &noopLogger),
serializedIdentity: serializedIdentity,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
responder.AddTraceContext(tracing.NewTracedContext(ctx, tracingIdentity.String(), &noopLogger), serializedIdentity)
echo := &icmp.Echo{
ID: 12910,
@@ -156,7 +151,7 @@ func TestTraceICMPRouterEcho(t *testing.T) {
},
}
require.NoError(t, router.Request(ctx, &pk, &responder))
require.NoError(t, router.Request(ctx, &pk, responder))
firstPK := <-muxer.cfdToEdge
var requestSpan *quicpogs.TracingSpanPacket
// The order of receiving reply or request span is not deterministic
@@ -194,10 +189,8 @@ func TestTraceICMPRouterEcho(t *testing.T) {
echo.Seq++
pk.Body = echo
// Only first request for a flow is traced. The edge will not send tracing context for the second request
newResponder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, router.Request(ctx, &pk, &newResponder))
newResponder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, router.Request(ctx, &pk, newResponder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
select {
@@ -221,7 +214,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
endSeq = 5
)
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -240,9 +233,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
defer wg.Done()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
for seq := 0; seq < endSeq; seq++ {
pk := &packet.ICMP{
IP: &packet.IP{
@@ -261,16 +252,14 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
},
},
}
require.NoError(t, router.Request(ctx, pk, &responder))
require.NoError(t, router.Request(ctx, pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, pk)
}
}()
go func() {
defer wg.Done()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
for seq := 0; seq < endSeq; seq++ {
pk := &packet.ICMP{
IP: &packet.IP{
@@ -289,7 +278,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
},
},
}
require.NoError(t, router.Request(ctx, pk, &responder))
require.NoError(t, router.Request(ctx, pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, pk)
}
}()
@@ -358,13 +347,11 @@ func TestICMPRouterRejectNotEcho(t *testing.T) {
}
func testICMPRouterRejectNotEcho(t *testing.T, srcDstIP netip.Addr, msgs []icmp.Message) {
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
protocol := layers.IPProtocolICMPv4
if srcDstIP.Is6() {
protocol = layers.IPProtocolICMPv6
@@ -379,7 +366,7 @@ func testICMPRouterRejectNotEcho(t *testing.T, srcDstIP netip.Addr, msgs []icmp.
},
Message: &m,
}
require.Error(t, router.Request(context.Background(), &pk, &responder))
require.Error(t, router.Request(context.Background(), &pk, responder))
}
}
+53 -45
View File
@@ -3,7 +3,6 @@ package ingress
import (
"context"
"fmt"
"net/netip"
"github.com/rs/zerolog"
"go.opentelemetry.io/otel/attribute"
@@ -23,29 +22,23 @@ type muxer interface {
// PacketRouter routes packets between Upstream and ICMPRouter. Currently it rejects all other type of ICMP packets
type PacketRouter struct {
globalConfig *GlobalRouterConfig
muxer muxer
logger *zerolog.Logger
icmpDecoder *packet.ICMPDecoder
encoder *packet.Encoder
}
// GlobalRouterConfig is the configuration shared by all instance of Router.
type GlobalRouterConfig struct {
ICMPRouter *icmpRouter
IPv4Src netip.Addr
IPv6Src netip.Addr
Zone string
icmpRouter ICMPRouter
muxer muxer
connIndex uint8
logger *zerolog.Logger
encoder *packet.Encoder
decoder *packet.ICMPDecoder
}
// NewPacketRouter creates a PacketRouter that handles ICMP packets. Packets are read from muxer but dropped if globalConfig is nil.
func NewPacketRouter(globalConfig *GlobalRouterConfig, muxer muxer, logger *zerolog.Logger) *PacketRouter {
func NewPacketRouter(icmpRouter ICMPRouter, muxer muxer, connIndex uint8, logger *zerolog.Logger) *PacketRouter {
return &PacketRouter{
globalConfig: globalConfig,
muxer: muxer,
logger: logger,
icmpDecoder: packet.NewICMPDecoder(),
encoder: packet.NewEncoder(),
icmpRouter: icmpRouter,
muxer: muxer,
connIndex: connIndex,
logger: logger,
encoder: packet.NewEncoder(),
decoder: packet.NewICMPDecoder(),
}
}
@@ -59,14 +52,13 @@ func (r *PacketRouter) Serve(ctx context.Context) error {
}
}
func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packetResponder, error) {
func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, ICMPResponder, error) {
pk, err := r.muxer.ReceivePacket(ctx)
if err != nil {
return packet.RawPacket{}, nil, err
}
responder := &packetResponder{
datagramMuxer: r.muxer,
}
responder := newPacketResponder(r.muxer, r.connIndex, packet.NewEncoder())
switch pk.Type() {
case quicpogs.DatagramTypeIP:
return packet.RawPacket{Data: pk.Payload()}, responder, nil
@@ -75,8 +67,8 @@ func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packe
if err := identity.UnmarshalBinary(pk.Metadata()); err != nil {
r.logger.Err(err).Bytes("tracingIdentity", pk.Metadata()).Msg("Failed to unmarshal tracing identity")
} else {
responder.tracedCtx = tracing.NewTracedContext(ctx, identity.String(), r.logger)
responder.serializedIdentity = pk.Metadata()
tracedCtx := tracing.NewTracedContext(ctx, identity.String(), r.logger)
responder.AddTraceContext(tracedCtx, pk.Metadata())
}
return packet.RawPacket{Data: pk.Payload()}, responder, nil
default:
@@ -84,27 +76,27 @@ func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packe
}
}
func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPacket, responder *packetResponder) {
func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPacket, responder ICMPResponder) {
// ICMP Proxy feature is disabled, drop packets
if r.globalConfig == nil {
if r.icmpRouter == nil {
return
}
icmpPacket, err := r.icmpDecoder.Decode(rawPacket)
icmpPacket, err := r.decoder.Decode(rawPacket)
if err != nil {
r.logger.Err(err).Msg("Failed to decode ICMP packet from quic datagram")
return
}
if icmpPacket.TTL <= 1 {
if err := r.sendTTLExceedMsg(ctx, icmpPacket, rawPacket, r.encoder); err != nil {
if err := r.sendTTLExceedMsg(icmpPacket, rawPacket); err != nil {
r.logger.Err(err).Msg("Failed to return ICMP TTL exceed error")
}
return
}
icmpPacket.TTL--
if err := r.globalConfig.ICMPRouter.Request(ctx, icmpPacket, responder); err != nil {
if err := r.icmpRouter.Request(ctx, icmpPacket, responder); err != nil {
r.logger.Err(err).
Str("src", icmpPacket.Src.String()).
Str("dst", icmpPacket.Dst.String()).
@@ -113,16 +105,9 @@ func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPac
}
}
func (r *PacketRouter) sendTTLExceedMsg(ctx context.Context, pk *packet.ICMP, rawPacket packet.RawPacket, encoder *packet.Encoder) error {
var srcIP netip.Addr
if pk.Dst.Is4() {
srcIP = r.globalConfig.IPv4Src
} else {
srcIP = r.globalConfig.IPv6Src
}
ttlExceedPacket := packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, srcIP)
encodedTTLExceed, err := encoder.Encode(ttlExceedPacket)
func (r *PacketRouter) sendTTLExceedMsg(pk *packet.ICMP, rawPacket packet.RawPacket) error {
icmpTTLPacket := r.icmpRouter.ConvertToTTLExceeded(pk, rawPacket)
encodedTTLExceed, err := r.encoder.Encode(icmpTTLPacket)
if err != nil {
return err
}
@@ -132,22 +117,45 @@ func (r *PacketRouter) sendTTLExceedMsg(ctx context.Context, pk *packet.ICMP, ra
// packetResponder should not be used concurrently. This assumption is upheld because reply packets are ready one-by-one
type packetResponder struct {
datagramMuxer muxer
connIndex uint8
encoder *packet.Encoder
tracedCtx *tracing.TracedContext
serializedIdentity []byte
// hadReply tracks if there has been any reply for this flow
hadReply bool
}
func newPacketResponder(datagramMuxer muxer, connIndex uint8, encoder *packet.Encoder) ICMPResponder {
return &packetResponder{
datagramMuxer: datagramMuxer,
connIndex: connIndex,
encoder: encoder,
}
}
func (pr *packetResponder) tracingEnabled() bool {
return pr.tracedCtx != nil
}
func (pr *packetResponder) returnPacket(rawPacket packet.RawPacket) error {
func (pr *packetResponder) ConnectionIndex() uint8 {
return pr.connIndex
}
func (pr *packetResponder) ReturnPacket(pk *packet.ICMP) error {
rawPacket, err := pr.encoder.Encode(pk)
if err != nil {
return err
}
pr.hadReply = true
return pr.datagramMuxer.SendPacket(quicpogs.RawPacket(rawPacket))
}
func (pr *packetResponder) requestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
func (pr *packetResponder) AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte) {
pr.tracedCtx = tracedCtx
pr.serializedIdentity = serializedIdentity
}
func (pr *packetResponder) RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
if !pr.tracingEnabled() {
return ctx, tracing.NewNoopSpan()
}
@@ -157,14 +165,14 @@ func (pr *packetResponder) requestSpan(ctx context.Context, pk *packet.ICMP) (co
))
}
func (pr *packetResponder) replySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
func (pr *packetResponder) ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
if !pr.tracingEnabled() || pr.hadReply {
return ctx, tracing.NewNoopSpan()
}
return pr.tracedCtx.Tracer().Start(pr.tracedCtx, "icmp-echo-reply")
}
func (pr *packetResponder) exportSpan() {
func (pr *packetResponder) ExportSpan() {
if !pr.tracingEnabled() {
return
}
+8 -7
View File
@@ -19,16 +19,17 @@ import (
)
var (
packetConfig = &GlobalRouterConfig{
ICMPRouter: nil,
IPv4Src: netip.MustParseAddr("172.16.0.1"),
IPv6Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
defaultRouter = &icmpRouter{
ipv4Proxy: nil,
ipv4Src: netip.MustParseAddr("172.16.0.1"),
ipv6Proxy: nil,
ipv6Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
}
)
func TestRouterReturnTTLExceed(t *testing.T) {
muxer := newMockMuxer(0)
router := NewPacketRouter(packetConfig, muxer, &noopLogger)
router := NewPacketRouter(defaultRouter, muxer, 0, &noopLogger)
ctx, cancel := context.WithCancel(context.Background())
routerStopped := make(chan struct{})
go func() {
@@ -53,7 +54,7 @@ func TestRouterReturnTTLExceed(t *testing.T) {
},
},
}
assertTTLExceed(t, &pk, router.globalConfig.IPv4Src, muxer)
assertTTLExceed(t, &pk, defaultRouter.ipv4Src, muxer)
pk = packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
@@ -71,7 +72,7 @@ func TestRouterReturnTTLExceed(t *testing.T) {
},
},
}
assertTTLExceed(t, &pk, router.globalConfig.IPv6Src, muxer)
assertTTLExceed(t, &pk, defaultRouter.ipv6Src, muxer)
cancel()
<-routerStopped
+3 -3
View File
@@ -76,10 +76,10 @@ func CreateConfig(
var file *FileConfig
var rolling *RollingConfig
if rollingLogPath != "" {
rolling = createRollingConfig(rollingLogPath)
} else if nonRollingLogFilePath != "" {
if nonRollingLogFilePath != "" {
file = createFileConfig(nonRollingLogFilePath)
} else if rollingLogPath != "" {
rolling = createRollingConfig(rollingLogPath)
}
if minLevel == "" {
+70
View File
@@ -10,10 +10,13 @@ import (
"sync"
"time"
"github.com/facebookgo/grace/gracenet"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/rs/zerolog"
"golang.org/x/net/trace"
"github.com/cloudflare/cloudflared/diagnostic"
)
const (
@@ -21,8 +24,37 @@ const (
defaultShutdownTimeout = time.Second * 15
)
// This variable is set at compile time to allow the default local address to change.
var Runtime = "host"
func GetMetricsDefaultAddress(runtimeType string) string {
// When issuing the diagnostic command we may have to reach a server that is
// running in a virtual environment and in that case we must bind to 0.0.0.0
// otherwise the server won't be reachable.
switch runtimeType {
case "virtual":
return "0.0.0.0:0"
default:
return "localhost:0"
}
}
// GetMetricsKnownAddresses returns the addresses used by the metrics server to bind at
// startup time to allow a semi-deterministic approach to know where the server is listening at.
// The ports were selected because at the time we are in 2024 and they do not collide with any
// know/registered port according https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers.
func GetMetricsKnownAddresses(runtimeType string) []string {
switch runtimeType {
case "virtual":
return []string{"0.0.0.0:20241", "0.0.0.0:20242", "0.0.0.0:20243", "0.0.0.0:20244", "0.0.0.0:20245"}
default:
return []string{"localhost:20241", "localhost:20242", "localhost:20243", "localhost:20244", "localhost:20245"}
}
}
type Config struct {
ReadyServer *ReadyServer
DiagnosticHandler *diagnostic.Handler
QuickTunnelHostname string
Orchestrator orchestrator
@@ -62,9 +94,47 @@ func newMetricsHandler(
})
}
config.DiagnosticHandler.InstallEndpoints(router)
return router
}
// CreateMetricsListener will create a new [net.Listener] by using an
// known set of ports when the default address is passed with the fallback
// of choosing a random port when none is available.
//
// In case the provided address is not the default one then it will be used
// as is.
func CreateMetricsListener(listeners *gracenet.Net, laddr string) (net.Listener, error) {
if laddr == GetMetricsDefaultAddress(Runtime) {
// On the presence of the default address select
// a port from the known set of addresses iteratively.
addresses := GetMetricsKnownAddresses(Runtime)
for _, address := range addresses {
listener, err := listeners.Listen("tcp", address)
if err == nil {
return listener, nil
}
}
// When no port is available then bind to a random one
listener, err := listeners.Listen("tcp", laddr)
if err != nil {
return nil, fmt.Errorf("failed to listen to default metrics address: %w", err)
}
return listener, nil
}
// Explicitly got a local address then bind to it
listener, err := listeners.Listen("tcp", laddr)
if err != nil {
return nil, fmt.Errorf("failed to bind to address (%s): %w", laddr, err)
}
return listener, nil
}
func ServeMetrics(
l net.Listener,
ctx context.Context,
+52
View File
@@ -0,0 +1,52 @@
package metrics_test
import (
"testing"
"github.com/facebookgo/grace/gracenet"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/metrics"
)
func TestMetricsListenerCreation(t *testing.T) {
t.Parallel()
listeners := gracenet.Net{}
listener1, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20241", listener1.Addr().String())
require.NoError(t, err)
listener2, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20242", listener2.Addr().String())
require.NoError(t, err)
listener3, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20243", listener3.Addr().String())
require.NoError(t, err)
listener4, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20244", listener4.Addr().String())
require.NoError(t, err)
listener5, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20245", listener5.Addr().String())
require.NoError(t, err)
listener6, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
addresses := [5]string{"127.0.0.1:20241", "127.0.0.1:20242", "127.0.0.1:20243", "127.0.0.1:20244", "127.0.0.1:20245"}
assert.NotContains(t, addresses, listener6.Addr().String())
require.NoError(t, err)
listener7, err := metrics.CreateMetricsListener(&listeners, "localhost:12345")
assert.Equal(t, "127.0.0.1:12345", listener7.Addr().String())
require.NoError(t, err)
err = listener1.Close()
require.NoError(t, err)
err = listener2.Close()
require.NoError(t, err)
err = listener3.Close()
require.NoError(t, err)
err = listener4.Close()
require.NoError(t, err)
err = listener5.Close()
require.NoError(t, err)
err = listener6.Close()
require.NoError(t, err)
err = listener7.Close()
require.NoError(t, err)
}
+6 -9
View File
@@ -6,9 +6,7 @@ import (
"net/http"
"github.com/google/uuid"
"github.com/rs/zerolog"
conn "github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/tunnelstate"
)
@@ -19,17 +17,16 @@ type ReadyServer struct {
}
// NewReadyServer initializes a ReadyServer and starts listening for dis/connection events.
func NewReadyServer(log *zerolog.Logger, clientID uuid.UUID) *ReadyServer {
func NewReadyServer(
clientID uuid.UUID,
tracker *tunnelstate.ConnTracker,
) *ReadyServer {
return &ReadyServer{
clientID: clientID,
tracker: tunnelstate.NewConnTracker(log),
clientID,
tracker,
}
}
func (rs *ReadyServer) OnTunnelEvent(c conn.Event) {
rs.tracker.OnTunnelEvent(c)
}
type body struct {
Status int `json:"status"`
ReadyConnections uint `json:"readyConnections"`
+44 -74
View File
@@ -1,136 +1,106 @@
package metrics
package metrics_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/metrics"
"github.com/cloudflare/cloudflared/tunnelstate"
)
func TestReadyServer_makeResponse(t *testing.T) {
type fields struct {
isConnected map[uint8]tunnelstate.ConnectionInfo
}
tests := []struct {
name string
fields fields
wantOK bool
wantReadyConnections uint
}{
{
name: "One connection online => HTTP 200",
fields: fields{
isConnected: map[uint8]tunnelstate.ConnectionInfo{
0: {IsConnected: false},
1: {IsConnected: false},
2: {IsConnected: true},
3: {IsConnected: false},
},
},
wantOK: true,
wantReadyConnections: 1,
},
{
name: "No connections online => no HTTP 200",
fields: fields{
isConnected: map[uint8]tunnelstate.ConnectionInfo{
0: {IsConnected: false},
1: {IsConnected: false},
2: {IsConnected: false},
3: {IsConnected: false},
},
},
wantReadyConnections: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rs := &ReadyServer{
tracker: tunnelstate.MockedConnTracker(tt.fields.isConnected),
}
gotStatusCode, gotReadyConnections := rs.makeResponse()
if tt.wantOK && gotStatusCode != http.StatusOK {
t.Errorf("ReadyServer.makeResponse() gotStatusCode = %v, want ok = %v", gotStatusCode, tt.wantOK)
}
if gotReadyConnections != tt.wantReadyConnections {
t.Errorf("ReadyServer.makeResponse() gotReadyConnections = %v, want %v", gotReadyConnections, tt.wantReadyConnections)
}
})
func mockRequest(t *testing.T, readyServer *metrics.ReadyServer) (int, uint) {
t.Helper()
var readyreadyConnections struct {
Status int `json:"status"`
ReadyConnections uint `json:"readyConnections"`
ConnectorID uuid.UUID `json:"connectorId"`
}
rec := httptest.NewRecorder()
readyServer.ServeHTTP(rec, nil)
decoder := json.NewDecoder(rec.Body)
err := decoder.Decode(&readyreadyConnections)
require.NoError(t, err)
return rec.Code, readyreadyConnections.ReadyConnections
}
func TestReadinessEventHandling(t *testing.T) {
nopLogger := zerolog.Nop()
rs := NewReadyServer(&nopLogger, uuid.Nil)
tracker := tunnelstate.NewConnTracker(&nopLogger)
rs := metrics.NewReadyServer(uuid.Nil, tracker)
// start not ok
code, ready := rs.makeResponse()
code, readyConnections := mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// one connected => ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
assert.EqualValues(t, 1, readyConnections)
// another connected => still ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 2,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 2, ready)
assert.EqualValues(t, 2, readyConnections)
// one reconnecting => still ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 2,
EventType: connection.Reconnecting,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
assert.EqualValues(t, 1, readyConnections)
// Regression test for TUN-3777
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.RegisteringTunnel,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// other connected then unregistered => not ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
rs.OnTunnelEvent(connection.Event{
assert.EqualValues(t, 1, readyConnections)
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Unregistering,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// other disconnected => not ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Disconnected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
}
+4 -2
View File
@@ -1,6 +1,8 @@
package packet
import "github.com/google/gopacket"
import (
"github.com/google/gopacket"
)
var (
serializeOpts = gopacket.SerializeOptions{
@@ -25,7 +27,7 @@ func NewEncoder() *Encoder {
}
}
func (e Encoder) Encode(packet Packet) (RawPacket, error) {
func (e *Encoder) Encode(packet Packet) (RawPacket, error) {
encodedLayers, err := packet.EncodeLayers()
if err != nil {
return RawPacket{}, err
+63 -5
View File
@@ -222,11 +222,11 @@ const (
//
// This method should be used in-place of MarshalBinary which will allocate in-place the required byte array to return.
func MarshalPayloadHeaderTo(requestID RequestID, payload []byte) error {
if len(payload) < 17 {
if len(payload) < DatagramPayloadHeaderLen {
return wrapMarshalErr(ErrDatagramPayloadHeaderTooSmall)
}
payload[0] = byte(UDPSessionPayloadType)
return requestID.MarshalBinaryTo(payload[1:17])
return requestID.MarshalBinaryTo(payload[1:DatagramPayloadHeaderLen])
}
func (s *UDPSessionPayloadDatagram) UnmarshalBinary(data []byte) error {
@@ -239,18 +239,18 @@ func (s *UDPSessionPayloadDatagram) UnmarshalBinary(data []byte) error {
}
// Make sure that the slice provided is the right size to be parsed.
if len(data) < 17 || len(data) > maxPayloadPlusHeaderLen {
if len(data) < DatagramPayloadHeaderLen || len(data) > maxPayloadPlusHeaderLen {
return wrapUnmarshalErr(ErrDatagramPayloadInvalidSize)
}
requestID, err := RequestIDFromSlice(data[1:17])
requestID, err := RequestIDFromSlice(data[1:DatagramPayloadHeaderLen])
if err != nil {
return wrapUnmarshalErr(err)
}
*s = UDPSessionPayloadDatagram{
RequestID: requestID,
Payload: data[17:],
Payload: data[DatagramPayloadHeaderLen:],
}
return nil
}
@@ -370,3 +370,61 @@ func (s *UDPSessionRegistrationResponseDatagram) UnmarshalBinary(data []byte) er
}
return nil
}
// ICMPDatagram is used to propagate ICMPv4 and ICMPv6 payloads.
type ICMPDatagram struct {
Payload []byte
}
// The maximum size that an ICMP packet can be.
const maxICMPPayloadLen = maxDatagramPayloadLen
// The datagram structure for ICMPDatagram is:
//
// 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// 0| Type | |
// +-+-+-+-+-+-+-+-+ +
// . Payload .
// . .
// . .
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
func (d *ICMPDatagram) MarshalBinary() (data []byte, err error) {
if len(d.Payload) > maxICMPPayloadLen {
return nil, wrapMarshalErr(ErrDatagramICMPPayloadTooLarge)
}
// We shouldn't attempt to marshal an ICMP datagram with no ICMP payload provided
if len(d.Payload) == 0 {
return nil, wrapMarshalErr(ErrDatagramICMPPayloadMissing)
}
// Make room for the 1 byte ICMPType header
datagram := make([]byte, len(d.Payload)+datagramTypeLen)
datagram[0] = byte(ICMPType)
copy(datagram[1:], d.Payload)
return datagram, nil
}
func (d *ICMPDatagram) UnmarshalBinary(data []byte) error {
datagramType, err := ParseDatagramType(data)
if err != nil {
return wrapUnmarshalErr(err)
}
if datagramType != ICMPType {
return wrapUnmarshalErr(ErrInvalidDatagramType)
}
if len(data[1:]) > maxDatagramPayloadLen {
return wrapUnmarshalErr(ErrDatagramICMPPayloadTooLarge)
}
// We shouldn't attempt to unmarshal an ICMP datagram with no ICMP payload provided
if len(data[1:]) == 0 {
return wrapUnmarshalErr(ErrDatagramICMPPayloadMissing)
}
payload := make([]byte, len(data[1:]))
copy(payload, data[1:])
d.Payload = payload
return nil
}
+2
View File
@@ -15,6 +15,8 @@ var (
ErrDatagramResponseInvalidSize error = errors.New("datagram response is an invalid size")
ErrDatagramResponseMsgTooLargeMaximum error = fmt.Errorf("datagram response error message length exceeds the length of the datagram maximum: %d", maxResponseErrorMessageLen)
ErrDatagramResponseMsgTooLargeDatagram error = fmt.Errorf("datagram response error message length exceeds the length of the provided datagram")
ErrDatagramICMPPayloadTooLarge error = fmt.Errorf("datagram icmp payload exceeds %d bytes", maxICMPPayloadLen)
ErrDatagramICMPPayloadMissing error = errors.New("datagram icmp payload is missing")
)
func wrapMarshalErr(err error) error {
+70
View File
@@ -160,6 +160,12 @@ func TestTypeUnmarshalErrors(t *testing.T) {
if !errors.Is(err, v3.ErrDatagramHeaderTooSmall) {
t.Errorf("expected invalid length to throw error")
}
d4 := v3.ICMPDatagram{}
err = d4.UnmarshalBinary([]byte{})
if !errors.Is(err, v3.ErrDatagramHeaderTooSmall) {
t.Errorf("expected invalid length to throw error")
}
})
t.Run("invalid types", func(t *testing.T) {
@@ -180,6 +186,12 @@ func TestTypeUnmarshalErrors(t *testing.T) {
if !errors.Is(err, v3.ErrInvalidDatagramType) {
t.Errorf("expected invalid type to throw error")
}
d4 := v3.ICMPDatagram{}
err = d4.UnmarshalBinary([]byte{byte(v3.UDPSessionPayloadType)})
if !errors.Is(err, v3.ErrInvalidDatagramType) {
t.Errorf("expected invalid type to throw error")
}
})
}
@@ -343,6 +355,54 @@ func TestSessionRegistrationResponse(t *testing.T) {
})
}
func TestICMPDatagram(t *testing.T) {
t.Run("basic", func(t *testing.T) {
payload := makePayload(128)
datagram := v3.ICMPDatagram{Payload: payload}
marshaled, err := datagram.MarshalBinary()
if err != nil {
t.Error(err)
}
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(marshaled)
if err != nil {
t.Error(err)
}
require.Equal(t, payload, unmarshaled.Payload)
})
t.Run("payload size empty", func(t *testing.T) {
payload := []byte{}
datagram := v3.ICMPDatagram{Payload: payload}
_, err := datagram.MarshalBinary()
if !errors.Is(err, v3.ErrDatagramICMPPayloadMissing) {
t.Errorf("expected an error: %s", err)
}
payload = []byte{byte(v3.ICMPType)}
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(payload)
if !errors.Is(err, v3.ErrDatagramICMPPayloadMissing) {
t.Errorf("expected an error: %s", err)
}
})
t.Run("payload size too large", func(t *testing.T) {
payload := makePayload(1280 + 1) // larger than the datagram size could be
datagram := v3.ICMPDatagram{Payload: payload}
_, err := datagram.MarshalBinary()
if !errors.Is(err, v3.ErrDatagramICMPPayloadTooLarge) {
t.Errorf("expected an error: %s", err)
}
payload = makePayload(1280 + 2) // larger than the datagram size could be + header
payload[0] = byte(v3.ICMPType)
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(payload)
if !errors.Is(err, v3.ErrDatagramICMPPayloadTooLarge) {
t.Errorf("expected an error: %s", err)
}
})
}
func compareRegistrationDatagrams(t *testing.T, l *v3.UDPSessionRegistrationDatagram, r *v3.UDPSessionRegistrationDatagram) bool {
require.Equal(t, l.Payload, r.Payload)
return l.RequestID == r.RequestID &&
@@ -377,3 +437,13 @@ func FuzzRegistrationResponseDatagram(f *testing.F) {
}
})
}
func FuzzICMPDatagram(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
unmarshaled := v3.ICMPDatagram{}
err := unmarshaled.UnmarshalBinary(data)
if err == nil {
_, _ = unmarshaled.MarshalBinary()
}
})
}
+52
View File
@@ -0,0 +1,52 @@
package v3
import (
"context"
"github.com/rs/zerolog"
"go.opentelemetry.io/otel/trace"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
"github.com/cloudflare/cloudflared/tracing"
)
// packetResponder is an implementation of the [ingress.ICMPResponder] which provides the ICMP Flow manager the
// return path to return and ICMP Echo response back to the QUIC muxer.
type packetResponder struct {
datagramMuxer DatagramICMPWriter
connID uint8
}
func newPacketResponder(datagramMuxer DatagramICMPWriter, connID uint8) ingress.ICMPResponder {
return &packetResponder{
datagramMuxer,
connID,
}
}
func (pr *packetResponder) ConnectionIndex() uint8 {
return pr.connID
}
func (pr *packetResponder) ReturnPacket(pk *packet.ICMP) error {
return pr.datagramMuxer.SendICMPPacket(pk)
}
func (pr *packetResponder) AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte) {
// datagram v3 does not support tracing ICMP packets
}
func (pr *packetResponder) RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
// datagram v3 does not support tracing ICMP packets
return ctx, tracing.NewNoopSpan()
}
func (pr *packetResponder) ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
// datagram v3 does not support tracing ICMP packets
return ctx, tracing.NewNoopSpan()
}
func (pr *packetResponder) ExportSpan() {
// datagram v3 does not support tracing ICMP packets
}
+45
View File
@@ -0,0 +1,45 @@
package v3_test
import (
"context"
"testing"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
)
type noopICMPRouter struct{}
func (noopICMPRouter) Request(ctx context.Context, pk *packet.ICMP, responder ingress.ICMPResponder) error {
return nil
}
func (noopICMPRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
return nil
}
type mockICMPRouter struct {
recv chan *packet.ICMP
}
func newMockICMPRouter() *mockICMPRouter {
return &mockICMPRouter{
recv: make(chan *packet.ICMP, 1),
}
}
func (m *mockICMPRouter) Request(ctx context.Context, pk *packet.ICMP, responder ingress.ICMPResponder) error {
m.recv <- pk
return nil
}
func (mockICMPRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
return packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, testLocalAddr.AddrPort().Addr())
}
func assertICMPEqual(t *testing.T, expected *packet.ICMP, actual *packet.ICMP) {
if expected.Src != actual.Src {
t.Fatalf("Src address not equal: %+v\t%+v", expected, actual)
}
if expected.Dst != actual.Dst {
t.Fatalf("Dst address not equal: %+v\t%+v", expected, actual)
}
}
+117 -9
View File
@@ -3,9 +3,14 @@ package v3
import (
"context"
"errors"
"fmt"
"sync"
"time"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
)
const (
@@ -15,24 +20,31 @@ const (
logSrcKey = "src"
logDstKey = "dst"
logICMPTypeKey = "type"
logDurationKey = "durationMS"
)
// DatagramConn is the bridge that multiplexes writes and reads of datagrams for UDP sessions and ICMP packets to
// a connection.
type DatagramConn interface {
DatagramWriter
DatagramUDPWriter
DatagramICMPWriter
// Serve provides a server interface to process and handle incoming QUIC datagrams and demux their datagram v3 payloads.
Serve(context.Context) error
// ID indicates connection index identifier
ID() uint8
}
// DatagramWriter provides the Muxer interface to create proper Datagrams when sending over a connection.
type DatagramWriter interface {
// DatagramUDPWriter provides the Muxer interface to create proper UDP Datagrams when sending over a connection.
type DatagramUDPWriter interface {
SendUDPSessionDatagram(datagram []byte) error
SendUDPSessionResponse(id RequestID, resp SessionRegistrationResp) error
//SendICMPPacket(packet packet.IP) error
}
// DatagramICMPWriter provides the Muxer interface to create ICMP Datagrams when sending over a connection.
type DatagramICMPWriter interface {
SendICMPPacket(icmp *packet.ICMP) error
SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error
}
// QuicConnection provides an interface that matches [quic.Connection] for only the datagram operations.
@@ -50,27 +62,41 @@ type datagramConn struct {
conn QuicConnection
index uint8
sessionManager SessionManager
icmpRouter ingress.ICMPRouter
metrics Metrics
logger *zerolog.Logger
datagrams chan []byte
readErrors chan error
datagrams chan []byte
readErrors chan error
icmpEncoderPool sync.Pool // a pool of *packet.Encoder
icmpDecoderPool sync.Pool
}
func NewDatagramConn(conn QuicConnection, sessionManager SessionManager, index uint8, metrics Metrics, logger *zerolog.Logger) DatagramConn {
func NewDatagramConn(conn QuicConnection, sessionManager SessionManager, icmpRouter ingress.ICMPRouter, index uint8, metrics Metrics, logger *zerolog.Logger) DatagramConn {
log := logger.With().Uint8("datagramVersion", 3).Logger()
return &datagramConn{
conn: conn,
index: index,
sessionManager: sessionManager,
icmpRouter: icmpRouter,
metrics: metrics,
logger: &log,
datagrams: make(chan []byte, demuxChanCapacity),
readErrors: make(chan error, 2),
icmpEncoderPool: sync.Pool{
New: func() any {
return packet.NewEncoder()
},
},
icmpDecoderPool: sync.Pool{
New: func() any {
return packet.NewICMPDecoder()
},
},
}
}
func (c datagramConn) ID() uint8 {
func (c *datagramConn) ID() uint8 {
return c.index
}
@@ -90,6 +116,33 @@ func (c *datagramConn) SendUDPSessionResponse(id RequestID, resp SessionRegistra
return c.conn.SendDatagram(data)
}
func (c *datagramConn) SendICMPPacket(icmp *packet.ICMP) error {
cachedEncoder := c.icmpEncoderPool.Get()
// The encoded packet is a slice to a buffer owned by the encoder, so we shouldn't return the encoder back to the
// pool until the encoded packet is sent.
defer c.icmpEncoderPool.Put(cachedEncoder)
encoder, ok := cachedEncoder.(*packet.Encoder)
if !ok {
return fmt.Errorf("encoderPool returned %T, expect *packet.Encoder", cachedEncoder)
}
payload, err := encoder.Encode(icmp)
if err != nil {
return err
}
icmpDatagram := ICMPDatagram{
Payload: payload.Data,
}
datagram, err := icmpDatagram.MarshalBinary()
if err != nil {
return err
}
return c.conn.SendDatagram(datagram)
}
func (c *datagramConn) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error {
return c.SendICMPPacket(c.icmpRouter.ConvertToTTLExceeded(icmp, rawPacket))
}
var errReadTimeout error = errors.New("receive datagram timeout")
// pollDatagrams will read datagrams from the underlying connection until the provided context is done.
@@ -165,6 +218,14 @@ func (c *datagramConn) Serve(ctx context.Context) error {
}
logger := c.logger.With().Str(logFlowID, payload.RequestID.String()).Logger()
c.handleSessionPayloadDatagram(payload, &logger)
case ICMPType:
packet := &ICMPDatagram{}
err := packet.UnmarshalBinary(datagram)
if err != nil {
c.logger.Err(err).Msgf("unable to unmarshal icmp datagram")
return
}
c.handleICMPPacket(packet)
case UDPSessionRegistrationResponseType:
// cloudflared should never expect to receive UDP session responses as it will not initiate new
// sessions towards the edge.
@@ -267,7 +328,7 @@ func (c *datagramConn) handleSessionMigration(requestID RequestID, logger *zerol
// Migrate the session to use this edge connection instead of the currently running one.
// We also pass in this connection's logger to override the existing logger for the session.
session.Migrate(c, c.logger)
session.Migrate(c, c.conn.Context(), c.logger)
// Send another registration response since the session is already active
err = c.SendUDPSessionResponse(requestID, ResponseOk)
@@ -299,3 +360,50 @@ func (c *datagramConn) handleSessionPayloadDatagram(datagram *UDPSessionPayloadD
return
}
}
// Handles incoming ICMP datagrams.
func (c *datagramConn) handleICMPPacket(datagram *ICMPDatagram) {
if c.icmpRouter == nil {
// ICMPRouter is disabled so we drop the current packet and ignore all incoming ICMP packets
return
}
// Decode the provided ICMPDatagram as an ICMP packet
rawPacket := packet.RawPacket{Data: datagram.Payload}
cachedDecoder := c.icmpDecoderPool.Get()
defer c.icmpDecoderPool.Put(cachedDecoder)
decoder, ok := cachedDecoder.(*packet.ICMPDecoder)
if !ok {
c.logger.Error().Msg("Could not get ICMPDecoder from the pool. Dropping packet")
return
}
icmp, err := decoder.Decode(rawPacket)
if err != nil {
c.logger.Err(err).Msgf("unable to marshal icmp packet")
return
}
// If the ICMP packet's TTL is expired, we won't send it to the origin and immediately return a TTL Exceeded Message
if icmp.TTL <= 1 {
if err := c.SendICMPTTLExceed(icmp, rawPacket); err != nil {
c.logger.Err(err).Msg("failed to return ICMP TTL exceed error")
}
return
}
icmp.TTL--
// The context isn't really needed here since it's only really used throughout the ICMP router as a way to store
// the tracing context, however datagram V3 does not support tracing ICMP packets, so we just pass the current
// connection context which will have no tracing information available.
err = c.icmpRouter.Request(c.conn.Context(), icmp, newPacketResponder(c, c.index))
if err != nil {
c.logger.Err(err).
Str(logSrcKey, icmp.Src.String()).
Str(logDstKey, icmp.Dst.String()).
Interface(logICMPTypeKey, icmp.Type).
Msgf("unable to write icmp datagram to origin")
return
}
}
+248 -20
View File
@@ -4,16 +4,24 @@ import (
"bytes"
"context"
"errors"
"fmt"
"net"
"net/netip"
"slices"
"sort"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/google/gopacket/layers"
"github.com/rs/zerolog"
"golang.org/x/net/icmp"
"golang.org/x/net/ipv4"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
v3 "github.com/cloudflare/cloudflared/quic/v3"
)
@@ -27,6 +35,8 @@ func (noopEyeball) SendUDPSessionDatagram(datagram []byte) error { return nil }
func (noopEyeball) SendUDPSessionResponse(id v3.RequestID, resp v3.SessionRegistrationResp) error {
return nil
}
func (noopEyeball) SendICMPPacket(icmp *packet.ICMP) error { return nil }
func (noopEyeball) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error { return nil }
type mockEyeball struct {
connID uint8
@@ -70,9 +80,14 @@ func (m *mockEyeball) SendUDPSessionResponse(id v3.RequestID, resp v3.SessionReg
return nil
}
func (m *mockEyeball) SendICMPPacket(icmp *packet.ICMP) error { return nil }
func (m *mockEyeball) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error {
return nil
}
func TestDatagramConn_New(t *testing.T) {
log := zerolog.Nop()
conn := v3.NewDatagramConn(newMockQuicConn(), v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(newMockQuicConn(), v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
if conn == nil {
t.Fatal("expected valid connection")
}
@@ -81,7 +96,7 @@ func TestDatagramConn_New(t *testing.T) {
func TestDatagramConn_SendUDPSessionDatagram(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
payload := []byte{0xef, 0xef}
conn.SendUDPSessionDatagram(payload)
@@ -94,7 +109,7 @@ func TestDatagramConn_SendUDPSessionDatagram(t *testing.T) {
func TestDatagramConn_SendUDPSessionResponse(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
conn.SendUDPSessionResponse(testRequestID, v3.ResponseDestinationUnreachable)
resp := <-quic.recv
@@ -115,7 +130,7 @@ func TestDatagramConn_SendUDPSessionResponse(t *testing.T) {
func TestDatagramConnServe_ApplicationClosed(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
@@ -131,7 +146,7 @@ func TestDatagramConnServe_ConnectionClosed(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
quic.ctx = ctx
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
err := conn.Serve(context.Background())
if !errors.Is(err, context.DeadlineExceeded) {
@@ -142,7 +157,7 @@ func TestDatagramConnServe_ConnectionClosed(t *testing.T) {
func TestDatagramConnServe_ReceiveDatagramError(t *testing.T) {
log := zerolog.Nop()
quic := &mockQuicConnReadError{err: net.ErrClosed}
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
err := conn.Serve(context.Background())
if !errors.Is(err, net.ErrClosed) {
@@ -177,7 +192,7 @@ func TestDatagramConnServe_ErrorDatagramTypes(t *testing.T) {
log := zerolog.New(logOutput)
quic := newMockQuicConn()
quic.send <- test.input
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
@@ -218,7 +233,7 @@ func TestDatagramConnServe_RegisterSession_SessionManagerError(t *testing.T) {
quic := newMockQuicConn()
expectedErr := errors.New("unable to register session")
sessionManager := mockSessionManager{expectedRegErr: expectedErr}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -253,7 +268,7 @@ func TestDatagramConnServe(t *testing.T) {
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -293,12 +308,97 @@ func TestDatagramConnServe(t *testing.T) {
assertContextClosed(t, ctx, done, cancel)
}
// This test exists because decoding multiple packets in parallel with the same decoder
// instances causes inteference resulting in multiple different raw packets being decoded
// as the same decoded packet.
func TestDatagramConnServeDecodeMultipleICMPInParallel(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
router := newMockICMPRouter()
conn := v3.NewDatagramConn(quic, &sessionManager, router, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
packetCount := 100
packets := make([]*packet.ICMP, 100)
ipTemplate := "10.0.0.%d"
for i := 1; i <= packetCount; i++ {
packets[i-1] = &packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("192.168.1.1"),
Dst: netip.MustParseAddr(fmt.Sprintf(ipTemplate, i)),
Protocol: layers.IPProtocolICMPv4,
TTL: 20,
},
Message: &icmp.Message{
Type: ipv4.ICMPTypeEcho,
Code: 0,
Body: &icmp.Echo{
ID: 25821,
Seq: 58129,
Data: []byte("test"),
},
},
}
}
wg := sync.WaitGroup{}
var receivedPackets []*packet.ICMP
go func() {
for ctx.Err() == nil {
select {
case icmpPacket := <-router.recv:
receivedPackets = append(receivedPackets, icmpPacket)
wg.Done()
}
}
}()
for _, p := range packets {
// We increment here but only decrement when receiving the packet
wg.Add(1)
go func() {
datagram := newICMPDatagram(p)
quic.send <- datagram
}()
}
wg.Wait()
// If there were duplicates then we won't have the same number of IPs
packetSet := make(map[netip.Addr]*packet.ICMP, 0)
for _, p := range receivedPackets {
packetSet[p.Dst] = p
}
assert.Equal(t, len(packetSet), len(packets))
// Sort the slice by last byte of IP address (the one we increment for each destination)
// and then check that we have one match for each packet sent
sort.Slice(receivedPackets, func(i, j int) bool {
return receivedPackets[i].Dst.As4()[3] < receivedPackets[j].Dst.As4()[3]
})
for i, p := range receivedPackets {
assert.Equal(t, p.Dst, packets[i].Dst)
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_RegisterTwice(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -360,9 +460,9 @@ func TestDatagramConnServe_MigrateConnection(t *testing.T) {
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
quic2 := newMockQuicConn()
conn2 := v3.NewDatagramConn(quic2, &sessionManager, 1, &noopMetrics{}, &log)
conn2 := v3.NewDatagramConn(quic2, &sessionManager, &noopICMPRouter{}, 1, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -443,7 +543,7 @@ func TestDatagramConnServe_Payload_GetSessionError(t *testing.T) {
quic := newMockQuicConn()
// mockSessionManager will return the ErrSessionNotFound for any session attempting to be queried by the muxer
sessionManager := mockSessionManager{session: nil, expectedGetErr: v3.ErrSessionNotFound}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -471,7 +571,7 @@ func TestDatagramConnServe_Payload(t *testing.T) {
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, 0, &noopMetrics{}, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -496,6 +596,116 @@ func TestDatagramConnServe_Payload(t *testing.T) {
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_ICMPDatagram_TTLDecremented(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
router := newMockICMPRouter()
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, router, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
// Send new ICMP Echo request
expectedICMP := &packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("192.168.1.1"),
Dst: netip.MustParseAddr("10.0.0.1"),
Protocol: layers.IPProtocolICMPv4,
TTL: 20,
},
Message: &icmp.Message{
Type: ipv4.ICMPTypeEcho,
Code: 0,
Body: &icmp.Echo{
ID: 25821,
Seq: 58129,
Data: []byte("test ttl=0"),
},
},
}
datagram := newICMPDatagram(expectedICMP)
quic.send <- datagram
// Router should receive the packet
actualICMP := <-router.recv
assertICMPEqual(t, expectedICMP, actualICMP)
if expectedICMP.TTL-1 != actualICMP.TTL {
t.Fatalf("TTL should be decremented by one before sending to origin: %d, %d", expectedICMP.TTL, actualICMP.TTL)
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_ICMPDatagram_TTLExceeded(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
router := newMockICMPRouter()
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, router, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
// Send new ICMP Echo request
expectedICMP := &packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("192.168.1.1"),
Dst: netip.MustParseAddr("10.0.0.1"),
Protocol: layers.IPProtocolICMPv4,
TTL: 0,
},
Message: &icmp.Message{
Type: ipv4.ICMPTypeEcho,
Code: 0,
Body: &icmp.Echo{
ID: 25821,
Seq: 58129,
Data: []byte("test ttl=0"),
},
},
}
datagram := newICMPDatagram(expectedICMP)
quic.send <- datagram
// Origin should not recieve a packet
select {
case <-router.recv:
t.Fatalf("TTL should be expired and no origin ICMP sent")
default:
}
// Eyeball should receive the packet
datagram = <-quic.recv
icmpDatagram := v3.ICMPDatagram{}
err := icmpDatagram.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
decoder := packet.NewICMPDecoder()
ttlExpiredICMP, err := decoder.Decode(packet.RawPacket{Data: icmpDatagram.Payload})
if err != nil {
t.Fatal(err)
}
// Packet should be a TTL Exceeded ICMP
if ttlExpiredICMP.TTL != packet.DefaultTTL || ttlExpiredICMP.Message.Type != ipv4.ICMPTypeTimeExceeded {
t.Fatalf("ICMP packet should be a ICMP Exceeded: %+v", ttlExpiredICMP)
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func newRegisterSessionDatagram(id v3.RequestID) []byte {
datagram := v3.UDPSessionRegistrationDatagram{
RequestID: id,
@@ -531,6 +741,22 @@ func newSessionPayloadDatagram(id v3.RequestID, payload []byte) []byte {
return datagram
}
func newICMPDatagram(pk *packet.ICMP) []byte {
encoder := packet.NewEncoder()
rawPacket, err := encoder.Encode(pk)
if err != nil {
panic(err)
}
datagram := v3.ICMPDatagram{
Payload: rawPacket.Data,
}
payload, err := datagram.MarshalBinary()
if err != nil {
panic(err)
}
return payload
}
// Cancel the provided context and make sure it closes with the expected cancellation error
func assertContextClosed(t *testing.T, ctx context.Context, done <-chan error, cancel context.CancelCauseFunc) {
cancel(expectedContextCanceled)
@@ -619,12 +845,14 @@ func newMockSession() mockSession {
}
}
func (m *mockSession) ID() v3.RequestID { return testRequestID }
func (m *mockSession) RemoteAddr() net.Addr { return testOriginAddr }
func (m *mockSession) LocalAddr() net.Addr { return testLocalAddr }
func (m *mockSession) ConnectionID() uint8 { return 0 }
func (m *mockSession) Migrate(conn v3.DatagramConn, log *zerolog.Logger) { m.migrated <- conn.ID() }
func (m *mockSession) ResetIdleTimer() {}
func (m *mockSession) ID() v3.RequestID { return testRequestID }
func (m *mockSession) RemoteAddr() net.Addr { return testOriginAddr }
func (m *mockSession) LocalAddr() net.Addr { return testLocalAddr }
func (m *mockSession) ConnectionID() uint8 { return 0 }
func (m *mockSession) Migrate(conn v3.DatagramConn, ctx context.Context, log *zerolog.Logger) {
m.migrated <- conn.ID()
}
func (m *mockSession) ResetIdleTimer() {}
func (m *mockSession) Serve(ctx context.Context) error {
close(m.served)
+34 -15
View File
@@ -55,7 +55,7 @@ type Session interface {
RemoteAddr() net.Addr
LocalAddr() net.Addr
ResetIdleTimer()
Migrate(eyeball DatagramConn, logger *zerolog.Logger)
Migrate(eyeball DatagramConn, ctx context.Context, logger *zerolog.Logger)
// Serve starts the event loop for processing UDP packets
Serve(ctx context.Context) error
}
@@ -70,8 +70,12 @@ type session struct {
// activeAtChan is used to communicate the last read/write time
activeAtChan chan time.Time
closeChan chan error
contextChan chan context.Context
metrics Metrics
log *zerolog.Logger
// A special close function that we wrap with sync.Once to make sure it is only called once
closeFn func() error
}
func NewSession(
@@ -85,6 +89,10 @@ func NewSession(
log *zerolog.Logger,
) Session {
logger := log.With().Str(logFlowID, id.String()).Logger()
// closeChan has two slots to allow for both writers (the closeFn and the Serve routine) to both be able to
// write to the channel without blocking since there is only ever one value read from the closeChan by the
// waitForCloseCondition.
closeChan := make(chan error, 2)
session := &session{
id: id,
closeAfterIdle: closeAfterIdle,
@@ -95,9 +103,19 @@ func NewSession(
// activeAtChan has low capacity. It can be full when there are many concurrent read/write. markActive() will
// drop instead of blocking because last active time only needs to be an approximation
activeAtChan: make(chan time.Time, 1),
closeChan: make(chan error, 1),
metrics: metrics,
log: &logger,
closeChan: closeChan,
// contextChan is an unbounded channel to help enforce one active migration of a session at a time.
contextChan: make(chan context.Context),
metrics: metrics,
log: &logger,
closeFn: sync.OnceValue(func() error {
// We don't want to block on sending to the close channel if it is already full
select {
case closeChan <- SessionCloseErr:
default:
}
return origin.Close()
}),
}
session.eyeball.Store(&eyeball)
return session
@@ -120,11 +138,12 @@ func (s *session) ConnectionID() uint8 {
return eyeball.ID()
}
func (s *session) Migrate(eyeball DatagramConn, logger *zerolog.Logger) {
func (s *session) Migrate(eyeball DatagramConn, ctx context.Context, logger *zerolog.Logger) {
current := *(s.eyeball.Load())
// Only migrate if the connection ids are different.
if current.ID() != eyeball.ID() {
s.eyeball.Store(&eyeball)
s.contextChan <- ctx
log := logger.With().Str(logFlowID, s.id.String()).Logger()
s.log = &log
}
@@ -214,17 +233,11 @@ func (s *session) markActive() {
func (s *session) Close() error {
// Make sure that we only close the origin connection once
return sync.OnceValue(func() error {
// We don't want to block on sending to the close channel if it is already full
select {
case s.closeChan <- SessionCloseErr:
default:
}
return s.origin.Close()
})()
return s.closeFn()
}
func (s *session) waitForCloseCondition(ctx context.Context, closeAfterIdle time.Duration) error {
connCtx := ctx
// Closing the session at the end cancels read so Serve() can return
defer s.Close()
if closeAfterIdle == 0 {
@@ -237,8 +250,14 @@ func (s *session) waitForCloseCondition(ctx context.Context, closeAfterIdle time
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-connCtx.Done():
return connCtx.Err()
case newContext := <-s.contextChan:
// During migration of a session, we need to make sure that the context of the new connection is used instead
// of the old connection context. This will ensure that when the old connection goes away, this session will
// still be active on the existing connection.
connCtx = newContext
continue
case reason := <-s.closeChan:
return reason
case <-checkIdleTimer.C:
+155 -89
View File
@@ -3,13 +3,14 @@ package v3_test
import (
"context"
"errors"
"io"
"net"
"net/netip"
"slices"
"sync/atomic"
"testing"
"time"
"github.com/fortytw2/leaktest"
"github.com/rs/zerolog"
v3 "github.com/cloudflare/cloudflared/quic/v3"
@@ -32,45 +33,64 @@ func TestSessionNew(t *testing.T) {
func testSessionWrite(t *testing.T, payload []byte) {
log := zerolog.Nop()
origin := newTestOrigin(makePayload(1280))
session := v3.NewSession(testRequestID, 5*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
// Start origin server read
serverRead := make(chan []byte, 1)
go func() {
read := make([]byte, 1500)
server.Read(read[:])
serverRead <- read
}()
// Create session and write to origin
session := v3.NewSession(testRequestID, 5*time.Second, origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
n, err := session.Write(payload)
defer session.Close()
if err != nil {
t.Fatal(err)
}
if n != len(payload) {
t.Fatal("unable to write the whole payload")
}
if !slices.Equal(payload, origin.write[:len(payload)]) {
read := <-serverRead
if !slices.Equal(payload, read[:len(payload)]) {
t.Fatal("payload provided from origin and read value are not the same")
}
}
func TestSessionWrite_Max(t *testing.T) {
defer leaktest.Check(t)()
payload := makePayload(1280)
testSessionWrite(t, payload)
}
func TestSessionWrite_Min(t *testing.T) {
defer leaktest.Check(t)()
payload := makePayload(0)
testSessionWrite(t, payload)
}
func TestSessionServe_OriginMax(t *testing.T) {
defer leaktest.Check(t)()
payload := makePayload(1280)
testSessionServe_Origin(t, payload)
}
func TestSessionServe_OriginMin(t *testing.T) {
defer leaktest.Check(t)()
payload := makePayload(0)
testSessionServe_Origin(t, payload)
}
func testSessionServe_Origin(t *testing.T, payload []byte) {
log := zerolog.Nop()
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
eyeball := newMockEyeball()
origin := newTestOrigin(payload)
session := v3.NewSession(testRequestID, 3*time.Second, &origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
session := v3.NewSession(testRequestID, 3*time.Second, origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
ctx, cancel := context.WithCancelCause(context.Background())
@@ -80,13 +100,19 @@ func testSessionServe_Origin(t *testing.T, payload []byte) {
done <- session.Serve(ctx)
}()
// Write from the origin server
_, err := server.Write(payload)
if err != nil {
t.Fatal(err)
}
select {
case data := <-eyeball.recvData:
// check received data matches provided from origin
expectedData := makePayload(1500)
v3.MarshalPayloadHeaderTo(testRequestID, expectedData[:])
copy(expectedData[17:], payload)
if !slices.Equal(expectedData[:17+len(payload)], data) {
if !slices.Equal(expectedData[:v3.DatagramPayloadHeaderLen+len(payload)], data) {
t.Fatal("expected datagram did not equal expected")
}
cancel(expectedContextCanceled)
@@ -95,7 +121,7 @@ func testSessionServe_Origin(t *testing.T, payload []byte) {
t.Fatal(err)
}
err := <-done
err = <-done
if !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
@@ -105,11 +131,14 @@ func testSessionServe_Origin(t *testing.T, payload []byte) {
}
func TestSessionServe_OriginTooLarge(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
eyeball := newMockEyeball()
payload := makePayload(1281)
origin := newTestOrigin(payload)
session := v3.NewSession(testRequestID, 2*time.Second, &origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
session := v3.NewSession(testRequestID, 2*time.Second, origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
done := make(chan error)
@@ -117,6 +146,12 @@ func TestSessionServe_OriginTooLarge(t *testing.T) {
done <- session.Serve(context.Background())
}()
// Attempt to write a payload too large from the origin
_, err := server.Write(payload)
if err != nil {
t.Fatal(err)
}
select {
case data := <-eyeball.recvData:
// we never expect a read to make it here because the origin provided a payload that is too large
@@ -130,6 +165,7 @@ func TestSessionServe_OriginTooLarge(t *testing.T) {
}
func TestSessionServe_Migrate(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
eyeball := newMockEyeball()
pipe1, pipe2 := net.Pipe()
@@ -137,14 +173,28 @@ func TestSessionServe_Migrate(t *testing.T) {
defer session.Close()
done := make(chan error)
eyeball1Ctx, cancel := context.WithCancelCause(context.Background())
go func() {
done <- session.Serve(context.Background())
done <- session.Serve(eyeball1Ctx)
}()
// Migrate the session to a new connection before origin sends data
eyeball2 := newMockEyeball()
eyeball2.connID = 1
session.Migrate(&eyeball2, &log)
eyeball2Ctx := context.Background()
session.Migrate(&eyeball2, eyeball2Ctx, &log)
// Cancel the origin eyeball context; this should not cancel the session
contextCancelErr := errors.New("context canceled for first eyeball connection")
cancel(contextCancelErr)
select {
case <-done:
t.Fatalf("expected session to still be running")
default:
}
if context.Cause(eyeball1Ctx) != contextCancelErr {
t.Fatalf("first eyeball context should be cancelled manually: %+v", context.Cause(eyeball1Ctx))
}
// Origin sends data
payload2 := []byte{0xde}
@@ -166,37 +216,114 @@ func TestSessionServe_Migrate(t *testing.T) {
if !errors.Is(err, v3.SessionIdleErr{}) {
t.Error(err)
}
if eyeball2Ctx.Err() != nil {
t.Fatalf("second eyeball context should be not be cancelled")
}
}
func TestSessionServe_Migrate_CloseContext2(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
eyeball := newMockEyeball()
pipe1, pipe2 := net.Pipe()
session := v3.NewSession(testRequestID, 2*time.Second, pipe2, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
done := make(chan error)
eyeball1Ctx, cancel := context.WithCancelCause(context.Background())
go func() {
done <- session.Serve(eyeball1Ctx)
}()
// Migrate the session to a new connection before origin sends data
eyeball2 := newMockEyeball()
eyeball2.connID = 1
eyeball2Ctx, cancel2 := context.WithCancelCause(context.Background())
session.Migrate(&eyeball2, eyeball2Ctx, &log)
// Cancel the origin eyeball context; this should not cancel the session
contextCancelErr := errors.New("context canceled for first eyeball connection")
cancel(contextCancelErr)
select {
case <-done:
t.Fatalf("expected session to still be running")
default:
}
if context.Cause(eyeball1Ctx) != contextCancelErr {
t.Fatalf("first eyeball context should be cancelled manually: %+v", context.Cause(eyeball1Ctx))
}
// Origin sends data
payload2 := []byte{0xde}
pipe1.Write(payload2)
// Expect write to eyeball2
data := <-eyeball2.recvData
if len(data) <= 17 || !slices.Equal(payload2, data[17:]) {
t.Fatalf("expected data to write to eyeball2 after migration: %+v", data)
}
select {
case data := <-eyeball.recvData:
t.Fatalf("expected no data to write to eyeball1 after migration: %+v", data)
default:
}
// Close the connection2 context manually
contextCancel2Err := errors.New("context canceled for second eyeball connection")
cancel2(contextCancel2Err)
err := <-done
if err != context.Canceled {
t.Fatalf("session Serve should be done: %+v", err)
}
if context.Cause(eyeball2Ctx) != contextCancel2Err {
t.Fatalf("second eyeball context should have been cancelled manually: %+v", context.Cause(eyeball2Ctx))
}
}
func TestSessionClose_Multiple(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
origin := newTestOrigin(makePayload(128))
session := v3.NewSession(testRequestID, 5*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
session := v3.NewSession(testRequestID, 5*time.Second, origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
err := session.Close()
if err != nil {
t.Fatal(err)
}
if !origin.closed.Load() {
t.Fatal("origin wasn't closed")
b := [1500]byte{}
_, err = server.Read(b[:])
if !errors.Is(err, io.EOF) {
t.Fatalf("origin server connection should be closed: %s", err)
}
// subsequent closes shouldn't call close again or cause any errors
err = session.Close()
if err != nil {
t.Fatal(err)
}
_, err = server.Read(b[:])
if !errors.Is(err, io.EOF) {
t.Fatalf("origin server connection should still be closed: %s", err)
}
}
func TestSessionServe_IdleTimeout(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
origin := newTestIdleOrigin(10 * time.Second) // Make idle time longer than closeAfterIdle
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
closeAfterIdle := 2 * time.Second
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
session := v3.NewSession(testRequestID, closeAfterIdle, origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
err := session.Serve(context.Background())
if !errors.Is(err, v3.SessionIdleErr{}) {
t.Fatal(err)
}
// session should be closed
if !origin.closed {
b := [1500]byte{}
_, err = server.Read(b[:])
if !errors.Is(err, io.EOF) {
t.Fatalf("session should be closed after Serve returns")
}
// closing a session again should not return an error
@@ -207,12 +334,14 @@ func TestSessionServe_IdleTimeout(t *testing.T) {
}
func TestSessionServe_ParentContextCanceled(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
// Make idle time and idle timeout longer than closeAfterIdle
origin := newTestIdleOrigin(10 * time.Second)
origin, server := net.Pipe()
defer origin.Close()
defer server.Close()
closeAfterIdle := 10 * time.Second
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
session := v3.NewSession(testRequestID, closeAfterIdle, origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
err := session.Serve(ctx)
@@ -220,7 +349,9 @@ func TestSessionServe_ParentContextCanceled(t *testing.T) {
t.Fatal(err)
}
// session should be closed
if !origin.closed {
b := [1500]byte{}
_, err = server.Read(b[:])
if !errors.Is(err, io.EOF) {
t.Fatalf("session should be closed after Serve returns")
}
// closing a session again should not return an error
@@ -231,6 +362,7 @@ func TestSessionServe_ParentContextCanceled(t *testing.T) {
}
func TestSessionServe_ReadErrors(t *testing.T) {
defer leaktest.Check(t)()
log := zerolog.Nop()
origin := newTestErrOrigin(net.ErrClosed, nil)
session := v3.NewSession(testRequestID, 30*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
@@ -240,72 +372,6 @@ func TestSessionServe_ReadErrors(t *testing.T) {
}
}
type testOrigin struct {
// bytes from Write
write []byte
// bytes provided to Read
read []byte
readOnce atomic.Bool
closed atomic.Bool
}
func newTestOrigin(payload []byte) testOrigin {
return testOrigin{
read: payload,
}
}
func (o *testOrigin) Read(p []byte) (n int, err error) {
if o.closed.Load() {
return -1, net.ErrClosed
}
if o.readOnce.Load() {
// We only want to provide one read so all other reads will be blocked
time.Sleep(10 * time.Second)
}
o.readOnce.Store(true)
return copy(p, o.read), nil
}
func (o *testOrigin) Write(p []byte) (n int, err error) {
if o.closed.Load() {
return -1, net.ErrClosed
}
o.write = make([]byte, len(p))
copy(o.write, p)
return len(p), nil
}
func (o *testOrigin) Close() error {
o.closed.Store(true)
return nil
}
type testIdleOrigin struct {
duration time.Duration
closed bool
}
func newTestIdleOrigin(d time.Duration) testIdleOrigin {
return testIdleOrigin{
duration: d,
}
}
func (o *testIdleOrigin) Read(p []byte) (n int, err error) {
time.Sleep(o.duration)
return -1, nil
}
func (o *testIdleOrigin) Write(p []byte) (n int, err error) {
return 0, nil
}
func (o *testIdleOrigin) Close() error {
o.closed = true
return nil
}
type testErrOrigin struct {
readErr error
writeErr error
+1 -1
View File
@@ -113,7 +113,7 @@ class PkgCreator:
def create_rpm_pkgs(self, artifacts_path, gpg_key_name):
self._setup_rpm_pkg_directories(artifacts_path, gpg_key_name)
p = Popen(["createrepo", "./rpm"], stdout=PIPE, stderr=PIPE)
p = Popen(["createrepo_c", "./rpm"], stdout=PIPE, stderr=PIPE)
out, err = p.communicate()
if p.returncode != 0:
print(f"create rpm_pkgs result => {out}, {err}")
+2 -2
View File
@@ -119,9 +119,9 @@ func (s *Supervisor) Run(
ctx context.Context,
connectedSignal *signal.Signal,
) error {
if s.config.PacketConfig != nil {
if s.config.ICMPRouterServer != nil {
go func() {
if err := s.config.PacketConfig.ICMPRouter.Serve(ctx); err != nil {
if err := s.config.ICMPRouterServer.Serve(ctx); err != nil {
if errors.Is(err, net.ErrClosed) {
s.log.Logger().Info().Err(err).Msg("icmp router terminated")
} else {
+4 -2
View File
@@ -63,7 +63,7 @@ type TunnelConfig struct {
NamedTunnel *connection.TunnelProperties
ProtocolSelector connection.ProtocolSelector
EdgeTLSConfigs map[connection.Protocol]*tls.Config
PacketConfig *ingress.GlobalRouterConfig
ICMPRouterServer ingress.ICMPRouterServer
RPCTimeout time.Duration
WriteStreamTimeout time.Duration
@@ -607,6 +607,7 @@ func (e *EdgeTunnelServer) serveQUIC(
ctx,
conn,
e.sessionManager,
e.config.ICMPRouterServer,
connIndex,
e.datagramMetrics,
connLogger.Logger(),
@@ -615,7 +616,8 @@ func (e *EdgeTunnelServer) serveQUIC(
datagramSessionManager = connection.NewDatagramV2Connection(
ctx,
conn,
e.config.PacketConfig,
e.config.ICMPRouterServer,
connIndex,
e.config.RPCTimeout,
e.config.WriteStreamTimeout,
connLogger.Logger(),
+41 -18
View File
@@ -1,6 +1,7 @@
package tunnelstate
import (
"net"
"sync"
"github.com/rs/zerolog"
@@ -9,54 +10,58 @@ import (
)
type ConnTracker struct {
sync.RWMutex
mutex sync.RWMutex
// int is the connection Index
connectionInfo map[uint8]ConnectionInfo
log *zerolog.Logger
}
type ConnectionInfo struct {
IsConnected bool
Protocol connection.Protocol
IsConnected bool `json:"isConnected,omitempty"`
Protocol connection.Protocol `json:"protocol,omitempty"`
EdgeAddress net.IP `json:"edgeAddress,omitempty"`
}
func NewConnTracker(log *zerolog.Logger) *ConnTracker {
// Convinience struct to extend the connection with its index.
type IndexedConnectionInfo struct {
ConnectionInfo
Index uint8 `json:"index,omitempty"`
}
func NewConnTracker(
log *zerolog.Logger,
) *ConnTracker {
return &ConnTracker{
connectionInfo: make(map[uint8]ConnectionInfo, 0),
log: log,
}
}
func MockedConnTracker(mocked map[uint8]ConnectionInfo) *ConnTracker {
return &ConnTracker{
connectionInfo: mocked,
}
}
func (ct *ConnTracker) OnTunnelEvent(c connection.Event) {
switch c.EventType {
case connection.Connected:
ct.Lock()
ct.mutex.Lock()
ci := ConnectionInfo{
IsConnected: true,
Protocol: c.Protocol,
EdgeAddress: c.EdgeAddress,
}
ct.connectionInfo[c.Index] = ci
ct.Unlock()
ct.mutex.Unlock()
case connection.Disconnected, connection.Reconnecting, connection.RegisteringTunnel, connection.Unregistering:
ct.Lock()
ct.mutex.Lock()
ci := ct.connectionInfo[c.Index]
ci.IsConnected = false
ct.connectionInfo[c.Index] = ci
ct.Unlock()
ct.mutex.Unlock()
default:
ct.log.Error().Msgf("Unknown connection event case %v", c)
}
}
func (ct *ConnTracker) CountActiveConns() uint {
ct.RLock()
defer ct.RUnlock()
ct.mutex.RLock()
defer ct.mutex.RUnlock()
active := uint(0)
for _, ci := range ct.connectionInfo {
if ci.IsConnected {
@@ -69,8 +74,8 @@ func (ct *ConnTracker) CountActiveConns() uint {
// HasConnectedWith checks if we've ever had a successful connection to the edge
// with said protocol.
func (ct *ConnTracker) HasConnectedWith(protocol connection.Protocol) bool {
ct.RLock()
defer ct.RUnlock()
ct.mutex.RLock()
defer ct.mutex.RUnlock()
for _, ci := range ct.connectionInfo {
if ci.Protocol == protocol {
return true
@@ -78,3 +83,21 @@ func (ct *ConnTracker) HasConnectedWith(protocol connection.Protocol) bool {
}
return false
}
// Returns the connection information iff it is connected this
// also leverages the [IndexedConnectionInfo] to also provide the connection index
func (ct *ConnTracker) GetActiveConnections() []IndexedConnectionInfo {
ct.mutex.RLock()
defer ct.mutex.RUnlock()
connections := make([]IndexedConnectionInfo, 0)
for key, value := range ct.connectionInfo {
if value.IsConnected {
info := IndexedConnectionInfo{value, key}
connections = append(connections, info)
}
}
return connections
}