Compare commits

..

59 Commits

Author SHA1 Message Date
lneto e7dcb6edca Release 2024.12.1 2024-12-10 16:07:33 +00:00
Luis Neto 14cf0eff1d TUN-8795: update createrepo to createrepo_c to fix the release_pkgs.py script
## Summary
The default-flavour of cfsetup changed from bullseye to bookworm and in the latter the createrepo package was renamed to createrepo_c.

Closes TUN-8795
2024-12-10 08:07:02 -08:00
lneto a00c80f9e1 Release 2024.12.0 2024-12-09 16:55:31 +00:00
Luis Neto 12d878531c TUN-8789: make python package installation consistent
## Summary

The previous changes regarding python's distribution which broke CI the installation of python packages.

Python packages in cfsetup are now installed via virtual environment. The dependency python3-venv was added as builddep to allow the creation of venv and the python packages installation was moved to the post-cache resulting in the removal of 

* anchor build_release_pre_cache
* anchor component_test_pre_cache

Closes TUN-8789
2024-12-09 08:52:48 -08:00
Devin Carr 588ab7ebaa TUN-8640: Add ICMP support for datagram V3
Closes TUN-8640
2024-12-09 07:23:11 -08:00
Joe Groocock dfbccd917c TUN-8781: Add Trixie, drop Buster. Default to Bookworm
Closes TUN-8781
2024-12-06 05:04:16 -08:00
Devin Carr 37010529bc TUN-8775: Make sure the session Close can only be called once
The previous capture of the sync.OnceValue was re-initialized for each
call to `Close`. This needed to be initialized during the creation of
the session to ensure that the sync.OnceValue reference was held for
the session's lifetime.

Closes TUN-8775
2024-12-05 14:12:53 -08:00
Luis Neto f07d04d129 TUN-8768: add job report to diagnostic zipfile
## Summary

Add a new job that write to a file the result of all of the other tasks along with possible errors. This file is also added to the root of the diagnostic zip file.

 Closes TUN-8768
2024-12-04 10:32:49 -08:00
Luis Neto f12036c2da TUN-8770: add cli configuration and tunnel configuration to diagnostic zipfile
## Summary

Adds two new jobs which will export the cli configuration and tunnel configuration in separate files. These files will also be added to the zipfile's root.

 Closes TUN-8770
2024-12-04 05:26:51 -08:00
Luis Neto 520e266411 TUN-8767: include raw output from network collector in diagnostic zipfile
## Summary

Export raw format of traceroute is widely known and useful for debugging. This raw output is written to the zipfile's root at the end of the diagnostic.

Closes TUN-8767
2024-12-04 04:40:51 -08:00
Luis Neto 7bd86762a7 TUN-8725: implement diagnostic procedure
## Summary
Add a function that orchestrates the diagnostic jobs producing a zip file at the end.

Closes TUN-8725
2024-12-04 03:37:57 -08:00
lneto 451f98e1d1 TUN-8727: extend client to include function to get cli configuration and tunnel configuration 2024-12-03 21:39:03 +00:00
Luis Neto 60fe4a0800 TUN-8769: fix k8s log collector arguments
## Summary

The equal signs were making the exec command to fail removing them fixes the issue.

Closes TUN-8769
2024-12-03 06:24:14 -08:00
Luis Neto 1ef109c042 TUN-8762: fix argument order when invoking tracert and modify network info output parsing.
## Summary

The windows code path has three bugs:

* the -4 and -6 option cannot be passed in the last position
* since there are some lines in the command output that are not parsable the collection fails to parse any kind of output
* the timeout hop is not correctly parsed

This PR also guards the parsing code against empty domains

Closes TUN-8762
2024-12-03 04:56:28 -08:00
Luis Neto 65786597cc TUN-8732: implement port selection algorithm
## Summary
Implements the discovery of the metrics server given an array of addresses (known addresses), tunnelID, and connectorID.

Closes TUN-8732
2024-12-03 04:07:55 -08:00
Luis Neto f884b29d0d TUN-8726: implement compression routine to be used in diagnostic procedure
## Summary
Basic routine to compress the diagnostic files into the root of a zipfile.

Closes TUN-8726
2024-12-03 03:27:04 -08:00
Luis Neto b3304bf05b TUN-8727: implement metrics, runtime, system, and tunnelstate in diagnostic http client
## Summary
The diagnostic procedure needs to extract information available in the metrics server via HTTP calls. 
These changes add to the diagnostic client the remaining endpoints.

Closes TUN-8727
2024-11-29 09:08:42 -08:00
Luis Neto 28796c659e TUN-8729: implement network collection for diagnostic procedure
## Summary
This PR adds implementation for windows & unix that collect the tracert.exe & traceroute output in the form of hops.

Closes TUN-8729
2024-11-29 07:43:36 -08:00
Devin Carr 9da15b5d96 TUN-8640: Refactor ICMPRouter to support new ICMPResponders
A new ICMPResponder interface is introduced to provide different
implementations of how the ICMP flows should return to the QUIC
connection muxer.

Improves usages of netip.AddrPort to leverage the embedded zone
field for IPv6 addresses.

Closes TUN-8640
2024-11-27 12:46:08 -08:00
Luis Neto 46dc6316f9 TUN-8734: add log collection for kubernetes
## Summary
Adds the log collector for K8s based deployments.

Closes TUN-8734
2024-11-27 03:15:15 -08:00
Luis Neto 16e65c70ad TUN-8733: add log collection for docker
## Summary
Adds the log collector for docker based deployments

Closes TUN-8733
2024-11-27 01:08:54 -08:00
Luis Neto a6f9e68739 TUN-8735: add managed/local log collection
## Summary
Adds a log collector for the managed/local runtimes.


Closes TUN-8735 TUN-8736
2024-11-26 10:30:44 -08:00
Luis Neto f85c0f1cc0 TUN-8730: implement diag/configuration
Implements the endpoint that retrieves the configuration of a running instance.

The configuration consists in a map of cli flag to the provided value along with the uid that of the user that started the process
2024-11-25 11:24:51 -08:00
Luis Neto 4b0b6dc8c6 TUN-8728: implement diag/tunnel endpoint
## Summary
The new endpoint returns the current information to be used when calling the diagnostic procedure.
This also adds:
- add indexed connection info and method to extract active connections from connTracker
- add edge address to Event struct and conn tracker
- remove unnecessary event send
- add tunnel configuration handler
- adjust cmd and metrics to create diagnostic server

Closes TUN-8728
2024-11-25 10:43:32 -08:00
Luis Neto aab5364252 TUN-8731: Implement diag/system endpoint
## Summary
This PR will add a new endpoint, "diag/system" to the metrics server that collects system information from different operating systems.

Closes TUN-8731
2024-11-22 08:10:05 -08:00
Luis Neto e2c2b012f1 TUN-8737: update metrics server port selection
## Summary
Update how metrics server binds to a listener by using a known set of ports whenever the default address is used with the fallback to a random port in case all address are already in use. The default address changes at compile time in order to bind to a different default address when the final deliverable is a docker image.

Refactor ReadyServer tests.

Closes TUN-8737
2024-11-22 07:23:46 -08:00
Devin Carr d779394748 TUN-8748: Migrated datagram V3 flows to use migrated context
Previously, during local flow migration the current connection context
was not part of the migration and would cause the flow to still be listening
on the connection context of the old connection (before the migration).
This meant that if a flow was migrated from connection 0 to
connection 1, and connection 0 goes away, the flow would be early
terminated incorrectly with the context lifetime of connection 0.

The new connection context is provided during migration of a flow
and will trigger the observe loop for the flow lifetime to be rebound
to this provided context.
Closes TUN-8748
2024-11-21 12:56:47 -08:00
Chung-Ting Huang c59d56c655 Release 2024.11.1
## Summary
Some description...
2024-11-19 08:21:37 -08:00
chungthuang 3480a33fce Merge pull request #1135 from firecow/tunnel-health
Add cloudflared tunnel ready command
2024-11-18 10:08:45 -06:00
chungthuang a26b2a0097 Merge pull request #1355 from pkillarjun/fuzzing
add: new go-fuzz targets
2024-11-18 09:53:12 -06:00
chungthuang 37eee7e727 Merge pull request #1346 from emilioarvidisaksson/patch-1
Fixed 404 in README.md to TryCloudflare
2024-11-18 09:52:44 -06:00
chungthuang 9771f3309e Merge pull request #1329 from cloudflare/hrushikeshdeshpande-updating-semgrep-yml
Update semgrep.yml
2024-11-18 09:52:05 -06:00
Mads Jon Nielsen c39f0ae317 Make metrics a requirement for tunnel ready command 2024-11-14 18:38:17 +01:00
Mads Jon Nielsen 95dff74fc8 Merge branch 'cloudflare:master' into tunnel-health 2024-11-14 18:19:43 +01:00
Devin Carr ab3dc5f8fa TUN-8701: Simplify flow registration logs for datagram v3
To help reduce the volume of logs during the happy path of flow registration, there will only be one log message reported when a flow is completed.

There are additional fields added to all flow log messages:
1. `src`: local address
2. `dst`: origin address
3. `durationMS`: capturing the total duration of the flow in milliseconds

Additional logs were added to capture when a flow was migrated or when cloudflared sent off a registration response retry.

Closes TUN-8701
2024-11-12 10:54:37 -08:00
Arjun 53c523444e add: new go-fuzz targets
Signed-off-by: Arjun <pkillarjun@protonmail.com>
2024-11-11 20:45:49 +05:30
Devin Carr 1f3e3045ad TUN-8701: Add metrics and adjust logs for datagram v3
Closes TUN-8701
2024-11-07 11:02:55 -08:00
Devin Carr 952622a965 TUN-8709: Add session migration for datagram v3
When a registration response from cloudflared gets lost on it's way back to the edge, the edge service will retry and send another registration request. Since cloudflared already has bound the local UDP socket for the provided request id, we want to re-send the registration response.

There are three types of retries that the edge will send:

1. A retry from the same QUIC connection index; cloudflared will just respond back with a registration response and reset the idle timer for the session.
2. A retry from a different QUIC connection index; cloudflared will need to migrate the current session connection to this new QUIC connection and reset the idle timer for the session.
3. A retry to a different cloudflared connector; cloudflared will eventually time the session out since no further packets will arrive to the session at the original connector.

Closes TUN-8709
2024-11-06 12:06:07 -08:00
emilioarvidisaksson aa7abe7581 Fixed 404 in README.md to TryCloudflare 2024-11-04 23:25:53 +01:00
Mads Jon Nielsen 2cbe125e0b Merge branch 'cloudflare:master' into tunnel-health 2024-10-28 10:33:08 +01:00
Mads Jon Nielsen f2016e7f63 Merge branch 'cloudflare:master' into tunnel-health 2024-10-12 19:21:03 +02:00
Mads Jon Nielsen 4ce0e1bd38 Merge branch 'cloudflare:master' into tunnel-health 2024-10-07 18:38:43 +02:00
Hrushikesh Deshpande 659da3ebba Update semgrep.yml 2024-09-24 21:40:50 -04:00
hrushikeshdeshpande 244248f2b7 Update semgrep.yml
Updating Semgrep.yml file - Semgrep is a tool that will be used to scan Cloudflare's public repos for Supply chain, code and secrets. This work is part of Application & Product Security team's initiative to onboard Semgrep onto all of Cloudflare's public repos.

In case of any questions, please reach out to "Hrushikesh Deshpande" on cf internal chat.
2024-09-21 13:18:55 -04:00
Mads Jon Nielsen 72f8ecc521 Merge branch 'cloudflare:master' into tunnel-health 2024-08-22 07:39:56 +02:00
Mads Jon Nielsen bec84aeb7b Merge branch 'cloudflare:master' into tunnel-health 2024-08-01 10:46:25 +02:00
Mads Jon Nielsen 037f056d0c Merge branch 'cloudflare:master' into tunnel-health 2024-07-22 16:47:55 +02:00
Mads Jon Nielsen 5d229fd917 Merge branch 'cloudflare:master' into tunnel-health 2024-06-21 07:55:18 +02:00
Mads Jon Nielsen e759716ce7 Merge branch 'cloudflare:master' into tunnel-health 2024-05-21 09:09:53 +02:00
Mads Jon Nielsen 2941825577 Make sure body is properly printed when status code not equals 200 2024-04-23 09:00:39 +02:00
Mads Jon Nielsen d094e52bd1 Prettify Usage and Description 2024-04-23 08:53:49 +02:00
Mads Jon Nielsen d6b03fbabf Prettify Usage and Description 2024-04-23 08:53:32 +02:00
Mads Jon Nielsen e03f53144b Rename command description and usage 2024-04-23 08:37:37 +02:00
Mads Jon Nielsen b342c7403c Rename command to ready 2024-04-23 08:35:25 +02:00
Mads Jon Nielsen 37210ff661 Use /healthcheck over /ready 2024-04-23 08:20:53 +02:00
Mads Jon Nielsen 6db3cb2f1b Merge branch 'cloudflare:master' into tunnel-health 2024-04-23 08:08:04 +02:00
Mads Jon Nielsen 2bf652c6fd Update subcommands.go 2024-01-03 10:38:55 +01:00
Mads Jon Nielsen 521f5632d7 Just use c.String("metrics")
Co-authored-by: Julien Laffaye <jlaffaye@freebsd.org>
2024-01-03 10:37:41 +01:00
Mads Jon Nielsen f10247db90 Add cloudflared tunnel health command 2023-12-28 19:50:49 +01:00
79 changed files with 5765 additions and 537 deletions
+3 -4
View File
@@ -1,4 +1,3 @@
on:
pull_request: {}
workflow_dispatch: {}
@@ -12,14 +11,14 @@ name: Semgrep config
jobs:
semgrep:
name: semgrep/ci
runs-on: ubuntu-20.04
runs-on: ubuntu-latest
env:
SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}
SEMGREP_URL: https://cloudflare.semgrep.dev
SEMGREP_APP_URL: https://cloudflare.semgrep.dev
SEMGREP_VERSION_CHECK_URL: https://cloudflare.semgrep.dev/api/check-version
container:
image: returntocorp/semgrep
image: semgrep/semgrep
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- run: semgrep ci
+3 -1
View File
@@ -5,7 +5,9 @@ FROM golang:1.22.5 as builder
ENV GO111MODULE=on \
CGO_ENABLED=0 \
TARGET_GOOS=${TARGET_GOOS} \
TARGET_GOARCH=${TARGET_GOARCH}
TARGET_GOARCH=${TARGET_GOARCH} \
CONTAINER_BUILD=1
WORKDIR /go/src/github.com/cloudflare/cloudflared/
+16
View File
@@ -30,6 +30,10 @@ ifdef PACKAGE_MANAGER
VERSION_FLAGS := $(VERSION_FLAGS) -X "github.com/cloudflare/cloudflared/cmd/cloudflared/updater.BuiltForPackageManager=$(PACKAGE_MANAGER)"
endif
ifdef CONTAINER_BUILD
VERSION_FLAGS := $(VERSION_FLAGS) -X "github.com/cloudflare/cloudflared/metrics.Runtime=virtual"
endif
LINK_FLAGS :=
ifeq ($(FIPS), true)
LINK_FLAGS := -linkmode=external -extldflags=-static $(LINK_FLAGS)
@@ -165,6 +169,18 @@ cover:
# Generate the HTML report that can be viewed from the browser in CI.
$Q go tool cover -html ".cover/c.out" -o .cover/all.html
.PHONY: fuzz
fuzz:
@go test -fuzz=FuzzIPDecoder -fuzztime=600s ./packet
@go test -fuzz=FuzzICMPDecoder -fuzztime=600s ./packet
@go test -fuzz=FuzzSessionWrite -fuzztime=600s ./quic/v3
@go test -fuzz=FuzzSessionServe -fuzztime=600s ./quic/v3
@go test -fuzz=FuzzRegistrationDatagram -fuzztime=600s ./quic/v3
@go test -fuzz=FuzzPayloadDatagram -fuzztime=600s ./quic/v3
@go test -fuzz=FuzzRegistrationResponseDatagram -fuzztime=600s ./quic/v3
@go test -fuzz=FuzzNewIdentity -fuzztime=600s ./tracing
@go test -fuzz=FuzzNewAccessValidator -fuzztime=600s ./validation
.PHONY: install-go
install-go:
rm -rf ${CF_GO_PATH}
+1 -1
View File
@@ -49,7 +49,7 @@ Once installed, you can authenticate `cloudflared` into your Cloudflare account
## TryCloudflare
Want to test Cloudflare Tunnel before adding a website to Cloudflare? You can do so with TryCloudflare using the documentation [available here](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/trycloudflare).
Want to test Cloudflare Tunnel before adding a website to Cloudflare? You can do so with TryCloudflare using the documentation [available here](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/).
## Deprecated versions
+39
View File
@@ -1,3 +1,42 @@
2024.12.1
- 2024-12-10 TUN-8795: update createrepo to createrepo_c to fix the release_pkgs.py script
2024.12.0
- 2024-12-09 TUN-8640: Add ICMP support for datagram V3
- 2024-12-09 TUN-8789: make python package installation consistent
- 2024-12-06 TUN-8781: Add Trixie, drop Buster. Default to Bookworm
- 2024-12-05 TUN-8775: Make sure the session Close can only be called once
- 2024-12-04 TUN-8725: implement diagnostic procedure
- 2024-12-04 TUN-8767: include raw output from network collector in diagnostic zipfile
- 2024-12-04 TUN-8770: add cli configuration and tunnel configuration to diagnostic zipfile
- 2024-12-04 TUN-8768: add job report to diagnostic zipfile
- 2024-12-03 TUN-8726: implement compression routine to be used in diagnostic procedure
- 2024-12-03 TUN-8732: implement port selection algorithm
- 2024-12-03 TUN-8762: fix argument order when invoking tracert and modify network info output parsing.
- 2024-12-03 TUN-8769: fix k8s log collector arguments
- 2024-12-03 TUN-8727: extend client to include function to get cli configuration and tunnel configuration
- 2024-11-29 TUN-8729: implement network collection for diagnostic procedure
- 2024-11-29 TUN-8727: implement metrics, runtime, system, and tunnelstate in diagnostic http client
- 2024-11-27 TUN-8733: add log collection for docker
- 2024-11-27 TUN-8734: add log collection for kubernetes
- 2024-11-27 TUN-8640: Refactor ICMPRouter to support new ICMPResponders
- 2024-11-26 TUN-8735: add managed/local log collection
- 2024-11-25 TUN-8728: implement diag/tunnel endpoint
- 2024-11-25 TUN-8730: implement diag/configuration
- 2024-11-22 TUN-8737: update metrics server port selection
- 2024-11-22 TUN-8731: Implement diag/system endpoint
- 2024-11-21 TUN-8748: Migrated datagram V3 flows to use migrated context
2024.11.1
- 2024-11-18 Add cloudflared tunnel ready command
- 2024-11-14 Make metrics a requirement for tunnel ready command
- 2024-11-12 TUN-8701: Simplify flow registration logs for datagram v3
- 2024-11-11 add: new go-fuzz targets
- 2024-11-07 TUN-8701: Add metrics and adjust logs for datagram v3
- 2024-11-06 TUN-8709: Add session migration for datagram v3
- 2024-11-04 Fixed 404 in README.md to TryCloudflare
- 2024-09-24 Update semgrep.yml
2024.11.0
- 2024-11-05 VULN-66059: remove ssh server tests
- 2024-11-04 TUN-8700: Add datagram v3 muxer
+39 -34
View File
@@ -1,8 +1,9 @@
pinned_go: &pinned_go go-boring=1.22.5-1
build_dir: &build_dir /cfsetup_build
default-flavor: bullseye
buster: &buster
default-flavor: bookworm
bullseye: &bullseye
build-linux:
build_dir: *build_dir
builddeps: &build_deps
@@ -31,8 +32,8 @@ buster: &buster
builddeps: *build_deps
pre-cache: *build_pre_cache
post-cache:
- make cover
# except FIPS and macos
- make cover
# except FIPS and macos
build-linux-release:
build_dir: *build_dir
builddeps: &build_deps_release
@@ -46,19 +47,17 @@ buster: &buster
- python3-pip
- python3-setuptools
- wget
pre-cache: &build_release_pre_cache
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- pip3 install boto3==1.22.9
- pip3 install python-gnupg==0.4.9
- python3-venv
post-cache:
- python3 -m venv env
- . /cfsetup_build/env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55 boto3==1.22.9 python-gnupg==0.4.9
# build all packages (except macos and FIPS) and move them to /cfsetup/built_artifacts
- ./build-packages.sh
# handle FIPS separately so that we built with gofips compiler
build-linux-fips-release:
build_dir: *build_dir
builddeps: *build_deps_release
pre-cache: *build_release_pre_cache
post-cache:
# same logic as above, but for FIPS packages only
- ./build-packages-fips.sh
@@ -110,7 +109,7 @@ buster: &buster
- export GOOS=linux
- export GOARCH=arm64
- export NIGHTLY=true
#- export FIPS=true # TUN-7595
# - export FIPS=true # TUN-7595
- export ORIGINAL_NAME=true
- make cloudflared-deb
build-deb-arm64:
@@ -133,12 +132,14 @@ buster: &buster
# libmsi and libgcab are libraries the wixl binary depends on.
- libmsi-dev
- libgcab-dev
- python3-venv
pre-cache:
- wget https://github.com/sudarshan-reddy/msitools/releases/download/v0.101b/wixl -P /usr/local/bin
- chmod a+x /usr/local/bin/wixl
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- .teamcity/package-windows.sh
test:
build_dir: *build_dir
@@ -172,18 +173,22 @@ buster: &buster
build_dir: *build_dir
builddeps: &build_deps_component_test
- *pinned_go
- python3.7
- python3
- python3-pip
- python3-setuptools
# procps installs the ps command which is needed in test_sysv_service because the init script
# uses ps pid to determine if the agent is running
# procps installs the ps command which is needed in test_sysv_service
# because the init script uses ps pid to determine if the agent is
# running
- procps
- python3-venv
pre-cache-copy-paths:
- component-tests/requirements.txt
pre-cache: &component_test_pre_cache
- sudo pip3 install --upgrade -r component-tests/requirements.txt
post-cache: &component_test_post_cache
# Creates and routes a Named Tunnel for this build. Also constructs config file from env vars.
- python3 -m venv env
- . env/bin/activate
- pip install --upgrade -r component-tests/requirements.txt
# Creates and routes a Named Tunnel for this build. Also constructs
# config file from env vars.
- python3 component-tests/setup.py --type create
- pytest component-tests -o log_cli=true --log-cli-level=INFO
# The Named Tunnel is deleted and its route unprovisioned here.
@@ -193,7 +198,6 @@ buster: &buster
builddeps: *build_deps_component_test
pre-cache-copy-paths:
- component-tests/requirements.txt
pre-cache: *component_test_pre_cache
post-cache: *component_test_post_cache
github-release-dryrun:
build_dir: *build_dir
@@ -204,10 +208,11 @@ buster: &buster
- libffi-dev
- python3-setuptools
- python3-pip
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- make github-release-dryrun
github-release:
build_dir: *build_dir
@@ -218,10 +223,11 @@ buster: &buster
- libffi-dev
- python3-setuptools
- python3-pip
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55
- make github-release
r2-linux-release:
build_dir: *build_dir
@@ -237,14 +243,13 @@ buster: &buster
- python3-setuptools
- python3-pip
- reprepro
- createrepo
pre-cache:
- pip3 install pynacl==1.4.0
- pip3 install pygithub==1.55
- pip3 install boto3==1.22.9
- pip3 install python-gnupg==0.4.9
- createrepo-c
- python3-venv
post-cache:
- python3 -m venv env
- . env/bin/activate
- pip install pynacl==1.4.0 pygithub==1.55 boto3==1.22.9 python-gnupg==0.4.9
- make r2-linux-release
bullseye: *buster
bookworm: *buster
bookworm: *bullseye
trixie: *bullseye
+119 -7
View File
@@ -28,6 +28,7 @@ import (
"github.com/cloudflare/cloudflared/config"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/credentials"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/edgediscovery"
"github.com/cloudflare/cloudflared/features"
"github.com/cloudflare/cloudflared/ingress"
@@ -39,6 +40,7 @@ import (
"github.com/cloudflare/cloudflared/supervisor"
"github.com/cloudflare/cloudflared/tlsconfig"
"github.com/cloudflare/cloudflared/tunneldns"
"github.com/cloudflare/cloudflared/tunnelstate"
"github.com/cloudflare/cloudflared/validation"
)
@@ -125,6 +127,94 @@ var (
"most likely you already have a conflicting record there. You can also rerun this command with --%s to overwrite "+
"any existing DNS records for this hostname.", overwriteDNSFlag)
deprecatedClassicTunnelErr = fmt.Errorf("Classic tunnels have been deprecated, please use Named Tunnels. (https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/)")
// TODO: TUN-8756 the list below denotes the flags that do not possess any kind of sensitive information
// however this approach is not maintainble in the long-term.
nonSecretFlagsList = []string{
"config",
"autoupdate-freq",
"no-autoupdate",
"metrics",
"pidfile",
"url",
"hello-world",
"socks5",
"proxy-connect-timeout",
"proxy-tls-timeout",
"proxy-tcp-keepalive",
"proxy-no-happy-eyeballs",
"proxy-keepalive-connections",
"proxy-keepalive-timeout",
"proxy-connection-timeout",
"proxy-expect-continue-timeout",
"http-host-header",
"origin-server-name",
"unix-socket",
"origin-ca-pool",
"no-tls-verify",
"no-chunked-encoding",
"http2-origin",
"management-hostname",
"service-op-ip",
"local-ssh-port",
"ssh-idle-timeout",
"ssh-max-timeout",
"bucket-name",
"region-name",
"s3-url-host",
"host-key-path",
"ssh-server",
"bastion",
"proxy-address",
"proxy-port",
"loglevel",
"transport-loglevel",
"logfile",
"log-directory",
"trace-output",
"proxy-dns",
"proxy-dns-port",
"proxy-dns-address",
"proxy-dns-upstream",
"proxy-dns-max-upstream-conns",
"proxy-dns-bootstrap",
"is-autoupdated",
"edge",
"region",
"edge-ip-version",
"edge-bind-address",
"cacert",
"hostname",
"id",
"lb-pool",
"api-url",
"metrics-update-freq",
"tag",
"heartbeat-interval",
"heartbeat-count",
"max-edge-addr-retries",
"retries",
"ha-connections",
"rpc-timeout",
"write-stream-timeout",
"quic-disable-pmtu-discovery",
"quic-connection-level-flow-control-limit",
"quic-stream-level-flow-control-limit",
"label",
"grace-period",
"compression-quality",
"use-reconnect-token",
"dial-edge-timeout",
"stdin-control",
"name",
"ui",
"quick-service",
"max-fetch-size",
"post-quantum",
"management-diagnostics",
"protocol",
"overwrite-dns",
"help",
}
)
func Flags() []cli.Flag {
@@ -139,6 +229,7 @@ func Commands() []*cli.Command {
buildVirtualNetworkSubcommand(false),
buildRunCommand(),
buildListCommand(),
buildReadyCommand(),
buildInfoCommand(),
buildIngressSubcommand(),
buildDeleteCommand(),
@@ -419,7 +510,7 @@ func StartServer(
// Disable ICMP packet routing for quick tunnels
if quickTunnelURL != "" {
tunnelConfig.PacketConfig = nil
tunnelConfig.ICMPRouterServer = nil
}
internalRules := []ingress.Rule{}
@@ -447,19 +538,34 @@ func StartServer(
return err
}
metricsListener, err := listeners.Listen("tcp", c.String("metrics"))
metricsListener, err := metrics.CreateMetricsListener(&listeners, c.String("metrics"))
if err != nil {
log.Err(err).Msg("Error opening metrics server listener")
return errors.Wrap(err, "Error opening metrics server listener")
}
defer metricsListener.Close()
wg.Add(1)
go func() {
defer wg.Done()
readinessServer := metrics.NewReadyServer(log, clientID)
observer.RegisterSink(readinessServer)
tracker := tunnelstate.NewConnTracker(log)
observer.RegisterSink(tracker)
readinessServer := metrics.NewReadyServer(clientID, tracker)
diagnosticHandler := diagnostic.NewDiagnosticHandler(
log,
0,
diagnostic.NewSystemCollectorImpl(buildInfo.CloudflaredVersion),
tunnelConfig.NamedTunnel.Credentials.TunnelID,
clientID,
tracker,
c,
nonSecretFlagsList,
)
metricsConfig := metrics.Config{
ReadyServer: readinessServer,
DiagnosticHandler: diagnosticHandler,
QuickTunnelHostname: quickTunnelURL,
Orchestrator: orchestrator,
}
@@ -856,9 +962,15 @@ func configureCloudflaredFlags(shouldHide bool) []cli.Flag {
Hidden: shouldHide,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "metrics",
Value: "localhost:",
Usage: "Listen address for metrics reporting.",
Name: "metrics",
Value: metrics.GetMetricsDefaultAddress(metrics.Runtime),
Usage: fmt.Sprintf(
`Listen address for metrics reporting. If no address is passed cloudflared will try to bind to %v.
If all are unavailable, a random port will be used. Note that when running cloudflared from an virtual
environment the default address binds to all interfaces, hence, it is important to isolate the host
and virtualized host network stacks from each other`,
metrics.GetMetricsKnownAddresses(metrics.Runtime),
),
EnvVars: []string{"TUNNEL_METRICS"},
Hidden: shouldHide,
}),
+7 -13
View File
@@ -252,11 +252,11 @@ func prepareTunnelConfig(
QUICConnectionLevelFlowControlLimit: c.Uint64(quicConnLevelFlowControlLimit),
QUICStreamLevelFlowControlLimit: c.Uint64(quicStreamLevelFlowControlLimit),
}
packetConfig, err := newPacketConfig(c, log)
icmpRouter, err := newICMPRouter(c, log)
if err != nil {
log.Warn().Err(err).Msg("ICMP proxy feature is disabled")
} else {
tunnelConfig.PacketConfig = packetConfig
tunnelConfig.ICMPRouterServer = icmpRouter
}
orchestratorConfig := &orchestration.Config{
Ingress: &ingressRules,
@@ -351,7 +351,7 @@ func adjustIPVersionByBindAddress(ipVersion allregions.ConfigIPVersion, ip net.I
}
}
func newPacketConfig(c *cli.Context, logger *zerolog.Logger) (*ingress.GlobalRouterConfig, error) {
func newICMPRouter(c *cli.Context, logger *zerolog.Logger) (ingress.ICMPRouterServer, error) {
ipv4Src, err := determineICMPv4Src(c.String("icmpv4-src"), logger)
if err != nil {
return nil, errors.Wrap(err, "failed to determine IPv4 source address for ICMP proxy")
@@ -368,16 +368,11 @@ func newPacketConfig(c *cli.Context, logger *zerolog.Logger) (*ingress.GlobalRou
logger.Info().Msgf("ICMP proxy will use %s as source for IPv6", ipv6Src)
}
icmpRouter, err := ingress.NewICMPRouter(ipv4Src, ipv6Src, zone, logger, icmpFunnelTimeout)
icmpRouter, err := ingress.NewICMPRouter(ipv4Src, ipv6Src, logger, icmpFunnelTimeout)
if err != nil {
return nil, err
}
return &ingress.GlobalRouterConfig{
ICMPRouter: icmpRouter,
IPv4Src: ipv4Src,
IPv6Src: ipv6Src,
Zone: zone,
}, nil
return icmpRouter, nil
}
func determineICMPv4Src(userDefinedSrc string, logger *zerolog.Logger) (netip.Addr, error) {
@@ -407,13 +402,12 @@ type interfaceIP struct {
func determineICMPv6Src(userDefinedSrc string, logger *zerolog.Logger, ipv4Src netip.Addr) (addr netip.Addr, zone string, err error) {
if userDefinedSrc != "" {
userDefinedIP, zone, _ := strings.Cut(userDefinedSrc, "%")
addr, err := netip.ParseAddr(userDefinedIP)
addr, err := netip.ParseAddr(userDefinedSrc)
if err != nil {
return netip.Addr{}, "", err
}
if addr.Is6() {
return addr, zone, nil
return addr, addr.Zone(), nil
}
return netip.Addr{}, "", fmt.Errorf("expect IPv6, but %s is IPv4", userDefinedSrc)
}
+35
View File
@@ -5,6 +5,8 @@ import (
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
@@ -397,6 +399,39 @@ func fmtConnections(connections []cfapi.Connection, showRecentlyDisconnected boo
return strings.Join(output, ", ")
}
func buildReadyCommand() *cli.Command {
return &cli.Command{
Name: "ready",
Action: cliutil.ConfiguredAction(readyCommand),
Usage: "Call /ready endpoint and return proper exit code",
UsageText: "cloudflared tunnel [tunnel command options] ready [subcommand options]",
Description: "cloudflared tunnel ready will return proper exit code based on the /ready endpoint",
Flags: []cli.Flag{},
CustomHelpTemplate: commandHelpTemplate(),
}
}
func readyCommand(c *cli.Context) error {
metricsOpts := c.String("metrics")
if !c.IsSet("metrics") {
return fmt.Errorf("--metrics has to be provided")
}
requestURL := fmt.Sprintf("http://%s/ready", metricsOpts)
res, err := http.Get(requestURL)
if err != nil {
return err
}
if res.StatusCode != 200 {
body, err := io.ReadAll(res.Body)
if err != nil {
return err
}
return fmt.Errorf("http://%s/ready endpoint returned status code %d\n%s", metricsOpts, res.StatusCode, body)
}
return nil
}
func buildInfoCommand() *cli.Command {
return &cli.Command{
Name: "info",
+1 -1
View File
@@ -102,7 +102,7 @@ func (c *controlStream) ServeControlStream(
c.observer.metrics.regSuccess.WithLabelValues("registerConnection").Inc()
c.observer.logConnected(registrationDetails.UUID, c.connIndex, registrationDetails.Location, c.edgeAddress, c.protocol)
c.observer.sendConnectedEvent(c.connIndex, c.protocol, registrationDetails.Location)
c.observer.sendConnectedEvent(c.connIndex, c.protocol, registrationDetails.Location, c.edgeAddress)
c.connectedFuse.Connected()
// if conn index is 0 and tunnel is not remotely managed, then send local ingress rules configuration
+8 -5
View File
@@ -1,12 +1,15 @@
package connection
import "net"
// Event is something that happened to a connection, e.g. disconnection or registration.
type Event struct {
Index uint8
EventType Status
Location string
Protocol Protocol
URL string
Index uint8
EventType Status
Location string
Protocol Protocol
URL string
EdgeAddress net.IP
}
// Status is the status of a connection.
+2 -3
View File
@@ -47,7 +47,6 @@ func (o *Observer) RegisterSink(sink EventSink) {
}
func (o *Observer) logConnected(connectionID uuid.UUID, connIndex uint8, location string, address net.IP, protocol Protocol) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Location: location})
o.log.Info().
Int(management.EventTypeKey, int(management.Cloudflared)).
Str(LogFieldConnectionID, connectionID.String()).
@@ -63,8 +62,8 @@ func (o *Observer) sendRegisteringEvent(connIndex uint8) {
o.sendEvent(Event{Index: connIndex, EventType: RegisteringTunnel})
}
func (o *Observer) sendConnectedEvent(connIndex uint8, protocol Protocol, location string) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Protocol: protocol, Location: location})
func (o *Observer) sendConnectedEvent(connIndex uint8, protocol Protocol, location string, edgeAddress net.IP) {
o.sendEvent(Event{Index: connIndex, EventType: Connected, Protocol: protocol, Location: location, EdgeAddress: edgeAddress})
}
func (o *Observer) SendURL(url string) {
-27
View File
@@ -7,7 +7,6 @@ import (
"io"
"net"
"net/http"
"net/netip"
"strconv"
"strings"
"sync/atomic"
@@ -18,7 +17,6 @@ import (
"github.com/rs/zerolog"
"golang.org/x/sync/errgroup"
"github.com/cloudflare/cloudflared/packet"
cfdquic "github.com/cloudflare/cloudflared/quic"
"github.com/cloudflare/cloudflared/tracing"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
@@ -417,28 +415,3 @@ func (np *nopCloserReadWriter) Close() error {
return nil
}
// muxerWrapper wraps DatagramMuxerV2 to satisfy the packet.FunnelUniPipe interface
type muxerWrapper struct {
muxer *cfdquic.DatagramMuxerV2
}
func (rp *muxerWrapper) SendPacket(dst netip.Addr, pk packet.RawPacket) error {
return rp.muxer.SendPacket(cfdquic.RawPacket(pk))
}
func (rp *muxerWrapper) ReceivePacket(ctx context.Context) (packet.RawPacket, error) {
pk, err := rp.muxer.ReceivePacket(ctx)
if err != nil {
return packet.RawPacket{}, err
}
rawPacket, ok := pk.(cfdquic.RawPacket)
if ok {
return packet.RawPacket(rawPacket), nil
}
return packet.RawPacket{}, fmt.Errorf("unexpected packet type %+v", pk)
}
func (rp *muxerWrapper) Close() error {
return nil
}
+2 -1
View File
@@ -752,7 +752,8 @@ func testTunnelConnection(t *testing.T, serverAddr netip.AddrPort, index uint8)
sessionDemuxChan := make(chan *packet.Session, 4)
datagramMuxer := cfdquic.NewDatagramMuxerV2(conn, &log, sessionDemuxChan)
sessionManager := datagramsession.NewManager(&log, datagramMuxer.SendToSession, sessionDemuxChan)
packetRouter := ingress.NewPacketRouter(nil, datagramMuxer, &log)
var connIndex uint8 = 0
packetRouter := ingress.NewPacketRouter(nil, datagramMuxer, connIndex, &log)
datagramConn := &datagramV2Connection{
conn,
+3 -2
View File
@@ -54,7 +54,8 @@ type datagramV2Connection struct {
func NewDatagramV2Connection(ctx context.Context,
conn quic.Connection,
packetConfig *ingress.GlobalRouterConfig,
icmpRouter ingress.ICMPRouter,
index uint8,
rpcTimeout time.Duration,
streamWriteTimeout time.Duration,
logger *zerolog.Logger,
@@ -62,7 +63,7 @@ func NewDatagramV2Connection(ctx context.Context,
sessionDemuxChan := make(chan *packet.Session, demuxChanCapacity)
datagramMuxer := cfdquic.NewDatagramMuxerV2(conn, logger, sessionDemuxChan)
sessionManager := datagramsession.NewManager(logger, datagramMuxer.SendToSession, sessionDemuxChan)
packetRouter := ingress.NewPacketRouter(packetConfig, datagramMuxer, logger)
packetRouter := ingress.NewPacketRouter(icmpRouter, datagramMuxer, index, logger)
return &datagramV2Connection{
conn,
+11 -1
View File
@@ -10,6 +10,8 @@ import (
"github.com/quic-go/quic-go"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/management"
cfdquic "github.com/cloudflare/cloudflared/quic/v3"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
)
@@ -24,9 +26,17 @@ type datagramV3Connection struct {
func NewDatagramV3Connection(ctx context.Context,
conn quic.Connection,
sessionManager cfdquic.SessionManager,
icmpRouter ingress.ICMPRouter,
index uint8,
metrics cfdquic.Metrics,
logger *zerolog.Logger,
) DatagramSessionHandler {
datagramMuxer := cfdquic.NewDatagramConn(conn, sessionManager, logger)
log := logger.
With().
Int(management.EventTypeKey, int(management.UDP)).
Uint8(LogFieldConnIndex, index).
Logger()
datagramMuxer := cfdquic.NewDatagramConn(conn, sessionManager, icmpRouter, index, metrics, &log)
return &datagramV3Connection{
conn,
+194
View File
@@ -0,0 +1,194 @@
package diagnostic
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"github.com/cloudflare/cloudflared/logger"
)
type httpClient struct {
http.Client
baseURL *url.URL
}
func NewHTTPClient() *httpClient {
httpTransport := http.Transport{
TLSHandshakeTimeout: defaultTimeout,
ResponseHeaderTimeout: defaultTimeout,
}
return &httpClient{
http.Client{
Transport: &httpTransport,
Timeout: defaultTimeout,
},
nil,
}
}
func (client *httpClient) SetBaseURL(baseURL *url.URL) {
client.baseURL = baseURL
}
func (client *httpClient) GET(ctx context.Context, endpoint string) (*http.Response, error) {
if client.baseURL == nil {
return nil, ErrNoBaseURL
}
url := client.baseURL.JoinPath(endpoint)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url.String(), nil)
if err != nil {
return nil, fmt.Errorf("error creating GET request: %w", err)
}
req.Header.Add("Accept", "application/json;version=1")
response, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("error GET request: %w", err)
}
return response, nil
}
type LogConfiguration struct {
logFile string
logDirectory string
uid int // the uid of the user that started cloudflared
}
func (client *httpClient) GetLogConfiguration(ctx context.Context) (*LogConfiguration, error) {
response, err := client.GET(ctx, cliConfigurationEndpoint)
if err != nil {
return nil, err
}
defer response.Body.Close()
var data map[string]string
if err := json.NewDecoder(response.Body).Decode(&data); err != nil {
return nil, fmt.Errorf("failed to decode body: %w", err)
}
uidStr, exists := data[configurationKeyUID]
if !exists {
return nil, ErrKeyNotFound
}
uid, err := strconv.Atoi(uidStr)
if err != nil {
return nil, fmt.Errorf("error convertin pid to int: %w", err)
}
logFile, exists := data[logger.LogFileFlag]
if exists {
return &LogConfiguration{logFile, "", uid}, nil
}
logDirectory, exists := data[logger.LogDirectoryFlag]
if exists {
return &LogConfiguration{"", logDirectory, uid}, nil
}
// No log configured may happen when cloudflared is executed as a managed service or
// when containerized
return &LogConfiguration{"", "", uid}, nil
}
func (client *httpClient) GetMemoryDump(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, memoryDumpEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetGoroutineDump(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, goroutineDumpEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetTunnelState(ctx context.Context) (*TunnelState, error) {
response, err := client.GET(ctx, tunnelStateEndpoint)
if err != nil {
return nil, err
}
defer response.Body.Close()
var state TunnelState
if err := json.NewDecoder(response.Body).Decode(&state); err != nil {
return nil, fmt.Errorf("failed to decode body: %w", err)
}
return &state, nil
}
func (client *httpClient) GetSystemInformation(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, systemInformationEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetMetrics(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, metricsEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetTunnelConfiguration(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, tunnelConfigurationEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func (client *httpClient) GetCliConfiguration(ctx context.Context, writer io.Writer) error {
response, err := client.GET(ctx, cliConfigurationEndpoint)
if err != nil {
return err
}
return copyToWriter(response, writer)
}
func copyToWriter(response *http.Response, writer io.Writer) error {
defer response.Body.Close()
_, err := io.Copy(writer, response.Body)
if err != nil {
return fmt.Errorf("error writing metrics: %w", err)
}
return nil
}
type HTTPClient interface {
GetLogConfiguration(ctx context.Context) (*LogConfiguration, error)
GetMemoryDump(ctx context.Context, writer io.Writer) error
GetGoroutineDump(ctx context.Context, writer io.Writer) error
GetTunnelState(ctx context.Context) (*TunnelState, error)
GetSystemInformation(ctx context.Context, writer io.Writer) error
GetMetrics(ctx context.Context, writer io.Writer) error
GetCliConfiguration(ctx context.Context, writer io.Writer) error
GetTunnelConfiguration(ctx context.Context, writer io.Writer) error
}
+37
View File
@@ -0,0 +1,37 @@
package diagnostic
import "time"
const (
defaultCollectorTimeout = time.Second * 10 // This const define the timeout value of a collector operation.
collectorField = "collector" // used for logging purposes
systemCollectorName = "system" // used for logging purposes
tunnelStateCollectorName = "tunnelState" // used for logging purposes
configurationCollectorName = "configuration" // used for logging purposes
defaultTimeout = 15 * time.Second // timeout for the collectors
twoWeeksOffset = -14 * 24 * time.Hour // maximum offset for the logs
logFilename = "cloudflared_logs.txt" // name of the output log file
configurationKeyUID = "uid" // Key used to set and get the UID value from the configuration map
tailMaxNumberOfLines = "10000" // maximum number of log lines from a virtual runtime (docker or kubernetes)
// Endpoints used by the diagnostic HTTP Client.
cliConfigurationEndpoint = "/diag/configuration"
tunnelStateEndpoint = "/diag/tunnel"
systemInformationEndpoint = "/diag/system"
memoryDumpEndpoint = "debug/pprof/heap"
goroutineDumpEndpoint = "debug/pprof/goroutine"
metricsEndpoint = "metrics"
tunnelConfigurationEndpoint = "/config"
// Base for filenames of the diagnostic procedure
systemInformationBaseName = "systeminformation.json"
metricsBaseName = "metrics.txt"
zipName = "cloudflared-diag"
heapPprofBaseName = "heap.pprof"
goroutinePprofBaseName = "goroutine.pprof"
networkBaseName = "network.json"
rawNetworkBaseName = "raw-network.txt"
tunnelStateBaseName = "tunnelstate.json"
cliConfigurationBaseName = "cli-configuration.json"
configurationBaseName = "configuration.json"
taskResultBaseName = "task-result.json"
)
+536
View File
@@ -0,0 +1,536 @@
package diagnostic
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/url"
"os"
"path/filepath"
"sync"
"time"
"github.com/rs/zerolog"
network "github.com/cloudflare/cloudflared/diagnostic/network"
)
const (
taskSuccess = "success"
taskFailure = "failure"
jobReportName = "job report"
tunnelStateJobName = "tunnel state"
systemInformationJobName = "system information"
goroutineJobName = "goroutine profile"
heapJobName = "heap profile"
metricsJobName = "metrics"
logInformationJobName = "log information"
rawNetworkInformationJobName = "raw network information"
networkInformationJobName = "network information"
cliConfigurationJobName = "cli configuration"
configurationJobName = "configuration"
)
// Struct used to hold the results of different routines executing the network collection.
type taskResult struct {
Result string `json:"result,omitempty"`
Err error `json:"error,omitempty"`
path string
}
// Struct used to hold the results of different routines executing the network collection.
type networkCollectionResult struct {
name string
info []*network.Hop
raw string
err error
}
// This type represents the most common functions from the diagnostic http client
// functions.
type collectToWriterFunc func(ctx context.Context, writer io.Writer) error
// This type represents the common denominator among all the collection procedures.
type collectFunc func(ctx context.Context) (string, error)
// collectJob is an internal struct that denotes holds the information necessary
// to run a collection job.
type collectJob struct {
jobName string
fn collectFunc
bypass bool
}
// The Toggles structure denotes the available toggles for the diagnostic procedure.
// Each toggle enables/disables tasks from the diagnostic.
type Toggles struct {
NoDiagLogs bool
NoDiagMetrics bool
NoDiagSystem bool
NoDiagRuntime bool
NoDiagNetwork bool
}
// The Options structure holds every option necessary for
// the diagnostic procedure to work.
type Options struct {
KnownAddresses []string
Address string
ContainerID string
PodID string
Toggles Toggles
}
func collectLogs(
ctx context.Context,
client HTTPClient,
diagContainer, diagPod string,
) (string, error) {
var collector LogCollector
if diagPod != "" {
collector = NewKubernetesLogCollector(diagContainer, diagPod)
} else if diagContainer != "" {
collector = NewDockerLogCollector(diagContainer)
} else {
collector = NewHostLogCollector(client)
}
logInformation, err := collector.Collect(ctx)
if err != nil {
return "", fmt.Errorf("error collecting logs: %w", err)
}
if logInformation.isDirectory {
return CopyFilesFromDirectory(logInformation.path)
}
if logInformation.wasCreated {
return logInformation.path, nil
}
logHandle, err := os.Open(logInformation.path)
if err != nil {
return "", fmt.Errorf("error opening log file while collecting logs: %w", err)
}
defer logHandle.Close()
outputLogHandle, err := os.Create(filepath.Join(os.TempDir(), logFilename))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer outputLogHandle.Close()
_, err = io.Copy(outputLogHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying logs while collecting logs: %w", err)
}
return outputLogHandle.Name(), err
}
func collectNetworkResultRoutine(
ctx context.Context,
collector network.NetworkCollector,
hostname string,
useIPv4 bool,
results chan networkCollectionResult,
) {
const (
hopsNo = 5
timeout = time.Second * 5
)
name := hostname
if useIPv4 {
name += "-v4"
} else {
name += "-v6"
}
hops, raw, err := collector.Collect(ctx, network.NewTraceOptions(hopsNo, timeout, hostname, useIPv4))
if err != nil {
if raw == "" {
// An error happened and there is no raw output
results <- networkCollectionResult{name, nil, "", err}
} else {
// An error happened and there is raw output then write to file
results <- networkCollectionResult{name, nil, raw, nil}
}
} else {
results <- networkCollectionResult{name, hops, raw, nil}
}
}
func gatherNetworkInformation(ctx context.Context) map[string]networkCollectionResult {
networkCollector := network.NetworkCollectorImpl{}
hostAndIPversionPairs := []struct {
host string
useV4 bool
}{
{"region1.v2.argotunnel.com", true},
{"region1.v2.argotunnel.com", false},
{"region2.v2.argotunnel.com", true},
{"region2.v2.argotunnel.com", false},
}
// the number of results is known thus use len to avoid footguns
results := make(chan networkCollectionResult, len(hostAndIPversionPairs))
var wgroup sync.WaitGroup
for _, item := range hostAndIPversionPairs {
wgroup.Add(1)
go func() {
defer wgroup.Done()
collectNetworkResultRoutine(ctx, &networkCollector, item.host, item.useV4, results)
}()
}
// Wait for routines to end.
wgroup.Wait()
resultMap := make(map[string]networkCollectionResult)
for range len(hostAndIPversionPairs) {
result := <-results
if result.err != nil {
continue
}
resultMap[result.name] = result
}
return resultMap
}
func networkInformationCollectors() (rawNetworkCollector, jsonNetworkCollector collectFunc) {
// The network collector is an operation that takes most of the diagnostic time, thus,
// the sync.Once is used to memoize the result of the collector and then create different
// outputs.
var once sync.Once
var resultMap map[string]networkCollectionResult
rawNetworkCollector = func(ctx context.Context) (string, error) {
once.Do(func() { resultMap = gatherNetworkInformation(ctx) })
return rawNetworkInformationWriter(resultMap)
}
jsonNetworkCollector = func(ctx context.Context) (string, error) {
once.Do(func() { resultMap = gatherNetworkInformation(ctx) })
return jsonNetworkInformationWriter(resultMap)
}
return rawNetworkCollector, jsonNetworkCollector
}
func rawNetworkInformationWriter(resultMap map[string]networkCollectionResult) (string, error) {
networkDumpHandle, err := os.Create(filepath.Join(os.TempDir(), rawNetworkBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer networkDumpHandle.Close()
for k, v := range resultMap {
_, err := networkDumpHandle.WriteString(k + "\n" + v.raw + "\n")
if err != nil {
return "", fmt.Errorf("error writing raw network information: %w", err)
}
}
return networkDumpHandle.Name(), nil
}
func jsonNetworkInformationWriter(resultMap map[string]networkCollectionResult) (string, error) {
jsonMap := make(map[string][]*network.Hop, len(resultMap))
for k, v := range resultMap {
jsonMap[k] = v.info
}
networkDumpHandle, err := os.Create(filepath.Join(os.TempDir(), networkBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer networkDumpHandle.Close()
err = json.NewEncoder(networkDumpHandle).Encode(jsonMap)
if err != nil {
return "", fmt.Errorf("error encoding network information results: %w", err)
}
return networkDumpHandle.Name(), nil
}
func collectFromEndpointAdapter(collect collectToWriterFunc, fileName string) collectFunc {
return func(ctx context.Context) (string, error) {
dumpHandle, err := os.Create(filepath.Join(os.TempDir(), fileName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer dumpHandle.Close()
err = collect(ctx, dumpHandle)
if err != nil {
return "", ErrCreatingTemporaryFile
}
return dumpHandle.Name(), nil
}
}
func tunnelStateCollectEndpointAdapter(client HTTPClient, tunnel *TunnelState, fileName string) collectFunc {
endpointFunc := func(ctx context.Context, writer io.Writer) error {
if tunnel == nil {
// When the metrics server is not passed the diagnostic will query all known hosts
// and get the tunnel state, however, when the metrics server is passed that won't
// happen hence the check for nil in this function.
tunnelResponse, err := client.GetTunnelState(ctx)
if err != nil {
return fmt.Errorf("error retrieving tunnel state: %w", err)
}
tunnel = tunnelResponse
}
encoder := json.NewEncoder(writer)
err := encoder.Encode(tunnel)
return fmt.Errorf("error encoding tunnel state: %w", err)
}
return collectFromEndpointAdapter(endpointFunc, fileName)
}
// resolveInstanceBaseURL is responsible to
// resolve the base URL of the instance that should be diagnosed.
// To resolve the instance it may be necessary to query the
// /diag/tunnel endpoint of the known instances, thus, if a single
// instance is found its state is also returned; if multiple instances
// are found then their states are returned in an array along with an
// error.
func resolveInstanceBaseURL(
metricsServerAddress string,
log *zerolog.Logger,
client *httpClient,
addresses []string,
) (*url.URL, *TunnelState, []*AddressableTunnelState, error) {
if metricsServerAddress != "" {
url, err := url.Parse(metricsServerAddress)
if err != nil {
return nil, nil, nil, fmt.Errorf("provided address is not valid: %w", err)
}
if url.Scheme == "" {
url.Scheme = "http://"
}
return url, nil, nil, nil
}
tunnelState, foundTunnelStates, err := FindMetricsServer(log, client, addresses)
if err != nil {
return nil, nil, foundTunnelStates, err
}
return tunnelState.URL, tunnelState.TunnelState, nil, nil
}
func createJobs(
client *httpClient,
tunnel *TunnelState,
diagContainer string,
diagPod string,
noDiagSystem bool,
noDiagRuntime bool,
noDiagMetrics bool,
noDiagLogs bool,
noDiagNetwork bool,
) []collectJob {
rawNetworkCollectorFunc, jsonNetworkCollectorFunc := networkInformationCollectors()
jobs := []collectJob{
{
jobName: tunnelStateJobName,
fn: tunnelStateCollectEndpointAdapter(client, tunnel, tunnelStateBaseName),
bypass: false,
},
{
jobName: systemInformationJobName,
fn: collectFromEndpointAdapter(client.GetSystemInformation, systemInformationBaseName),
bypass: noDiagSystem,
},
{
jobName: goroutineJobName,
fn: collectFromEndpointAdapter(client.GetGoroutineDump, goroutinePprofBaseName),
bypass: noDiagRuntime,
},
{
jobName: heapJobName,
fn: collectFromEndpointAdapter(client.GetMemoryDump, heapPprofBaseName),
bypass: noDiagRuntime,
},
{
jobName: metricsJobName,
fn: collectFromEndpointAdapter(client.GetMetrics, metricsBaseName),
bypass: noDiagMetrics,
},
{
jobName: logInformationJobName,
fn: func(ctx context.Context) (string, error) {
return collectLogs(ctx, client, diagContainer, diagPod)
},
bypass: noDiagLogs,
},
{
jobName: rawNetworkInformationJobName,
fn: rawNetworkCollectorFunc,
bypass: noDiagNetwork,
},
{
jobName: networkInformationJobName,
fn: jsonNetworkCollectorFunc,
bypass: noDiagNetwork,
},
{
jobName: cliConfigurationJobName,
fn: collectFromEndpointAdapter(client.GetCliConfiguration, cliConfigurationBaseName),
bypass: false,
},
{
jobName: configurationJobName,
fn: collectFromEndpointAdapter(client.GetTunnelConfiguration, configurationBaseName),
bypass: false,
},
}
return jobs
}
func createTaskReport(taskReport map[string]taskResult) (string, error) {
dumpHandle, err := os.Create(filepath.Join(os.TempDir(), taskResultBaseName))
if err != nil {
return "", ErrCreatingTemporaryFile
}
defer dumpHandle.Close()
err = json.NewEncoder(dumpHandle).Encode(taskReport)
if err != nil {
return "", fmt.Errorf("error encoding task results: %w", err)
}
return dumpHandle.Name(), nil
}
func runJobs(ctx context.Context, jobs []collectJob, log *zerolog.Logger) map[string]taskResult {
jobReport := make(map[string]taskResult, len(jobs))
for _, job := range jobs {
if job.bypass {
continue
}
log.Info().Msgf("Collecting %s...", job.jobName)
path, err := job.fn(ctx)
var result taskResult
if err != nil {
result = taskResult{Result: taskFailure, Err: err, path: path}
log.Error().Err(err).Msgf("Job: %s finished with error.", job.jobName)
} else {
result = taskResult{Result: taskSuccess, Err: nil, path: path}
log.Info().Msgf("Collected %s.", job.jobName)
}
jobReport[job.jobName] = result
}
taskReportName, err := createTaskReport(jobReport)
var result taskResult
if err != nil {
result = taskResult{
Result: taskFailure,
path: taskReportName,
Err: err,
}
} else {
result = taskResult{
Result: taskSuccess,
path: taskReportName,
Err: nil,
}
}
jobReport[jobReportName] = result
return jobReport
}
func RunDiagnostic(
log *zerolog.Logger,
options Options,
) ([]*AddressableTunnelState, error) {
client := NewHTTPClient()
baseURL, tunnel, foundTunnels, err := resolveInstanceBaseURL(options.Address, log, client, options.KnownAddresses)
if err != nil {
return foundTunnels, err
}
log.Info().Msgf("Selected server %s starting diagnostic...", baseURL.String())
client.SetBaseURL(baseURL)
const timeout = 45 * time.Second
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
jobs := createJobs(
client,
tunnel,
options.ContainerID,
options.PodID,
options.Toggles.NoDiagSystem,
options.Toggles.NoDiagRuntime,
options.Toggles.NoDiagMetrics,
options.Toggles.NoDiagLogs,
options.Toggles.NoDiagNetwork,
)
jobsReport := runJobs(ctx, jobs, log)
paths := make([]string, 0)
for _, v := range jobsReport {
paths = append(paths, v.path)
defer func() {
if !errors.Is(v.Err, ErrCreatingTemporaryFile) {
os.Remove(v.path)
}
}()
}
zipfile, err := CreateDiagnosticZipFile(zipName, paths)
if err != nil {
if zipfile != "" {
os.Remove(zipfile)
}
return nil, err
}
log.Info().Msgf("Diagnostic file written: %v", zipfile)
return nil, nil
}
+140
View File
@@ -0,0 +1,140 @@
package diagnostic
import (
"archive/zip"
"context"
"fmt"
"io"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"github.com/google/uuid"
"github.com/rs/zerolog"
)
// CreateDiagnosticZipFile create a zip file with the contents from the all
// files paths. The files will be written in the root of the zip file.
// In case of an error occurs after whilst writing to the zip file
// this will be removed.
func CreateDiagnosticZipFile(base string, paths []string) (zipFileName string, err error) {
// Create a zip file with all files from paths added to the root
suffix := time.Now().Format(time.RFC3339)
zipFileName = base + "-" + suffix + ".zip"
zipFileName = strings.ReplaceAll(zipFileName, ":", "-")
archive, cerr := os.Create(zipFileName)
if cerr != nil {
return "", fmt.Errorf("error creating file %s: %w", zipFileName, cerr)
}
archiveWriter := zip.NewWriter(archive)
defer func() {
archiveWriter.Close()
archive.Close()
if err != nil {
os.Remove(zipFileName)
}
}()
for _, file := range paths {
if file == "" {
continue
}
var handle *os.File
handle, err = os.Open(file)
if err != nil {
return "", fmt.Errorf("error opening file %s: %w", zipFileName, err)
}
defer handle.Close()
// Keep the base only to not create sub directories in the
// zip file.
var writer io.Writer
writer, err = archiveWriter.Create(filepath.Base(file))
if err != nil {
return "", fmt.Errorf("error creating archive writer from %s: %w", file, err)
}
if _, err = io.Copy(writer, handle); err != nil {
return "", fmt.Errorf("error copying file %s: %w", file, err)
}
}
zipFileName = archive.Name()
return zipFileName, nil
}
type AddressableTunnelState struct {
*TunnelState
URL *url.URL
}
func findMetricsServerPredicate(tunnelID, connectorID uuid.UUID) func(state *TunnelState) bool {
if tunnelID != uuid.Nil && connectorID != uuid.Nil {
return func(state *TunnelState) bool {
return state.ConnectorID == connectorID && state.TunnelID == tunnelID
}
} else if tunnelID == uuid.Nil && connectorID != uuid.Nil {
return func(state *TunnelState) bool {
return state.ConnectorID == connectorID
}
} else if tunnelID != uuid.Nil && connectorID == uuid.Nil {
return func(state *TunnelState) bool {
return state.TunnelID == tunnelID
}
}
return func(*TunnelState) bool {
return true
}
}
// The FindMetricsServer will try to find the metrics server url.
// There are two possible error scenarios:
// 1. No instance is found which will only return ErrMetricsServerNotFound
// 2. Multiple instances are found which will return an array of state and ErrMultipleMetricsServerFound
// In case of success, only the state for the instance is returned.
func FindMetricsServer(
log *zerolog.Logger,
client *httpClient,
addresses []string,
) (*AddressableTunnelState, []*AddressableTunnelState, error) {
instances := make([]*AddressableTunnelState, 0)
for _, address := range addresses {
url, err := url.Parse("http://" + address)
if err != nil {
log.Debug().Err(err).Msgf("error parsing address %s", address)
continue
}
client.SetBaseURL(url)
state, err := client.GetTunnelState(context.Background())
if err == nil {
instances = append(instances, &AddressableTunnelState{state, url})
} else {
log.Debug().Err(err).Msgf("error getting tunnel state from address %s", address)
}
}
if len(instances) == 0 {
return nil, nil, ErrMetricsServerNotFound
}
if len(instances) == 1 {
return instances[0], nil, nil
}
return nil, instances, ErrMultipleMetricsServerFound
}
+147
View File
@@ -0,0 +1,147 @@
package diagnostic_test
import (
"context"
"net/http"
"net/url"
"sync"
"testing"
"time"
"github.com/facebookgo/grace/gracenet"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/metrics"
"github.com/cloudflare/cloudflared/tunnelstate"
)
func helperCreateServer(t *testing.T, listeners *gracenet.Net, tunnelID uuid.UUID, connectorID uuid.UUID) func() {
t.Helper()
listener, err := metrics.CreateMetricsListener(listeners, "localhost:0")
require.NoError(t, err)
log := zerolog.Nop()
tracker := tunnelstate.NewConnTracker(&log)
handler := diagnostic.NewDiagnosticHandler(&log, 0, nil, tunnelID, connectorID, tracker, nil, []string{})
router := http.NewServeMux()
router.HandleFunc("/diag/tunnel", handler.TunnelStateHandler)
server := &http.Server{
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
Handler: router,
}
var wgroup sync.WaitGroup
wgroup.Add(1)
go func() {
defer wgroup.Done()
_ = server.Serve(listener)
}()
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
cleanUp := func() {
_ = server.Shutdown(ctx)
cancel()
wgroup.Wait()
}
return cleanUp
}
func TestFindMetricsServer_WhenSingleServerIsRunning_ReturnState(t *testing.T) {
listeners := gracenet.Net{}
tid1 := uuid.New()
cid1 := uuid.New()
cleanUp := helperCreateServer(t, &listeners, tid1, cid1)
defer cleanUp()
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
url1, err := url.Parse("http://localhost:20241")
require.NoError(t, err)
tunnel1 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid1,
Connections: nil,
},
URL: url1,
}
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
if err != nil {
require.ErrorIs(t, err, diagnostic.ErrMultipleMetricsServerFound)
}
assert.Equal(t, tunnel1, state)
assert.Nil(t, tunnels)
}
func TestFindMetricsServer_WhenMultipleServerAreRunning_ReturnError(t *testing.T) {
listeners := gracenet.Net{}
tid1 := uuid.New()
cid1 := uuid.New()
cid2 := uuid.New()
cleanUp := helperCreateServer(t, &listeners, tid1, cid1)
defer cleanUp()
cleanUp = helperCreateServer(t, &listeners, tid1, cid2)
defer cleanUp()
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
url1, err := url.Parse("http://localhost:20241")
require.NoError(t, err)
url2, err := url.Parse("http://localhost:20242")
require.NoError(t, err)
tunnel1 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid1,
Connections: nil,
},
URL: url1,
}
tunnel2 := &diagnostic.AddressableTunnelState{
TunnelState: &diagnostic.TunnelState{
TunnelID: tid1,
ConnectorID: cid2,
Connections: nil,
},
URL: url2,
}
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
if err != nil {
require.ErrorIs(t, err, diagnostic.ErrMultipleMetricsServerFound)
}
assert.Nil(t, state)
assert.Equal(t, []*diagnostic.AddressableTunnelState{tunnel1, tunnel2}, tunnels)
}
func TestFindMetricsServer_WhenNoInstanceIsRuning_ReturnError(t *testing.T) {
log := zerolog.Nop()
client := diagnostic.NewHTTPClient()
addresses := metrics.GetMetricsKnownAddresses("host")
state, tunnels, err := diagnostic.FindMetricsServer(&log, client, addresses[:])
require.ErrorIs(t, err, diagnostic.ErrMetricsServerNotFound)
assert.Nil(t, state)
assert.Nil(t, tunnels)
}
+28
View File
@@ -0,0 +1,28 @@
package diagnostic
import (
"errors"
)
var (
// Error used when there is no log directory available.
ErrManagedLogNotFound = errors.New("managed log directory not found")
// Error used when it is not possible to collect logs using the log configuration.
ErrLogConfigurationIsInvalid = errors.New("provided log configuration is invalid")
// Error used when parsing the fields of the output of collector.
ErrInsufficientLines = errors.New("insufficient lines")
// Error used when parsing the lines of the output of collector.
ErrInsuficientFields = errors.New("insufficient fields")
// Error used when given key is not found while parsing KV.
ErrKeyNotFound = errors.New("key not found")
// Error used when there is no disk volume information available.
ErrNoVolumeFound = errors.New("no disk volume information found")
// Error user when the base url of the diagnostic client is not provided.
ErrNoBaseURL = errors.New("no base url")
// Error used when no metrics server is found listening to the known addresses list (check [metrics.GetMetricsKnownAddresses]).
ErrMetricsServerNotFound = errors.New("metrics server not found")
// Error used when multiple metrics server are found listening to the known addresses list (check [metrics.GetMetricsKnownAddresses]).
ErrMultipleMetricsServerFound = errors.New("multiple metrics server found")
// Error used when a temporary file creation fails within the diagnostic procedure
ErrCreatingTemporaryFile = errors.New("temporary file creation failed")
)
+207
View File
@@ -0,0 +1,207 @@
package diagnostic
import (
"context"
"encoding/json"
"net/http"
"os"
"path/filepath"
"strconv"
"time"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/urfave/cli/v2"
"github.com/cloudflare/cloudflared/logger"
"github.com/cloudflare/cloudflared/tunnelstate"
)
type Handler struct {
log *zerolog.Logger
timeout time.Duration
systemCollector SystemCollector
tunnelID uuid.UUID
connectorID uuid.UUID
tracker *tunnelstate.ConnTracker
cli *cli.Context
flagInclusionList []string
}
func NewDiagnosticHandler(
log *zerolog.Logger,
timeout time.Duration,
systemCollector SystemCollector,
tunnelID uuid.UUID,
connectorID uuid.UUID,
tracker *tunnelstate.ConnTracker,
cli *cli.Context,
flagInclusionList []string,
) *Handler {
logger := log.With().Logger()
if timeout == 0 {
timeout = defaultCollectorTimeout
}
return &Handler{
log: &logger,
timeout: timeout,
systemCollector: systemCollector,
tunnelID: tunnelID,
connectorID: connectorID,
tracker: tracker,
cli: cli,
flagInclusionList: flagInclusionList,
}
}
func (handler *Handler) InstallEndpoints(router *http.ServeMux) {
router.HandleFunc(cliConfigurationEndpoint, handler.ConfigurationHandler)
router.HandleFunc(tunnelStateEndpoint, handler.TunnelStateHandler)
router.HandleFunc(systemInformationEndpoint, handler.SystemHandler)
}
func (handler *Handler) SystemHandler(writer http.ResponseWriter, request *http.Request) {
logger := handler.log.With().Str(collectorField, systemCollectorName).Logger()
logger.Info().Msg("Collection started")
defer logger.Info().Msg("Collection finished")
ctx, cancel := context.WithTimeout(request.Context(), handler.timeout)
defer cancel()
info, rawInfo, err := handler.systemCollector.Collect(ctx)
if err != nil {
logger.Error().Err(err).Msg("error occurred whilst collecting system information")
if rawInfo != "" {
logger.Info().Msg("using raw information fallback")
bytes := []byte(rawInfo)
writeResponse(writer, bytes, &logger)
} else {
logger.Error().Msg("no raw information available")
writer.WriteHeader(http.StatusInternalServerError)
}
return
}
if info == nil {
logger.Error().Msgf("system information collection is nil")
writer.WriteHeader(http.StatusInternalServerError)
}
encoder := json.NewEncoder(writer)
err = encoder.Encode(info)
if err != nil {
logger.Error().Err(err).Msgf("error occurred whilst serializing information")
writer.WriteHeader(http.StatusInternalServerError)
}
}
type TunnelState struct {
TunnelID uuid.UUID `json:"tunnelID,omitempty"`
ConnectorID uuid.UUID `json:"connectorID,omitempty"`
Connections []tunnelstate.IndexedConnectionInfo `json:"connections,omitempty"`
}
func (handler *Handler) TunnelStateHandler(writer http.ResponseWriter, _ *http.Request) {
log := handler.log.With().Str(collectorField, tunnelStateCollectorName).Logger()
log.Info().Msg("Collection started")
defer log.Info().Msg("Collection finished")
body := TunnelState{
handler.tunnelID,
handler.connectorID,
handler.tracker.GetActiveConnections(),
}
encoder := json.NewEncoder(writer)
err := encoder.Encode(body)
if err != nil {
handler.log.Error().Err(err).Msgf("error occurred whilst serializing information")
writer.WriteHeader(http.StatusInternalServerError)
}
}
func (handler *Handler) ConfigurationHandler(writer http.ResponseWriter, _ *http.Request) {
log := handler.log.With().Str(collectorField, configurationCollectorName).Logger()
log.Info().Msg("Collection started")
defer func() {
log.Info().Msg("Collection finished")
}()
flagsNames := handler.cli.FlagNames()
flags := make(map[string]string, len(flagsNames))
for _, flag := range flagsNames {
value := handler.cli.String(flag)
// empty values are not relevant
if value == "" {
continue
}
// exclude flags that are sensitive
isIncluded := handler.isFlagIncluded(flag)
if !isIncluded {
continue
}
switch flag {
case logger.LogDirectoryFlag:
fallthrough
case logger.LogFileFlag:
{
// the log directory may be relative to the instance thus it must be resolved
absolute, err := filepath.Abs(value)
if err != nil {
handler.log.Error().Err(err).Msgf("could not convert %s path to absolute", flag)
} else {
flags[flag] = absolute
}
}
default:
flags[flag] = value
}
}
// The UID is included to help the
// diagnostic tool to understand
// if this instance is managed or not.
flags[configurationKeyUID] = strconv.Itoa(os.Getuid())
encoder := json.NewEncoder(writer)
err := encoder.Encode(flags)
if err != nil {
handler.log.Error().Err(err).Msgf("error occurred whilst serializing response")
writer.WriteHeader(http.StatusInternalServerError)
}
}
func (handler *Handler) isFlagIncluded(flag string) bool {
isIncluded := false
for _, include := range handler.flagInclusionList {
if include == flag {
isIncluded = true
break
}
}
return isIncluded
}
func writeResponse(w http.ResponseWriter, bytes []byte, logger *zerolog.Logger) {
bytesWritten, err := w.Write(bytes)
if err != nil {
logger.Error().Err(err).Msg("error occurred writing response")
} else if bytesWritten != len(bytes) {
logger.Error().Msgf("error incomplete write response %d/%d", bytesWritten, len(bytes))
}
}
+254
View File
@@ -0,0 +1,254 @@
package diagnostic_test
import (
"context"
"encoding/json"
"errors"
"flag"
"io"
"net"
"net/http"
"net/http/httptest"
"testing"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/urfave/cli/v2"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/diagnostic"
"github.com/cloudflare/cloudflared/tunnelstate"
)
type SystemCollectorMock struct{}
const (
systemInformationKey = "sikey"
rawInformationKey = "rikey"
errorKey = "errkey"
)
func buildCliContext(t *testing.T, flags map[string]string) *cli.Context {
t.Helper()
flagSet := flag.NewFlagSet("", flag.PanicOnError)
ctx := cli.NewContext(cli.NewApp(), flagSet, nil)
for k, v := range flags {
flagSet.String(k, v, "")
err := ctx.Set(k, v)
require.NoError(t, err)
}
return ctx
}
func newTrackerFromConns(t *testing.T, connections []tunnelstate.IndexedConnectionInfo) *tunnelstate.ConnTracker {
t.Helper()
log := zerolog.Nop()
tracker := tunnelstate.NewConnTracker(&log)
for _, conn := range connections {
tracker.OnTunnelEvent(connection.Event{
Index: conn.Index,
EventType: connection.Connected,
Protocol: conn.Protocol,
EdgeAddress: conn.EdgeAddress,
})
}
return tracker
}
func setCtxValuesForSystemCollector(
systemInfo *diagnostic.SystemInformation,
rawInfo string,
err error,
) context.Context {
ctx := context.Background()
ctx = context.WithValue(ctx, systemInformationKey, systemInfo)
ctx = context.WithValue(ctx, rawInformationKey, rawInfo)
ctx = context.WithValue(ctx, errorKey, err)
return ctx
}
func (*SystemCollectorMock) Collect(ctx context.Context) (*diagnostic.SystemInformation, string, error) {
si, _ := ctx.Value(systemInformationKey).(*diagnostic.SystemInformation)
ri, _ := ctx.Value(rawInformationKey).(string)
err, _ := ctx.Value(errorKey).(error)
return si, ri, err
}
func TestSystemHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
systemInfo *diagnostic.SystemInformation
rawInfo string
err error
statusCode int
}{
{
name: "happy path",
systemInfo: diagnostic.NewSystemInformation(
0, 0, 0, 0,
"string", "string", "string", "string",
"string", "string", nil,
),
rawInfo: "",
err: nil,
statusCode: http.StatusOK,
},
{
name: "on error and raw info", systemInfo: nil,
rawInfo: "raw info", err: errors.New("an error"), statusCode: http.StatusOK,
},
{
name: "on error and no raw info", systemInfo: nil,
rawInfo: "", err: errors.New("an error"), statusCode: http.StatusInternalServerError,
},
{
name: "malformed response", systemInfo: nil, rawInfo: "", err: nil, statusCode: http.StatusInternalServerError,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
handler := diagnostic.NewDiagnosticHandler(&log, 0, &SystemCollectorMock{}, uuid.New(), uuid.New(), nil, nil, nil)
recorder := httptest.NewRecorder()
ctx := setCtxValuesForSystemCollector(tCase.systemInfo, tCase.rawInfo, tCase.err)
request, err := http.NewRequestWithContext(ctx, http.MethodGet, "/diag/syste,", nil)
require.NoError(t, err)
handler.SystemHandler(recorder, request)
assert.Equal(t, tCase.statusCode, recorder.Code)
if tCase.statusCode == http.StatusOK && tCase.systemInfo != nil {
var response diagnostic.SystemInformation
decoder := json.NewDecoder(recorder.Body)
err = decoder.Decode(&response)
require.NoError(t, err)
assert.Equal(t, tCase.systemInfo, &response)
} else if tCase.statusCode == http.StatusOK && tCase.rawInfo != "" {
rawBytes, err := io.ReadAll(recorder.Body)
require.NoError(t, err)
assert.Equal(t, tCase.rawInfo, string(rawBytes))
}
})
}
}
func TestTunnelStateHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
tunnelID uuid.UUID
clientID uuid.UUID
connections []tunnelstate.IndexedConnectionInfo
}{
{
name: "case1",
tunnelID: uuid.New(),
clientID: uuid.New(),
},
{
name: "case2",
tunnelID: uuid.New(),
clientID: uuid.New(),
connections: []tunnelstate.IndexedConnectionInfo{{
ConnectionInfo: tunnelstate.ConnectionInfo{
IsConnected: true,
Protocol: connection.QUIC,
EdgeAddress: net.IPv4(100, 100, 100, 100),
},
Index: 0,
}},
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
tracker := newTrackerFromConns(t, tCase.connections)
handler := diagnostic.NewDiagnosticHandler(&log, 0, nil, tCase.tunnelID, tCase.clientID, tracker, nil, nil)
recorder := httptest.NewRecorder()
handler.TunnelStateHandler(recorder, nil)
decoder := json.NewDecoder(recorder.Body)
var response diagnostic.TunnelState
err := decoder.Decode(&response)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, recorder.Code)
assert.Equal(t, tCase.tunnelID, response.TunnelID)
assert.Equal(t, tCase.clientID, response.ConnectorID)
assert.Equal(t, tCase.connections, response.Connections)
})
}
}
func TestConfigurationHandler(t *testing.T) {
t.Parallel()
log := zerolog.Nop()
tests := []struct {
name string
flags map[string]string
expected map[string]string
}{
{
name: "empty cli",
flags: make(map[string]string),
expected: map[string]string{
"uid": "0",
},
},
{
name: "cli with flags",
flags: map[string]string{
"a": "a",
"b": "a",
"c": "a",
"d": "a",
},
expected: map[string]string{
"b": "a",
"c": "a",
"d": "a",
"uid": "0",
},
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
var response map[string]string
t.Parallel()
ctx := buildCliContext(t, tCase.flags)
handler := diagnostic.NewDiagnosticHandler(&log, 0, nil, uuid.New(), uuid.New(), nil, ctx, []string{"b", "c", "d"})
recorder := httptest.NewRecorder()
handler.ConfigurationHandler(recorder, nil)
decoder := json.NewDecoder(recorder.Body)
err := decoder.Decode(&response)
require.NoError(t, err)
_, ok := response["uid"]
assert.True(t, ok)
delete(tCase.expected, "uid")
delete(response, "uid")
assert.Equal(t, http.StatusOK, recorder.Code)
assert.Equal(t, tCase.expected, response)
})
}
}
+34
View File
@@ -0,0 +1,34 @@
package diagnostic
import (
"context"
)
// Represents the path of the log file or log directory.
// This struct is meant to give some ergonimics regarding
// the logging information.
type LogInformation struct {
path string // path to a file or directory
wasCreated bool // denotes if `path` was created
isDirectory bool // denotes if `path` is a directory
}
func NewLogInformation(
path string,
wasCreated bool,
isDirectory bool,
) *LogInformation {
return &LogInformation{
path,
wasCreated,
isDirectory,
}
}
type LogCollector interface {
// This function is responsible for returning a path to a single file
// whose contents are the logs of a cloudflared instance.
// A new file may be create by a LogCollector, thus, its the caller
// responsibility to remove the newly create file.
Collect(ctx context.Context) (*LogInformation, error)
}
+47
View File
@@ -0,0 +1,47 @@
package diagnostic
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
)
type DockerLogCollector struct {
containerID string // This member identifies the container by identifier or name
}
func NewDockerLogCollector(containerID string) *DockerLogCollector {
return &DockerLogCollector{
containerID,
}
}
func (collector *DockerLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
tmp := os.TempDir()
outputHandle, err := os.Create(filepath.Join(tmp, logFilename))
if err != nil {
return nil, fmt.Errorf("error opening output file: %w", err)
}
defer outputHandle.Close()
// Calculate 2 weeks ago
since := time.Now().Add(twoWeeksOffset).Format(time.RFC3339)
command := exec.CommandContext(
ctx,
"docker",
"logs",
"--tail",
tailMaxNumberOfLines,
"--since",
since,
collector.containerID,
)
return PipeCommandOutputToFile(command, outputHandle)
}
+73
View File
@@ -0,0 +1,73 @@
package diagnostic
import (
"context"
"fmt"
"os"
"path/filepath"
"runtime"
)
const (
linuxManagedLogsPath = "/var/log/cloudflared.err"
darwinManagedLogsPath = "/Library/Logs/com.cloudflare.cloudflared.err.log"
)
type HostLogCollector struct {
client HTTPClient
}
func NewHostLogCollector(client HTTPClient) *HostLogCollector {
return &HostLogCollector{
client,
}
}
func getServiceLogPath() (string, error) {
switch runtime.GOOS {
case "darwin":
{
path := darwinManagedLogsPath
if _, err := os.Stat(path); err == nil {
return path, nil
}
userHomeDir, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("error getting user home: %w", err)
}
return filepath.Join(userHomeDir, darwinManagedLogsPath), nil
}
case "linux":
{
return linuxManagedLogsPath, nil
}
default:
return "", ErrManagedLogNotFound
}
}
func (collector *HostLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
logConfiguration, err := collector.client.GetLogConfiguration(ctx)
if err != nil {
return nil, fmt.Errorf("error getting log configuration: %w", err)
}
if logConfiguration.uid == 0 {
path, err := getServiceLogPath()
if err != nil {
return nil, err
}
return NewLogInformation(path, false, false), nil
}
if logConfiguration.logFile != "" {
return NewLogInformation(logConfiguration.logFile, false, false), nil
} else if logConfiguration.logDirectory != "" {
return NewLogInformation(logConfiguration.logDirectory, false, true), nil
}
return nil, ErrLogConfigurationIsInvalid
}
+63
View File
@@ -0,0 +1,63 @@
package diagnostic
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
)
type KubernetesLogCollector struct {
containerID string // This member identifies the container by identifier or name
pod string // This member identifies the pod where the container is deployed
}
func NewKubernetesLogCollector(containerID, pod string) *KubernetesLogCollector {
return &KubernetesLogCollector{
containerID,
pod,
}
}
func (collector *KubernetesLogCollector) Collect(ctx context.Context) (*LogInformation, error) {
tmp := os.TempDir()
outputHandle, err := os.Create(filepath.Join(tmp, logFilename))
if err != nil {
return nil, fmt.Errorf("error opening output file: %w", err)
}
defer outputHandle.Close()
var command *exec.Cmd
// Calculate 2 weeks ago
since := time.Now().Add(twoWeeksOffset).Format(time.RFC3339)
if collector.containerID != "" {
command = exec.CommandContext(
ctx,
"kubectl",
"logs",
collector.pod,
"--since-time",
since,
"--tail",
tailMaxNumberOfLines,
"-c",
collector.containerID,
)
} else {
command = exec.CommandContext(
ctx,
"kubectl",
"logs",
collector.pod,
"--since-time",
since,
"--tail",
tailMaxNumberOfLines,
)
}
return PipeCommandOutputToFile(command, outputHandle)
}
+90
View File
@@ -0,0 +1,90 @@
package diagnostic
import (
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
)
func PipeCommandOutputToFile(command *exec.Cmd, outputHandle *os.File) (*LogInformation, error) {
stdoutReader, err := command.StdoutPipe()
if err != nil {
return nil, fmt.Errorf(
"error retrieving output from command '%s': %w",
command.String(),
err,
)
}
if err := command.Start(); err != nil {
return nil, fmt.Errorf(
"error running command '%s': %w",
command.String(),
err,
)
}
_, err = io.Copy(outputHandle, stdoutReader)
if err != nil {
return nil, fmt.Errorf(
"error copying output from %s to file %s: %w",
command.String(),
outputHandle.Name(),
err,
)
}
if err := command.Wait(); err != nil {
return nil, fmt.Errorf(
"error waiting from command '%s': %w",
command.String(),
err,
)
}
return NewLogInformation(outputHandle.Name(), true, false), nil
}
func CopyFilesFromDirectory(path string) (string, error) {
// rolling logs have as suffix the current date thus
// when iterating the path files they are already in
// chronological order
files, err := os.ReadDir(path)
if err != nil {
return "", fmt.Errorf("error reading directory %s: %w", path, err)
}
outputHandle, err := os.Create(filepath.Join(os.TempDir(), logFilename))
if err != nil {
return "", fmt.Errorf("creating file %s: %w", outputHandle.Name(), err)
}
defer outputHandle.Close()
for _, file := range files {
logHandle, err := os.Open(filepath.Join(path, file.Name()))
if err != nil {
return "", fmt.Errorf("error opening file %s:%w", file.Name(), err)
}
defer logHandle.Close()
_, err = io.Copy(outputHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying file %s:%w", logHandle.Name(), err)
}
}
logHandle, err := os.Open(filepath.Join(path, "cloudflared.log"))
if err != nil {
return "", fmt.Errorf("error opening file %s:%w", logHandle.Name(), err)
}
defer logHandle.Close()
_, err = io.Copy(outputHandle, logHandle)
if err != nil {
return "", fmt.Errorf("error copying file %s:%w", logHandle.Name(), err)
}
return outputHandle.Name(), nil
}
+77
View File
@@ -0,0 +1,77 @@
package diagnostic
import (
"context"
"errors"
"time"
)
const MicrosecondsFactor = 1000.0
var ErrEmptyDomain = errors.New("domain must not be empty")
// For now only support ICMP is provided.
type IPVersion int
const (
V4 IPVersion = iota
V6 IPVersion = iota
)
type Hop struct {
Hop uint8 `json:"hop,omitempty"` // hop number along the route
Domain string `json:"domain,omitempty"` // domain and/or ip of the hop, this field will be '*' if the hop is a timeout
Rtts []time.Duration `json:"rtts,omitempty"` // RTT measurements in microseconds
}
type TraceOptions struct {
ttl uint64 // number of hops to perform
timeout time.Duration // wait timeout for each response
address string // address to trace
useV4 bool
}
func NewTimeoutHop(
hop uint8,
) *Hop {
// Whenever there is a hop in the format of 'N * * *'
// it means that the hop in the path didn't answer to
// any probe.
return NewHop(
hop,
"*",
nil,
)
}
func NewHop(hop uint8, domain string, rtts []time.Duration) *Hop {
return &Hop{
hop,
domain,
rtts,
}
}
func NewTraceOptions(
ttl uint64,
timeout time.Duration,
address string,
useV4 bool,
) TraceOptions {
return TraceOptions{
ttl,
timeout,
address,
useV4,
}
}
type NetworkCollector interface {
// Performs a trace route operation with the specified options.
// In case the trace fails, it will return a non-nil error and
// it may return a string which represents the raw information
// obtained.
// In case it is successful it will only return an array of Hops
// an empty string and a nil error.
Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error)
}
+78
View File
@@ -0,0 +1,78 @@
//go:build darwin || linux
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
"time"
)
type NetworkCollectorImpl struct{}
func (tracer *NetworkCollectorImpl) Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error) {
args := []string{
"-I",
"-w",
strconv.FormatInt(int64(options.timeout.Seconds()), 10),
"-m",
strconv.FormatUint(options.ttl, 10),
options.address,
}
var command string
switch options.useV4 {
case false:
command = "traceroute6"
default:
command = "traceroute"
}
process := exec.CommandContext(ctx, command, args...)
return decodeNetworkOutputToFile(process, DecodeLine)
}
func DecodeLine(text string) (*Hop, error) {
fields := strings.Fields(text)
parts := []string{}
filter := func(s string) bool { return s != "*" && s != "ms" }
for _, field := range fields {
if filter(field) {
parts = append(parts, field)
}
}
index, err := strconv.ParseUint(parts[0], 10, 8)
if err != nil {
return nil, fmt.Errorf("couldn't parse index from timeout hop: %w", err)
}
if len(parts) == 1 {
return NewTimeoutHop(uint8(index)), nil
}
domain := ""
rtts := []time.Duration{}
for _, part := range parts[1:] {
rtt, err := strconv.ParseFloat(part, 64)
if err != nil {
domain += part + " "
} else {
rtts = append(rtts, time.Duration(rtt*MicrosecondsFactor))
}
}
domain, _ = strings.CutSuffix(domain, " ")
if domain == "" {
return nil, ErrEmptyDomain
}
return NewHop(uint8(index), domain, rtts), nil
}
+173
View File
@@ -0,0 +1,173 @@
//go:build darwin || linux
package diagnostic_test
import (
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
diagnostic "github.com/cloudflare/cloudflared/diagnostic/network"
)
func TestDecode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
text string
expectedHops []*diagnostic.Hop
}{
{
"repeated hop index parse failure",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
someletters * * *
4 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms `,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(4),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"hop index parse failure",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
someletters 8.8.8.8 8.8.8.9 abc ms 0.456 ms 0.789 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"missing rtt",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 * 8.8.8.8 8.8.8.9 0.456 ms 0.789 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"8.8.8.8 8.8.8.9",
[]time.Duration{
time.Duration(456),
time.Duration(789),
},
),
},
},
{
"simple example ipv4",
`1 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
2 172.68.101.121 (172.68.101.121) 12.874 ms 15.517 ms 15.311 ms
3 * * *`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
},
},
{
"simple example ipv6",
` 1 2400:cb00:107:1024::ac44:6550 12.780 ms 9.118 ms 10.046 ms
2 2a09:bac1:: 9.945 ms 10.033 ms 11.562 ms`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"2400:cb00:107:1024::ac44:6550",
[]time.Duration{
time.Duration(12780),
time.Duration(9118),
time.Duration(10046),
},
),
diagnostic.NewHop(
uint8(2),
"2a09:bac1::",
[]time.Duration{
time.Duration(9945),
time.Duration(10033),
time.Duration(11562),
},
),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
t.Parallel()
hops, err := diagnostic.Decode(strings.NewReader(test.text), diagnostic.DecodeLine)
require.NoError(t, err)
assert.Equal(t, test.expectedHops, hops)
})
}
}
+74
View File
@@ -0,0 +1,74 @@
package diagnostic
import (
"bufio"
"bytes"
"fmt"
"io"
"os/exec"
)
type DecodeLineFunc func(text string) (*Hop, error)
func decodeNetworkOutputToFile(command *exec.Cmd, decodeLine DecodeLineFunc) ([]*Hop, string, error) {
stdout, err := command.StdoutPipe()
if err != nil {
return nil, "", fmt.Errorf("error piping traceroute's output: %w", err)
}
if err := command.Start(); err != nil {
return nil, "", fmt.Errorf("error starting traceroute: %w", err)
}
// Tee the output to a string to have the raw information
// in case the decode call fails
// This error is handled only after the Wait call below returns
// otherwise the process can become a zombie
buf := bytes.NewBuffer([]byte{})
tee := io.TeeReader(stdout, buf)
hops, err := Decode(tee, decodeLine)
// regardless of success of the decoding
// consume all output to have available in buf
_, _ = io.ReadAll(tee)
if werr := command.Wait(); werr != nil {
return nil, "", fmt.Errorf("error finishing traceroute: %w", werr)
}
if err != nil {
return nil, buf.String(), err
}
return hops, buf.String(), nil
}
func Decode(reader io.Reader, decodeLine DecodeLineFunc) ([]*Hop, error) {
scanner := bufio.NewScanner(reader)
scanner.Split(bufio.ScanLines)
var hops []*Hop
for scanner.Scan() {
text := scanner.Text()
if text == "" {
continue
}
hop, err := decodeLine(text)
if err != nil {
// This continue is here on the error case because there are lines at the start and end
// that may not be parsable. (check windows tracert output)
// The skip is here because aside from the start and end lines the other lines should
// always be parsable without errors.
continue
}
hops = append(hops, hop)
}
if scanner.Err() != nil {
return nil, fmt.Errorf("scanner reported an error: %w", scanner.Err())
}
return hops, nil
}
+81
View File
@@ -0,0 +1,81 @@
//go:build windows
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
"time"
)
type NetworkCollectorImpl struct{}
func (tracer *NetworkCollectorImpl) Collect(ctx context.Context, options TraceOptions) ([]*Hop, string, error) {
ipversion := "-4"
if !options.useV4 {
ipversion = "-6"
}
args := []string{
ipversion,
"-w",
strconv.FormatInt(int64(options.timeout.Seconds()), 10),
"-h",
strconv.FormatUint(options.ttl, 10),
// Do not resolve host names (can add 30+ seconds to run time)
"-d",
options.address,
}
command := exec.CommandContext(ctx, "tracert.exe", args...)
return decodeNetworkOutputToFile(command, DecodeLine)
}
func DecodeLine(text string) (*Hop, error) {
const requestTimedOut = "Request timed out."
fields := strings.Fields(text)
parts := []string{}
filter := func(s string) bool { return s != "*" && s != "ms" }
for _, field := range fields {
if filter(field) {
parts = append(parts, field)
}
}
index, err := strconv.ParseUint(parts[0], 10, 8)
if err != nil {
return nil, fmt.Errorf("couldn't parse index from timeout hop: %w", err)
}
domain := ""
rtts := []time.Duration{}
for _, part := range parts[1:] {
rtt, err := strconv.ParseFloat(strings.TrimLeft(part, "<"), 64)
if err != nil {
domain += part + " "
} else {
rtts = append(rtts, time.Duration(rtt*MicrosecondsFactor))
}
}
domain, _ = strings.CutSuffix(domain, " ")
// If the domain is equal to "Request timed out." then we build a
// timeout hop.
if domain == requestTimedOut {
return NewTimeoutHop(uint8(index)), nil
}
if domain == "" {
return nil, ErrEmptyDomain
}
return NewHop(uint8(index), domain, rtts), nil
}
@@ -0,0 +1,210 @@
//go:build windows
package diagnostic_test
import (
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
diagnostic "github.com/cloudflare/cloudflared/diagnostic/network"
)
func TestDecode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
text string
expectedHops []*diagnostic.Hop
}{
{
"tracert output",
`
Tracing route to region2.v2.argotunnel.com [198.41.200.73]
over a maximum of 5 hops:
1 10 ms <1 ms 1 ms 192.168.64.1
2 27 ms 14 ms 5 ms 192.168.1.254
3 * * * Request timed out.
4 * * * Request timed out.
5 27 ms 5 ms 5 ms 195.8.30.245
Trace complete.
`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"192.168.64.1",
[]time.Duration{
time.Duration(10000),
time.Duration(1000),
time.Duration(1000),
},
),
diagnostic.NewHop(
uint8(2),
"192.168.1.254",
[]time.Duration{
time.Duration(27000),
time.Duration(14000),
time.Duration(5000),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
diagnostic.NewTimeoutHop(uint8(4)),
diagnostic.NewHop(
uint8(5),
"195.8.30.245",
[]time.Duration{
time.Duration(27000),
time.Duration(5000),
time.Duration(5000),
},
),
},
},
{
"repeated hop index parse failure",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
someletters * * *`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"hop index parse failure",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
someletters abc ms 0.456 ms 0.789 ms 8.8.8.8 8.8.8.9`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
},
},
{
"missing rtt",
`1 <12.874 ms <15.517 ms <15.311 ms 172.68.101.121 (172.68.101.121)
2 * 0.456 ms 0.789 ms 8.8.8.8 8.8.8.9`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"8.8.8.8 8.8.8.9",
[]time.Duration{
time.Duration(456),
time.Duration(789),
},
),
},
},
{
"simple example ipv4",
`1 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
2 12.874 ms 15.517 ms 15.311 ms 172.68.101.121 (172.68.101.121)
3 * * * Request timed out.`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewHop(
uint8(2),
"172.68.101.121 (172.68.101.121)",
[]time.Duration{
time.Duration(12874),
time.Duration(15517),
time.Duration(15311),
},
),
diagnostic.NewTimeoutHop(uint8(3)),
},
},
{
"simple example ipv6",
` 1 12.780 ms 9.118 ms 10.046 ms 2400:cb00:107:1024::ac44:6550
2 9.945 ms 10.033 ms 11.562 ms 2a09:bac1::`,
[]*diagnostic.Hop{
diagnostic.NewHop(
uint8(1),
"2400:cb00:107:1024::ac44:6550",
[]time.Duration{
time.Duration(12780),
time.Duration(9118),
time.Duration(10046),
},
),
diagnostic.NewHop(
uint8(2),
"2a09:bac1::",
[]time.Duration{
time.Duration(9945),
time.Duration(10033),
time.Duration(11562),
},
),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
t.Parallel()
hops, err := diagnostic.Decode(strings.NewReader(test.text), diagnostic.DecodeLine)
require.NoError(t, err)
assert.Equal(t, test.expectedHops, hops)
})
}
}
+70
View File
@@ -0,0 +1,70 @@
package diagnostic
import "context"
type DiskVolumeInformation struct {
Name string `json:"name"` // represents the filesystem in linux/macos or device name in windows
SizeMaximum uint64 `json:"sizeMaximum"` // represents the maximum size of the disk in kilobytes
SizeCurrent uint64 `json:"sizeCurrent"` // represents the current size of the disk in kilobytes
}
func NewDiskVolumeInformation(name string, maximum, current uint64) *DiskVolumeInformation {
return &DiskVolumeInformation{
name,
maximum,
current,
}
}
type SystemInformation struct {
MemoryMaximum uint64 `json:"memoryMaximum"` // represents the maximum memory of the system in kilobytes
MemoryCurrent uint64 `json:"memoryCurrent"` // represents the system's memory in use in kilobytes
FileDescriptorMaximum uint64 `json:"fileDescriptorMaximum"` // represents the maximum number of file descriptors of the system
FileDescriptorCurrent uint64 `json:"fileDescriptorCurrent"` // represents the system's file descriptors in use
OsSystem string `json:"osSystem"` // represents the operating system name i.e.: linux, windows, darwin
HostName string `json:"hostName"` // represents the system host name
OsVersion string `json:"osVersion"` // detailed information about the system's release version level
OsRelease string `json:"osRelease"` // detailed information about the system's release
Architecture string `json:"architecture"` // represents the system's hardware platform i.e: arm64/amd64
CloudflaredVersion string `json:"cloudflaredVersion"` // the runtime version of cloudflared
Disk []*DiskVolumeInformation `json:"disk"`
}
func NewSystemInformation(
memoryMaximum,
memoryCurrent,
filesMaximum,
filesCurrent uint64,
osystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion string,
disk []*DiskVolumeInformation,
) *SystemInformation {
return &SystemInformation{
memoryMaximum,
memoryCurrent,
filesMaximum,
filesCurrent,
osystem,
name,
osVersion,
osRelease,
architecture,
cloudflaredVersion,
disk,
}
}
type SystemCollector interface {
// If the collection is successful it will return `SystemInformation` struct,
// an empty string, and a nil error.
// In case there is an error a string with the raw data will be returned
// however the returned string not contain all the data points.
//
// This function expects that the caller sets the context timeout to prevent
// long-lived collectors.
Collect(ctx context.Context) (*SystemInformation, string, error)
}
+120
View File
@@ -0,0 +1,120 @@
//go:build linux
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
)
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, string, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
fdInfo, fdInfoRaw, fdInfoErr := collectFileDescriptorInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformationUnix(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformationUnix(ctx)
if memoryInfoErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw)
return nil, raw, memoryInfoErr
}
if fdInfoErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw)
return nil, raw, fdInfoErr
}
if diskErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw)
return nil, raw, diskErr
}
if osInfoErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw)
return nil, raw, osInfoErr
}
return NewSystemInformation(
memoryInfo.MemoryMaximum,
memoryInfo.MemoryCurrent,
fdInfo.FileDescriptorMaximum,
fdInfo.FileDescriptorCurrent,
osInfo.OsSystem,
osInfo.Name,
osInfo.OsVersion,
osInfo.OsRelease,
osInfo.Architecture,
collector.version,
disks,
), "", nil
}
func collectMemoryInformation(ctx context.Context) (*MemoryInformation, string, error) {
// This function relies on the output of `cat /proc/meminfo` to retrieve
// memoryMax and memoryCurrent.
// The expected output is in the format of `KEY VALUE UNIT`.
const (
memTotalPrefix = "MemTotal"
memAvailablePrefix = "MemAvailable"
)
command := exec.CommandContext(ctx, "cat", "/proc/meminfo")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
mapper := func(field string) (uint64, error) {
field = strings.TrimRight(field, " kB")
return strconv.ParseUint(field, 10, 64)
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memTotalPrefix, memAvailablePrefix, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
func collectFileDescriptorInformation(ctx context.Context) (*FileDescriptorInformation, string, error) {
// Command retrieved from https://docs.kernel.org/admin-guide/sysctl/fs.html#file-max-file-nr.
// If the sysctl is not available the command with fail.
command := exec.CommandContext(ctx, "sysctl", "-n", "fs.file-nr")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
fileDescriptorInfo, err := ParseSysctlFileDescriptorInformation(output)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return fileDescriptorInfo, output, nil
}
+132
View File
@@ -0,0 +1,132 @@
//go:build darwin
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
)
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, string, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
fdInfo, fdInfoRaw, fdInfoErr := collectFileDescriptorInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformationUnix(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformationUnix(ctx)
if memoryInfoErr != nil {
return nil, RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw), memoryInfoErr
}
if fdInfoErr != nil {
return nil, RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw), fdInfoErr
}
if diskErr != nil {
return nil, RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw), diskErr
}
if osInfoErr != nil {
return nil, RawSystemInformation(osInfoRaw, memoryInfoRaw, fdInfoRaw, disksRaw), osInfoErr
}
return NewSystemInformation(
memoryInfo.MemoryMaximum,
memoryInfo.MemoryCurrent,
fdInfo.FileDescriptorMaximum,
fdInfo.FileDescriptorCurrent,
osInfo.OsSystem,
osInfo.Name,
osInfo.OsVersion,
osInfo.OsRelease,
osInfo.Architecture,
collector.version,
disks,
), "", nil
}
func collectFileDescriptorInformation(ctx context.Context) (
*FileDescriptorInformation,
string,
error,
) {
const (
fileDescriptorMaximumKey = "kern.maxfiles"
fileDescriptorCurrentKey = "kern.num_files"
)
command := exec.CommandContext(ctx, "sysctl", fileDescriptorMaximumKey, fileDescriptorCurrentKey)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
fileDescriptorInfo, err := ParseFileDescriptorInformationFromKV(
output,
fileDescriptorMaximumKey,
fileDescriptorCurrentKey,
)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return fileDescriptorInfo, output, nil
}
func collectMemoryInformation(ctx context.Context) (
*MemoryInformation,
string,
error,
) {
const (
memoryMaximumKey = "hw.memsize"
memoryAvailableKey = "hw.memsize_usable"
)
command := exec.CommandContext(
ctx,
"sysctl",
memoryMaximumKey,
memoryAvailableKey,
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
mapper := func(field string) (uint64, error) {
const kiloBytes = 1024
value, err := strconv.ParseUint(field, 10, 64)
return value / kiloBytes, err
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memoryMaximumKey, memoryAvailableKey, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
+466
View File
@@ -0,0 +1,466 @@
package diagnostic_test
import (
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/diagnostic"
)
func TestParseMemoryInformationFromKV(t *testing.T) {
t.Parallel()
mapper := func(field string) (uint64, error) {
value, err := strconv.ParseUint(field, 10, 64)
return value, err
}
linuxMapper := func(field string) (uint64, error) {
field = strings.TrimRight(field, " kB")
return strconv.ParseUint(field, 10, 64)
}
windowsMemoryOutput := `
FreeVirtualMemory : 5350472
TotalVirtualMemorySize : 8903424
`
macosMemoryOutput := `hw.memsize: 38654705664
hw.memsize_usable: 38009012224`
memoryOutputWithMissingKey := `hw.memsize: 38654705664`
linuxMemoryOutput := `MemTotal: 8028860 kB
MemFree: 731396 kB
MemAvailable: 4678844 kB
Buffers: 472632 kB
Cached: 3186492 kB
SwapCached: 4196 kB
Active: 3088988 kB
Inactive: 3468560 kB`
tests := []struct {
name string
output string
memoryMaximumKey string
memoryAvailableKey string
expected *diagnostic.MemoryInformation
expectedErr bool
mapper func(string) (uint64, error)
}{
{
name: "parse linux memory values",
output: linuxMemoryOutput,
memoryMaximumKey: "MemTotal",
memoryAvailableKey: "MemAvailable",
expected: &diagnostic.MemoryInformation{
8028860,
8028860 - 4678844,
},
expectedErr: false,
mapper: linuxMapper,
},
{
name: "parse memory values with missing key",
output: memoryOutputWithMissingKey,
memoryMaximumKey: "hw.memsize",
memoryAvailableKey: "hw.memsize_usable",
expected: nil,
expectedErr: true,
mapper: mapper,
},
{
name: "parse macos memory values",
output: macosMemoryOutput,
memoryMaximumKey: "hw.memsize",
memoryAvailableKey: "hw.memsize_usable",
expected: &diagnostic.MemoryInformation{
38654705664,
38654705664 - 38009012224,
},
expectedErr: false,
mapper: mapper,
},
{
name: "parse windows memory values",
output: windowsMemoryOutput,
memoryMaximumKey: "TotalVirtualMemorySize",
memoryAvailableKey: "FreeVirtualMemory",
expected: &diagnostic.MemoryInformation{
8903424,
8903424 - 5350472,
},
expectedErr: false,
mapper: mapper,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
memoryInfo, err := diagnostic.ParseMemoryInformationFromKV(
tCase.output,
tCase.memoryMaximumKey,
tCase.memoryAvailableKey,
tCase.mapper,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, memoryInfo)
}
})
}
}
func TestParseUnameOutput(t *testing.T) {
t.Parallel()
tests := []struct {
name string
output string
os string
expected *diagnostic.OsInfo
expectedErr bool
}{
{
name: "darwin machine",
output: "Darwin APC 23.6.0 Darwin Kernel Version 99.6.0: Wed Jul 31 20:48:04 PDT 1997; root:xnu-66666.666.6.666.6~1/RELEASE_ARM64_T6666 arm64",
os: "darwin",
expected: &diagnostic.OsInfo{
Architecture: "arm64",
Name: "APC",
OsSystem: "Darwin",
OsRelease: "Darwin Kernel Version 99.6.0: Wed Jul 31 20:48:04 PDT 1997; root:xnu-66666.666.6.666.6~1/RELEASE_ARM64_T6666",
OsVersion: "23.6.0",
},
expectedErr: false,
},
{
name: "linux machine",
output: "Linux dab00d565591 6.6.31-linuxkit #1 SMP Thu May 23 08:36:57 UTC 2024 aarch64 GNU/Linux",
os: "linux",
expected: &diagnostic.OsInfo{
Architecture: "aarch64",
Name: "dab00d565591",
OsSystem: "Linux",
OsRelease: "#1 SMP Thu May 23 08:36:57 UTC 2024",
OsVersion: "6.6.31-linuxkit",
},
expectedErr: false,
},
{
name: "not enough fields",
output: "Linux ",
os: "linux",
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
memoryInfo, err := diagnostic.ParseUnameOutput(
tCase.output,
tCase.os,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, memoryInfo)
}
})
}
}
func TestParseFileDescriptorInformationFromKV(t *testing.T) {
const (
fileDescriptorMaximumKey = "kern.maxfiles"
fileDescriptorCurrentKey = "kern.num_files"
)
t.Parallel()
memoryOutput := `kern.maxfiles: 276480
kern.num_files: 11787`
memoryOutputWithMissingKey := `kern.maxfiles: 276480`
tests := []struct {
name string
output string
expected *diagnostic.FileDescriptorInformation
expectedErr bool
}{
{
name: "parse memory values with missing key",
output: memoryOutputWithMissingKey,
expected: nil,
expectedErr: true,
},
{
name: "parse macos memory values",
output: memoryOutput,
expected: &diagnostic.FileDescriptorInformation{
276480,
11787,
},
expectedErr: false,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
fdInfo, err := diagnostic.ParseFileDescriptorInformationFromKV(
tCase.output,
fileDescriptorMaximumKey,
fileDescriptorCurrentKey,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, fdInfo)
}
})
}
}
func TestParseSysctlFileDescriptorInformation(t *testing.T) {
t.Parallel()
tests := []struct {
name string
output string
expected *diagnostic.FileDescriptorInformation
expectedErr bool
}{
{
name: "expected output",
output: "111 0 1111111",
expected: &diagnostic.FileDescriptorInformation{
FileDescriptorMaximum: 1111111,
FileDescriptorCurrent: 111,
},
expectedErr: false,
},
{
name: "not enough fields",
output: "111 111 ",
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
fdsInfo, err := diagnostic.ParseSysctlFileDescriptorInformation(
tCase.output,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, fdsInfo)
}
})
}
}
func TestParseWinOperatingSystemInfo(t *testing.T) {
const (
architecturePrefix = "OSArchitecture"
osSystemPrefix = "Caption"
osVersionPrefix = "Version"
osReleasePrefix = "BuildNumber"
namePrefix = "CSName"
)
t.Parallel()
windowsIncompleteOsInfo := `
OSArchitecture : ARM 64 bits
Caption : Microsoft Windows 11 Home
Morekeys : 121314
CSName : UTILIZA-QO859QP
`
windowsCompleteOsInfo := `
OSArchitecture : ARM 64 bits
Caption : Microsoft Windows 11 Home
Version : 10.0.22631
BuildNumber : 22631
Morekeys : 121314
CSName : UTILIZA-QO859QP
`
tests := []struct {
name string
output string
expected *diagnostic.OsInfo
expectedErr bool
}{
{
name: "expected output",
output: windowsCompleteOsInfo,
expected: &diagnostic.OsInfo{
Architecture: "ARM 64 bits",
Name: "UTILIZA-QO859QP",
OsSystem: "Microsoft Windows 11 Home",
OsRelease: "22631",
OsVersion: "10.0.22631",
},
expectedErr: false,
},
{
name: "missing keys",
output: windowsIncompleteOsInfo,
expected: nil,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
osInfo, err := diagnostic.ParseWinOperatingSystemInfo(
tCase.output,
architecturePrefix,
osSystemPrefix,
osVersionPrefix,
osReleasePrefix,
namePrefix,
)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, osInfo)
}
})
}
}
func TestParseDiskVolumeInformationOutput(t *testing.T) {
t.Parallel()
invalidUnixDiskVolumeInfo := `Filesystem Size Used Avail Use% Mounted on
overlay 59G 19G 38G 33% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/run/host_mark/Users 461G 266G 195G 58% /tmp/cloudflared
/dev/vda1 59G 19G 38G 33% /etc/hosts
tmpfs 3.9G 0 3.9G 0% /sys/firmware
`
unixDiskVolumeInfo := `Filesystem Size Used Avail Use% Mounted on
overlay 61202244 18881444 39179476 33% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
/run/host_mark/Users 482797652 278648468 204149184 58% /tmp/cloudflared
/dev/vda1 61202244 18881444 39179476 33% /etc/hosts
tmpfs 4014428 0 4014428 0% /sys/firmware`
missingFields := ` DeviceID Size
-------- ----
C: size
E: 235563008
Z: 67754782720
`
invalidTypeField := ` DeviceID Size FreeSpace
-------- ---- ---------
C: size 31318736896
D:
E: 235563008 0
Z: 67754782720 31318732800
`
windowsDiskVolumeInfo := `
DeviceID Size FreeSpace
-------- ---- ---------
C: 67754782720 31318736896
E: 235563008 0
Z: 67754782720 31318732800`
tests := []struct {
name string
output string
expected []*diagnostic.DiskVolumeInformation
skipLines int
expectedErr bool
}{
{
name: "invalid unix disk volume information (numbers have units)",
output: invalidUnixDiskVolumeInfo,
expected: []*diagnostic.DiskVolumeInformation{},
skipLines: 1,
expectedErr: true,
},
{
name: "unix disk volume information",
output: unixDiskVolumeInfo,
skipLines: 1,
expected: []*diagnostic.DiskVolumeInformation{
diagnostic.NewDiskVolumeInformation("overlay", 61202244, 18881444),
diagnostic.NewDiskVolumeInformation("tmpfs", 65536, 0),
diagnostic.NewDiskVolumeInformation("shm", 65536, 0),
diagnostic.NewDiskVolumeInformation("/run/host_mark/Users", 482797652, 278648468),
diagnostic.NewDiskVolumeInformation("/dev/vda1", 61202244, 18881444),
diagnostic.NewDiskVolumeInformation("tmpfs", 4014428, 0),
},
expectedErr: false,
},
{
name: "windows disk volume information",
output: windowsDiskVolumeInfo,
expected: []*diagnostic.DiskVolumeInformation{
diagnostic.NewDiskVolumeInformation("C:", 67754782720, 31318736896),
diagnostic.NewDiskVolumeInformation("E:", 235563008, 0),
diagnostic.NewDiskVolumeInformation("Z:", 67754782720, 31318732800),
},
skipLines: 4,
expectedErr: false,
},
{
name: "insuficient fields",
output: missingFields,
expected: nil,
skipLines: 2,
expectedErr: true,
},
{
name: "invalid field",
output: invalidTypeField,
expected: nil,
skipLines: 2,
expectedErr: true,
},
}
for _, tCase := range tests {
t.Run(tCase.name, func(t *testing.T) {
t.Parallel()
disks, err := diagnostic.ParseDiskVolumeInformationOutput(tCase.output, tCase.skipLines, 1)
if tCase.expectedErr {
assert.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, tCase.expected, disks)
}
})
}
}
+377
View File
@@ -0,0 +1,377 @@
package diagnostic
import (
"context"
"fmt"
"os/exec"
"runtime"
"sort"
"strconv"
"strings"
)
func findColonSeparatedPairs[V any](output string, keys []string, mapper func(string) (V, error)) map[string]V {
const (
memoryField = 1
memoryInformationFields = 2
)
lines := strings.Split(output, "\n")
pairs := make(map[string]V, 0)
// sort keys and lines to allow incremental search
sort.Strings(lines)
sort.Strings(keys)
// keeps track of the last key found
lastIndex := 0
for _, line := range lines {
if lastIndex == len(keys) {
// already found all keys no need to continue iterating
// over the other values
break
}
for index, key := range keys[lastIndex:] {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, key) {
fields := strings.Split(line, ":")
if len(fields) < memoryInformationFields {
lastIndex = index + 1
break
}
field, err := mapper(strings.TrimSpace(fields[memoryField]))
if err != nil {
lastIndex = lastIndex + index + 1
break
}
pairs[key] = field
lastIndex = lastIndex + index + 1
break
}
}
}
return pairs
}
func ParseDiskVolumeInformationOutput(output string, skipLines int, scale float64) ([]*DiskVolumeInformation, error) {
const (
diskFieldsMinimum = 3
nameField = 0
sizeMaximumField = 1
sizeCurrentField = 2
)
disksRaw := strings.Split(output, "\n")
disks := make([]*DiskVolumeInformation, 0)
if skipLines > len(disksRaw) || skipLines < 0 {
skipLines = 0
}
for _, disk := range disksRaw[skipLines:] {
if disk == "" {
// skip empty line
continue
}
fields := strings.Fields(disk)
if len(fields) < diskFieldsMinimum {
return nil, fmt.Errorf("expected disk volume to have %d fields got %d: %w",
diskFieldsMinimum, len(fields), ErrInsuficientFields,
)
}
name := fields[nameField]
sizeMaximum, err := strconv.ParseUint(fields[sizeMaximumField], 10, 64)
if err != nil {
continue
}
sizeCurrent, err := strconv.ParseUint(fields[sizeCurrentField], 10, 64)
if err != nil {
continue
}
diskInfo := NewDiskVolumeInformation(
name, uint64(float64(sizeMaximum)*scale), uint64(float64(sizeCurrent)*scale),
)
disks = append(disks, diskInfo)
}
if len(disks) == 0 {
return nil, ErrNoVolumeFound
}
return disks, nil
}
type OsInfo struct {
OsSystem string
Name string
OsVersion string
OsRelease string
Architecture string
}
func ParseUnameOutput(output string, system string) (*OsInfo, error) {
const (
osystemField = 0
nameField = 1
osVersionField = 2
osReleaseStartField = 3
osInformationFieldsMinimum = 6
darwin = "darwin"
)
architectureOffset := 2
if system == darwin {
architectureOffset = 1
}
fields := strings.Fields(output)
if len(fields) < osInformationFieldsMinimum {
return nil, fmt.Errorf("expected system information to have %d fields got %d: %w",
osInformationFieldsMinimum, len(fields), ErrInsuficientFields,
)
}
architectureField := len(fields) - architectureOffset
osystem := fields[osystemField]
name := fields[nameField]
osVersion := fields[osVersionField]
osRelease := strings.Join(fields[osReleaseStartField:architectureField], " ")
architecture := fields[architectureField]
return &OsInfo{
osystem,
name,
osVersion,
osRelease,
architecture,
}, nil
}
func ParseWinOperatingSystemInfo(
output string,
architectureKey string,
osSystemKey string,
osVersionKey string,
osReleaseKey string,
nameKey string,
) (*OsInfo, error) {
identity := func(s string) (string, error) { return s, nil }
keys := []string{architectureKey, osSystemKey, osVersionKey, osReleaseKey, nameKey}
pairs := findColonSeparatedPairs(
output,
keys,
identity,
)
architecture, exists := pairs[architectureKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, architectureKey)
}
osSystem, exists := pairs[osSystemKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osSystemKey)
}
osVersion, exists := pairs[osVersionKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osVersionKey)
}
osRelease, exists := pairs[osReleaseKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, osReleaseKey)
}
name, exists := pairs[nameKey]
if !exists {
return nil, fmt.Errorf("parsing os information: %w, key=%s", ErrKeyNotFound, nameKey)
}
return &OsInfo{osSystem, name, osVersion, osRelease, architecture}, nil
}
type FileDescriptorInformation struct {
FileDescriptorMaximum uint64
FileDescriptorCurrent uint64
}
func ParseSysctlFileDescriptorInformation(output string) (*FileDescriptorInformation, error) {
const (
openFilesField = 0
maxFilesField = 2
fileDescriptorLimitsFields = 3
)
fields := strings.Fields(output)
if len(fields) != fileDescriptorLimitsFields {
return nil,
fmt.Errorf(
"expected file descriptor information to have %d fields got %d: %w",
fileDescriptorLimitsFields,
len(fields),
ErrInsuficientFields,
)
}
fileDescriptorCurrent, err := strconv.ParseUint(fields[openFilesField], 10, 64)
if err != nil {
return nil, fmt.Errorf(
"error parsing files current field '%s': %w",
fields[openFilesField],
err,
)
}
fileDescriptorMaximum, err := strconv.ParseUint(fields[maxFilesField], 10, 64)
if err != nil {
return nil, fmt.Errorf("error parsing files max field '%s': %w", fields[maxFilesField], err)
}
return &FileDescriptorInformation{fileDescriptorMaximum, fileDescriptorCurrent}, nil
}
func ParseFileDescriptorInformationFromKV(
output string,
fileDescriptorMaximumKey string,
fileDescriptorCurrentKey string,
) (*FileDescriptorInformation, error) {
mapper := func(field string) (uint64, error) {
return strconv.ParseUint(field, 10, 64)
}
pairs := findColonSeparatedPairs(output, []string{fileDescriptorMaximumKey, fileDescriptorCurrentKey}, mapper)
fileDescriptorMaximum, exists := pairs[fileDescriptorMaximumKey]
if !exists {
return nil, fmt.Errorf(
"parsing file descriptor information: %w, key=%s",
ErrKeyNotFound,
fileDescriptorMaximumKey,
)
}
fileDescriptorCurrent, exists := pairs[fileDescriptorCurrentKey]
if !exists {
return nil, fmt.Errorf(
"parsing file descriptor information: %w, key=%s",
ErrKeyNotFound,
fileDescriptorCurrentKey,
)
}
return &FileDescriptorInformation{fileDescriptorMaximum, fileDescriptorCurrent}, nil
}
type MemoryInformation struct {
MemoryMaximum uint64 // size in KB
MemoryCurrent uint64 // size in KB
}
func ParseMemoryInformationFromKV(
output string,
memoryMaximumKey string,
memoryAvailableKey string,
mapper func(field string) (uint64, error),
) (*MemoryInformation, error) {
pairs := findColonSeparatedPairs(output, []string{memoryMaximumKey, memoryAvailableKey}, mapper)
memoryMaximum, exists := pairs[memoryMaximumKey]
if !exists {
return nil, fmt.Errorf("parsing memory information: %w, key=%s", ErrKeyNotFound, memoryMaximumKey)
}
memoryAvailable, exists := pairs[memoryAvailableKey]
if !exists {
return nil, fmt.Errorf("parsing memory information: %w, key=%s", ErrKeyNotFound, memoryAvailableKey)
}
memoryCurrent := memoryMaximum - memoryAvailable
return &MemoryInformation{memoryMaximum, memoryCurrent}, nil
}
func RawSystemInformation(osInfoRaw string, memoryInfoRaw string, fdInfoRaw string, disksRaw string) string {
var builder strings.Builder
formatInfo := func(info string, builder *strings.Builder) {
if info == "" {
builder.WriteString("No information\n")
} else {
builder.WriteString(info)
builder.WriteString("\n")
}
}
builder.WriteString("---BEGIN Operating system information\n")
formatInfo(osInfoRaw, &builder)
builder.WriteString("---END Operating system information\n")
builder.WriteString("---BEGIN Memory information\n")
formatInfo(memoryInfoRaw, &builder)
builder.WriteString("---END Memory information\n")
builder.WriteString("---BEGIN File descriptors information\n")
formatInfo(fdInfoRaw, &builder)
builder.WriteString("---END File descriptors information\n")
builder.WriteString("---BEGIN Disks information\n")
formatInfo(disksRaw, &builder)
builder.WriteString("---END Disks information\n")
rawInformation := builder.String()
return rawInformation
}
func collectDiskVolumeInformationUnix(ctx context.Context) ([]*DiskVolumeInformation, string, error) {
command := exec.CommandContext(ctx, "df", "-k")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
disks, err := ParseDiskVolumeInformationOutput(output, 1, 1)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return disks, output, nil
}
func collectOSInformationUnix(ctx context.Context) (*OsInfo, string, error) {
command := exec.CommandContext(ctx, "uname", "-a")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
osInfo, err := ParseUnameOutput(output, runtime.GOOS)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return osInfo, output, nil
}
+153
View File
@@ -0,0 +1,153 @@
//go:build windows
package diagnostic
import (
"context"
"fmt"
"os/exec"
"strconv"
)
const kiloBytesScale = 1.0 / 1024
type SystemCollectorImpl struct {
version string
}
func NewSystemCollectorImpl(
version string,
) *SystemCollectorImpl {
return &SystemCollectorImpl{
version,
}
}
func (collector *SystemCollectorImpl) Collect(ctx context.Context) (*SystemInformation, string, error) {
memoryInfo, memoryInfoRaw, memoryInfoErr := collectMemoryInformation(ctx)
disks, disksRaw, diskErr := collectDiskVolumeInformation(ctx)
osInfo, osInfoRaw, osInfoErr := collectOSInformation(ctx)
if memoryInfoErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, "", disksRaw)
return nil, raw, memoryInfoErr
}
if diskErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, "", disksRaw)
return nil, raw, diskErr
}
if osInfoErr != nil {
raw := RawSystemInformation(osInfoRaw, memoryInfoRaw, "", disksRaw)
return nil, raw, osInfoErr
}
return NewSystemInformation(
memoryInfo.MemoryMaximum,
memoryInfo.MemoryCurrent,
// For windows we leave both the fileDescriptorMaximum and fileDescriptorCurrent with zero
// since there is no obvious way to get this information.
0,
0,
osInfo.OsSystem,
osInfo.Name,
osInfo.OsVersion,
osInfo.OsRelease,
osInfo.Architecture,
collector.version,
disks,
), "", nil
}
func collectMemoryInformation(ctx context.Context) (*MemoryInformation, string, error) {
const (
memoryTotalPrefix = "TotalVirtualMemorySize"
memoryAvailablePrefix = "FreeVirtualMemory"
)
command := exec.CommandContext(
ctx,
"powershell",
"-Command",
"Get-CimInstance -Class Win32_OperatingSystem | Select-Object FreeVirtualMemory, TotalVirtualMemorySize | Format-List",
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
// the result of the command above will return values in bytes hence
// they need to be converted to kilobytes
mapper := func(field string) (uint64, error) {
value, err := strconv.ParseUint(field, 10, 64)
return uint64(float64(value) * kiloBytesScale), err
}
memoryInfo, err := ParseMemoryInformationFromKV(output, memoryTotalPrefix, memoryAvailablePrefix, mapper)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return memoryInfo, output, nil
}
func collectDiskVolumeInformation(ctx context.Context) ([]*DiskVolumeInformation, string, error) {
command := exec.CommandContext(
ctx,
"powershell", "-Command", "Get-CimInstance -Class Win32_LogicalDisk | Select-Object DeviceID, Size, FreeSpace")
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
disks, err := ParseDiskVolumeInformationOutput(output, 2, kiloBytesScale)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return disks, output, nil
}
func collectOSInformation(ctx context.Context) (*OsInfo, string, error) {
const (
architecturePrefix = "OSArchitecture"
osSystemPrefix = "Caption"
osVersionPrefix = "Version"
osReleasePrefix = "BuildNumber"
namePrefix = "CSName"
)
command := exec.CommandContext(
ctx,
"powershell",
"-Command",
"Get-CimInstance -Class Win32_OperatingSystem | Select-Object OSArchitecture, Caption, Version, BuildNumber, CSName | Format-List",
)
stdout, err := command.Output()
if err != nil {
return nil, "", fmt.Errorf("error retrieving output from command '%s': %w", command.String(), err)
}
output := string(stdout)
osInfo, err := ParseWinOperatingSystemInfo(output, architecturePrefix, osSystemPrefix, osVersionPrefix, osReleasePrefix, namePrefix)
if err != nil {
return nil, output, err
}
// returning raw output in case other collected information
// resulted in errors
return osInfo, output, nil
}
+11 -14
View File
@@ -28,10 +28,8 @@ type icmpProxy struct {
srcFunnelTracker *packet.FunnelTracker
echoIDTracker *echoIDTracker
conn *icmp.PacketConn
// Response is handled in one-by-one, so encoder can be shared between funnels
encoder *packet.Encoder
logger *zerolog.Logger
idleTimeout time.Duration
logger *zerolog.Logger
idleTimeout time.Duration
}
// echoIDTracker tracks which ID has been assigned. It first loops through assignment from lastAssignment to then end,
@@ -114,8 +112,8 @@ func (snf echoFunnelID) String() string {
return strconv.FormatUint(uint64(snf), 10)
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
conn, err := newICMPConn(listenIP, zone)
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
conn, err := newICMPConn(listenIP)
if err != nil {
return nil, err
}
@@ -123,16 +121,15 @@ func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idle
return &icmpProxy{
srcFunnelTracker: packet.NewFunnelTracker(),
echoIDTracker: newEchoIDTracker(),
encoder: packet.NewEncoder(),
conn: conn,
logger: logger,
idleTimeout: idleTimeout,
}, nil
}
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
_, span := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
_, span := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -154,7 +151,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
}
span.SetAttributes(attribute.Int("assignedEchoID", int(assignedEchoID)))
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder.datagramMuxer, pk, originalEcho.ID)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder, pk, originalEcho.ID)
newFunnelFunc := func() (packet.Funnel, error) {
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -164,7 +161,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
ip.echoIDTracker.release(echoIDTrackerKey, assignedEchoID)
return nil
}
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, ip.conn, responder, int(assignedEchoID), originalEcho.ID, ip.encoder)
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, ip.conn, responder, int(assignedEchoID), originalEcho.ID)
return icmpFlow, nil
}
funnelID := echoFunnelID(assignedEchoID)
@@ -265,8 +262,8 @@ func (ip *icmpProxy) sendReply(ctx context.Context, reply *echoReply) error {
return err
}
_, span := icmpFlow.responder.replySpan(ctx, ip.logger)
defer icmpFlow.responder.exportSpan()
_, span := icmpFlow.responder.ReplySpan(ctx, ip.logger)
defer icmpFlow.responder.ExportSpan()
if err := icmpFlow.returnToSrc(reply); err != nil {
tracing.EndWithErrorStatus(span, err)
+2 -2
View File
@@ -18,7 +18,7 @@ var errICMPProxyNotImplemented = fmt.Errorf("ICMP proxy is not implemented on %s
type icmpProxy struct{}
func (ip icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ip icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
return errICMPProxyNotImplemented
}
@@ -26,6 +26,6 @@ func (ip *icmpProxy) Serve(ctx context.Context) error {
return errICMPProxyNotImplemented
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
return nil, errICMPProxyNotImplemented
}
+12 -14
View File
@@ -37,25 +37,23 @@ var (
type icmpProxy struct {
srcFunnelTracker *packet.FunnelTracker
listenIP netip.Addr
ipv6Zone string
logger *zerolog.Logger
idleTimeout time.Duration
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
if err := testPermission(listenIP, zone, logger); err != nil {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
if err := testPermission(listenIP, logger); err != nil {
return nil, err
}
return &icmpProxy{
srcFunnelTracker: packet.NewFunnelTracker(),
listenIP: listenIP,
ipv6Zone: zone,
logger: logger,
idleTimeout: idleTimeout,
}, nil
}
func testPermission(listenIP netip.Addr, zone string, logger *zerolog.Logger) error {
func testPermission(listenIP netip.Addr, logger *zerolog.Logger) error {
// Opens a non-privileged ICMP socket. On Linux the group ID of the process needs to be in ping_group_range
// Only check ping_group_range once for IPv4
if listenIP.Is4() {
@@ -64,7 +62,7 @@ func testPermission(listenIP netip.Addr, zone string, logger *zerolog.Logger) er
return err
}
}
conn, err := newICMPConn(listenIP, zone)
conn, err := newICMPConn(listenIP)
if err != nil {
return err
}
@@ -98,9 +96,9 @@ func checkInPingGroup() error {
return fmt.Errorf("did not find group range in %s", pingGroupPath)
}
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
ctx, span := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
ctx, span := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
originalEcho, err := getICMPEcho(pk.Message)
if err != nil {
@@ -109,9 +107,9 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
}
observeICMPRequest(ip.logger, span, pk.Src.String(), pk.Dst.String(), originalEcho.ID, originalEcho.Seq)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder.datagramMuxer, pk, originalEcho.ID)
shouldReplaceFunnelFunc := createShouldReplaceFunnelFunc(ip.logger, responder, pk, originalEcho.ID)
newFunnelFunc := func() (packet.Funnel, error) {
conn, err := newICMPConn(ip.listenIP, ip.ipv6Zone)
conn, err := newICMPConn(ip.listenIP)
if err != nil {
tracing.EndWithErrorStatus(span, err)
return nil, errors.Wrap(err, "failed to open ICMP socket")
@@ -127,7 +125,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
span.SetAttributes(attribute.Int("port", localUDPAddr.Port))
echoID := localUDPAddr.Port
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, conn, responder, echoID, originalEcho.ID, packet.NewEncoder())
icmpFlow := newICMPEchoFlow(pk.Src, closeCallback, conn, responder, echoID, originalEcho.ID)
return icmpFlow, nil
}
funnelID := flow3Tuple{
@@ -181,8 +179,8 @@ func (ip *icmpProxy) listenResponse(ctx context.Context, flow *icmpEchoFlow) {
// Listens for ICMP response and handles error logging
func (ip *icmpProxy) handleResponse(ctx context.Context, flow *icmpEchoFlow, buf []byte) (done bool) {
_, span := flow.responder.replySpan(ctx, ip.logger)
defer flow.responder.exportSpan()
_, span := flow.responder.ReplySpan(ctx, ip.logger)
defer flow.responder.ExportSpan()
span.SetAttributes(
attribute.Int("originalEchoID", flow.originalEchoID),
+9 -19
View File
@@ -18,15 +18,11 @@ import (
)
// Opens a non-privileged ICMP socket on Linux and Darwin
func newICMPConn(listenIP netip.Addr, zone string) (*icmp.PacketConn, error) {
func newICMPConn(listenIP netip.Addr) (*icmp.PacketConn, error) {
if listenIP.Is4() {
return icmp.ListenPacket("udp4", listenIP.String())
}
listenAddr := listenIP.String()
if zone != "" {
listenAddr = listenAddr + "%" + zone
}
return icmp.ListenPacket("udp6", listenAddr)
return icmp.ListenPacket("udp6", listenIP.String())
}
func netipAddr(addr net.Addr) (netip.Addr, bool) {
@@ -34,7 +30,8 @@ func netipAddr(addr net.Addr) (netip.Addr, bool) {
if !ok {
return netip.Addr{}, false
}
return netip.AddrFromSlice(udpAddr.IP)
return udpAddr.AddrPort().Addr(), true
}
type flow3Tuple struct {
@@ -50,14 +47,12 @@ type icmpEchoFlow struct {
closed *atomic.Bool
src netip.Addr
originConn *icmp.PacketConn
responder *packetResponder
responder ICMPResponder
assignedEchoID int
originalEchoID int
// it's up to the user to ensure respEncoder is not used concurrently
respEncoder *packet.Encoder
}
func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icmp.PacketConn, responder *packetResponder, assignedEchoID, originalEchoID int, respEncoder *packet.Encoder) *icmpEchoFlow {
func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icmp.PacketConn, responder ICMPResponder, assignedEchoID, originalEchoID int) *icmpEchoFlow {
return &icmpEchoFlow{
ActivityTracker: packet.NewActivityTracker(),
closeCallback: closeCallback,
@@ -67,7 +62,6 @@ func newICMPEchoFlow(src netip.Addr, closeCallback func() error, originConn *icm
responder: responder,
assignedEchoID: assignedEchoID,
originalEchoID: originalEchoID,
respEncoder: respEncoder,
}
}
@@ -139,11 +133,7 @@ func (ief *icmpEchoFlow) returnToSrc(reply *echoReply) error {
},
Message: reply.msg,
}
serializedPacket, err := ief.respEncoder.Encode(&pk)
if err != nil {
return err
}
return ief.responder.returnPacket(serializedPacket)
return ief.responder.ReturnPacket(&pk)
}
type echoReply struct {
@@ -184,7 +174,7 @@ func toICMPEchoFlow(funnel packet.Funnel) (*icmpEchoFlow, error) {
return icmpFlow, nil
}
func createShouldReplaceFunnelFunc(logger *zerolog.Logger, muxer muxer, pk *packet.ICMP, originalEchoID int) func(packet.Funnel) bool {
func createShouldReplaceFunnelFunc(logger *zerolog.Logger, responder ICMPResponder, pk *packet.ICMP, originalEchoID int) func(packet.Funnel) bool {
return func(existing packet.Funnel) bool {
existingFlow, err := toICMPEchoFlow(existing)
if err != nil {
@@ -199,7 +189,7 @@ func createShouldReplaceFunnelFunc(logger *zerolog.Logger, muxer muxer, pk *pack
// If the existing flow has a different muxer, there's a new quic connection where return packets should be
// routed. Otherwise, return packets will be send to the first observed incoming connection, rather than the
// most recently observed connection.
if existingFlow.responder.datagramMuxer != muxer {
if existingFlow.responder.ConnectionIndex() != responder.ConnectionIndex() {
logger.Debug().
Str("src", pk.Src.String()).
Str("dst", pk.Dst.String()).
+11 -20
View File
@@ -27,7 +27,7 @@ func TestFunnelIdleTimeout(t *testing.T) {
startSeq = 8129
)
logger := zerolog.New(os.Stderr)
proxy, err := newICMPProxy(localhostIP, "", &logger, idleTimeout)
proxy, err := newICMPProxy(localhostIP, &logger, idleTimeout)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -56,24 +56,19 @@ func TestFunnelIdleTimeout(t *testing.T) {
},
}
muxer := newMockMuxer(0)
responder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &responder))
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
// Send second request, should reuse the funnel
require.NoError(t, proxy.Request(ctx, &pk, &packetResponder{
datagramMuxer: muxer,
}))
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
// New muxer on a different connection should use a new flow
time.Sleep(idleTimeout * 2)
newMuxer := newMockMuxer(0)
newResponder := packetResponder{
datagramMuxer: newMuxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &newResponder))
newResponder := newPacketResponder(newMuxer, 1, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, newResponder))
validateEchoFlow(t, <-newMuxer.cfdToEdge, &pk)
time.Sleep(idleTimeout * 2)
@@ -90,7 +85,7 @@ func TestReuseFunnel(t *testing.T) {
startSeq = 8129
)
logger := zerolog.New(os.Stderr)
proxy, err := newICMPProxy(localhostIP, "", &logger, idleTimeout)
proxy, err := newICMPProxy(localhostIP, &logger, idleTimeout)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -124,18 +119,14 @@ func TestReuseFunnel(t *testing.T) {
originalEchoID: echoID,
}
muxer := newMockMuxer(0)
responder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, proxy.Request(ctx, &pk, &responder))
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
funnel1, found := getFunnel(t, proxy, tuple)
require.True(t, found)
// Send second request, should reuse the funnel
require.NoError(t, proxy.Request(ctx, &pk, &packetResponder{
datagramMuxer: muxer,
}))
require.NoError(t, proxy.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
funnel2, found := getFunnel(t, proxy, tuple)
require.True(t, found)
+8 -30
View File
@@ -13,7 +13,6 @@ import (
"fmt"
"net/netip"
"runtime/debug"
"sync"
"syscall"
"time"
"unsafe"
@@ -222,11 +221,9 @@ type icmpProxy struct {
// This is a ICMPv6 if srcSocketAddr is not nil
srcSocketAddr *sockAddrIn6
logger *zerolog.Logger
// A pool of reusable *packet.Encoder
encoderPool sync.Pool
}
func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
func newICMPProxy(listenIP netip.Addr, logger *zerolog.Logger, idleTimeout time.Duration) (*icmpProxy, error) {
var (
srcSocketAddr *sockAddrIn6
handle uintptr
@@ -250,11 +247,6 @@ func newICMPProxy(listenIP netip.Addr, zone string, logger *zerolog.Logger, idle
handle: handle,
srcSocketAddr: srcSocketAddr,
logger: logger,
encoderPool: sync.Pool{
New: func() any {
return packet.NewEncoder()
},
},
}, nil
}
@@ -267,15 +259,15 @@ func (ip *icmpProxy) Serve(ctx context.Context) error {
// Request sends an ICMP echo request and wait for a reply or timeout.
// The async version of Win32 APIs take a callback whose memory is not garbage collected, so we use the synchronous version.
// It's possible that a slow request will block other requests, so we set the timeout to only 1s.
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
defer func() {
if r := recover(); r != nil {
ip.logger.Error().Interface("error", r).Msgf("Recover panic from sending icmp request/response, error %s", debug.Stack())
}
}()
_, requestSpan := responder.requestSpan(ctx, pk)
defer responder.exportSpan()
_, requestSpan := responder.RequestSpan(ctx, pk)
defer responder.ExportSpan()
echo, err := getICMPEcho(pk.Message)
if err != nil {
@@ -290,9 +282,9 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
return err
}
tracing.End(requestSpan)
responder.exportSpan()
responder.ExportSpan()
_, replySpan := responder.replySpan(ctx, ip.logger)
_, replySpan := responder.ReplySpan(ctx, ip.logger)
err = ip.handleEchoReply(pk, echo, resp, responder)
if err != nil {
ip.logger.Err(err).Msg("Failed to send ICMP reply")
@@ -308,7 +300,7 @@ func (ip *icmpProxy) Request(ctx context.Context, pk *packet.ICMP, responder *pa
return nil
}
func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, resp echoResp, responder *packetResponder) error {
func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, resp echoResp, responder ICMPResponder) error {
var replyType icmp.Type
if request.Dst.Is4() {
replyType = ipv4.ICMPTypeEchoReply
@@ -333,21 +325,7 @@ func (ip *icmpProxy) handleEchoReply(request *packet.ICMP, echoReq *icmp.Echo, r
},
},
}
cachedEncoder := ip.encoderPool.Get()
// The encoded packet is a slice to of the encoder, so we shouldn't return the encoder back to the pool until
// the encoded packet is sent.
defer ip.encoderPool.Put(cachedEncoder)
encoder, ok := cachedEncoder.(*packet.Encoder)
if !ok {
return fmt.Errorf("encoderPool returned %T, expect *packet.Encoder", cachedEncoder)
}
serializedPacket, err := encoder.Encode(&pk)
if err != nil {
return err
}
return responder.returnPacket(serializedPacket)
return responder.ReturnPacket(&pk)
}
func (ip *icmpProxy) icmpEchoRoundtrip(dst netip.Addr, echo *icmp.Echo) (echoResp, error) {
+1 -1
View File
@@ -132,7 +132,7 @@ func TestSendEchoErrors(t *testing.T) {
}
func testSendEchoErrors(t *testing.T, listenIP netip.Addr) {
proxy, err := newICMPProxy(listenIP, "", &noopLogger, time.Second)
proxy, err := newICMPProxy(listenIP, &noopLogger, time.Second)
require.NoError(t, err)
echo := icmp.Echo{
+46 -4
View File
@@ -14,6 +14,7 @@ import (
"golang.org/x/net/ipv6"
"github.com/cloudflare/cloudflared/packet"
"github.com/cloudflare/cloudflared/tracing"
)
const (
@@ -26,17 +27,46 @@ var (
errPacketNil = fmt.Errorf("packet is nil")
)
// ICMPRouterServer is a parent interface over-top of ICMPRouter that allows for the operation of the proxy origin listeners.
type ICMPRouterServer interface {
ICMPRouter
// Serve runs the ICMPRouter proxy origin listeners for any of the IPv4 or IPv6 interfaces configured.
Serve(ctx context.Context) error
}
// ICMPRouter manages out-going ICMP requests towards the origin.
type ICMPRouter interface {
// Request will send an ICMP packet towards the origin with an ICMPResponder to attach to the ICMP flow for the
// response to utilize.
Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error
// ConvertToTTLExceeded will take an ICMP packet and create a ICMP TTL Exceeded response origininating from the
// ICMPRouter's IP interface.
ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP
}
// ICMPResponder manages how to handle incoming ICMP messages coming from the origin to the edge.
type ICMPResponder interface {
ConnectionIndex() uint8
ReturnPacket(pk *packet.ICMP) error
AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte)
RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span)
ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span)
ExportSpan()
}
type icmpRouter struct {
ipv4Proxy *icmpProxy
ipv4Src netip.Addr
ipv6Proxy *icmpProxy
ipv6Src netip.Addr
}
// NewICMPRouter doesn't return an error if either ipv4 proxy or ipv6 proxy can be created. The machine might only
// support one of them.
// funnelIdleTimeout controls how long to wait to close a funnel without send/return
func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, ipv6Zone string, logger *zerolog.Logger, funnelIdleTimeout time.Duration) (*icmpRouter, error) {
ipv4Proxy, ipv4Err := newICMPProxy(ipv4Addr, "", logger, funnelIdleTimeout)
ipv6Proxy, ipv6Err := newICMPProxy(ipv6Addr, ipv6Zone, logger, funnelIdleTimeout)
func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, logger *zerolog.Logger, funnelIdleTimeout time.Duration) (ICMPRouterServer, error) {
ipv4Proxy, ipv4Err := newICMPProxy(ipv4Addr, logger, funnelIdleTimeout)
ipv6Proxy, ipv6Err := newICMPProxy(ipv6Addr, logger, funnelIdleTimeout)
if ipv4Err != nil && ipv6Err != nil {
err := fmt.Errorf("cannot create ICMPv4 proxy: %v nor ICMPv6 proxy: %v", ipv4Err, ipv6Err)
logger.Debug().Err(err).Msg("ICMP proxy feature is disabled")
@@ -52,7 +82,9 @@ func NewICMPRouter(ipv4Addr, ipv6Addr netip.Addr, ipv6Zone string, logger *zerol
}
return &icmpRouter{
ipv4Proxy: ipv4Proxy,
ipv4Src: ipv4Addr,
ipv6Proxy: ipv6Proxy,
ipv6Src: ipv6Addr,
}, nil
}
@@ -76,7 +108,7 @@ func (ir *icmpRouter) Serve(ctx context.Context) error {
return fmt.Errorf("ICMPv4 proxy and ICMPv6 proxy are both nil")
}
func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder *packetResponder) error {
func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder ICMPResponder) error {
if pk == nil {
return errPacketNil
}
@@ -92,6 +124,16 @@ func (ir *icmpRouter) Request(ctx context.Context, pk *packet.ICMP, responder *p
return fmt.Errorf("ICMPv6 proxy was not instantiated")
}
func (ir *icmpRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
var srcIP netip.Addr
if pk.Dst.Is4() {
srcIP = ir.ipv4Src
} else {
srcIP = ir.ipv6Src
}
return packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, srcIP)
}
func getICMPEcho(msg *icmp.Message) (*icmp.Echo, error) {
echo, ok := msg.Body.(*icmp.Echo)
if !ok {
+17 -30
View File
@@ -50,7 +50,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
endSeq = 20
)
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -61,9 +61,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
}()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
protocol := layers.IPProtocolICMPv6
if sendIPv4 {
@@ -98,7 +96,7 @@ func testICMPRouterEcho(t *testing.T, sendIPv4 bool) {
},
},
}
require.NoError(t, router.Request(ctx, &pk, &responder))
require.NoError(t, router.Request(ctx, &pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
}
}
@@ -114,7 +112,7 @@ func TestTraceICMPRouterEcho(t *testing.T) {
tracingCtx := "ec31ad8a01fde11fdcabe2efdce36873:52726f6cabc144f5:0:1"
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -131,11 +129,8 @@ func TestTraceICMPRouterEcho(t *testing.T) {
serializedIdentity, err := tracingIdentity.MarshalBinary()
require.NoError(t, err)
responder := packetResponder{
datagramMuxer: muxer,
tracedCtx: tracing.NewTracedContext(ctx, tracingIdentity.String(), &noopLogger),
serializedIdentity: serializedIdentity,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
responder.AddTraceContext(tracing.NewTracedContext(ctx, tracingIdentity.String(), &noopLogger), serializedIdentity)
echo := &icmp.Echo{
ID: 12910,
@@ -156,7 +151,7 @@ func TestTraceICMPRouterEcho(t *testing.T) {
},
}
require.NoError(t, router.Request(ctx, &pk, &responder))
require.NoError(t, router.Request(ctx, &pk, responder))
firstPK := <-muxer.cfdToEdge
var requestSpan *quicpogs.TracingSpanPacket
// The order of receiving reply or request span is not deterministic
@@ -194,10 +189,8 @@ func TestTraceICMPRouterEcho(t *testing.T) {
echo.Seq++
pk.Body = echo
// Only first request for a flow is traced. The edge will not send tracing context for the second request
newResponder := packetResponder{
datagramMuxer: muxer,
}
require.NoError(t, router.Request(ctx, &pk, &newResponder))
newResponder := newPacketResponder(muxer, 0, packet.NewEncoder())
require.NoError(t, router.Request(ctx, &pk, newResponder))
validateEchoFlow(t, <-muxer.cfdToEdge, &pk)
select {
@@ -221,7 +214,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
endSeq = 5
)
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
proxyDone := make(chan struct{})
@@ -240,9 +233,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
defer wg.Done()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
for seq := 0; seq < endSeq; seq++ {
pk := &packet.ICMP{
IP: &packet.IP{
@@ -261,16 +252,14 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
},
},
}
require.NoError(t, router.Request(ctx, pk, &responder))
require.NoError(t, router.Request(ctx, pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, pk)
}
}()
go func() {
defer wg.Done()
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
for seq := 0; seq < endSeq; seq++ {
pk := &packet.ICMP{
IP: &packet.IP{
@@ -289,7 +278,7 @@ func TestConcurrentRequestsToSameDst(t *testing.T) {
},
},
}
require.NoError(t, router.Request(ctx, pk, &responder))
require.NoError(t, router.Request(ctx, pk, responder))
validateEchoFlow(t, <-muxer.cfdToEdge, pk)
}
}()
@@ -358,13 +347,11 @@ func TestICMPRouterRejectNotEcho(t *testing.T) {
}
func testICMPRouterRejectNotEcho(t *testing.T, srcDstIP netip.Addr, msgs []icmp.Message) {
router, err := NewICMPRouter(localhostIP, localhostIPv6, "", &noopLogger, testFunnelIdleTimeout)
router, err := NewICMPRouter(localhostIP, localhostIPv6, &noopLogger, testFunnelIdleTimeout)
require.NoError(t, err)
muxer := newMockMuxer(1)
responder := packetResponder{
datagramMuxer: muxer,
}
responder := newPacketResponder(muxer, 0, packet.NewEncoder())
protocol := layers.IPProtocolICMPv4
if srcDstIP.Is6() {
protocol = layers.IPProtocolICMPv6
@@ -379,7 +366,7 @@ func testICMPRouterRejectNotEcho(t *testing.T, srcDstIP netip.Addr, msgs []icmp.
},
Message: &m,
}
require.Error(t, router.Request(context.Background(), &pk, &responder))
require.Error(t, router.Request(context.Background(), &pk, responder))
}
}
+1 -1
View File
@@ -39,7 +39,7 @@ func DialUDPAddrPort(dest netip.AddrPort) (*net.UDPConn, error) {
// address as context.
udpConn, err := net.DialUDP("udp", nil, addr)
if err != nil {
return nil, fmt.Errorf("unable to create UDP proxy to origin (%v:%v): %w", dest.Addr(), dest.Port(), err)
return nil, fmt.Errorf("unable to dial udp to origin %s: %w", dest, err)
}
return udpConn, nil
+53 -45
View File
@@ -3,7 +3,6 @@ package ingress
import (
"context"
"fmt"
"net/netip"
"github.com/rs/zerolog"
"go.opentelemetry.io/otel/attribute"
@@ -23,29 +22,23 @@ type muxer interface {
// PacketRouter routes packets between Upstream and ICMPRouter. Currently it rejects all other type of ICMP packets
type PacketRouter struct {
globalConfig *GlobalRouterConfig
muxer muxer
logger *zerolog.Logger
icmpDecoder *packet.ICMPDecoder
encoder *packet.Encoder
}
// GlobalRouterConfig is the configuration shared by all instance of Router.
type GlobalRouterConfig struct {
ICMPRouter *icmpRouter
IPv4Src netip.Addr
IPv6Src netip.Addr
Zone string
icmpRouter ICMPRouter
muxer muxer
connIndex uint8
logger *zerolog.Logger
encoder *packet.Encoder
decoder *packet.ICMPDecoder
}
// NewPacketRouter creates a PacketRouter that handles ICMP packets. Packets are read from muxer but dropped if globalConfig is nil.
func NewPacketRouter(globalConfig *GlobalRouterConfig, muxer muxer, logger *zerolog.Logger) *PacketRouter {
func NewPacketRouter(icmpRouter ICMPRouter, muxer muxer, connIndex uint8, logger *zerolog.Logger) *PacketRouter {
return &PacketRouter{
globalConfig: globalConfig,
muxer: muxer,
logger: logger,
icmpDecoder: packet.NewICMPDecoder(),
encoder: packet.NewEncoder(),
icmpRouter: icmpRouter,
muxer: muxer,
connIndex: connIndex,
logger: logger,
encoder: packet.NewEncoder(),
decoder: packet.NewICMPDecoder(),
}
}
@@ -59,14 +52,13 @@ func (r *PacketRouter) Serve(ctx context.Context) error {
}
}
func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packetResponder, error) {
func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, ICMPResponder, error) {
pk, err := r.muxer.ReceivePacket(ctx)
if err != nil {
return packet.RawPacket{}, nil, err
}
responder := &packetResponder{
datagramMuxer: r.muxer,
}
responder := newPacketResponder(r.muxer, r.connIndex, packet.NewEncoder())
switch pk.Type() {
case quicpogs.DatagramTypeIP:
return packet.RawPacket{Data: pk.Payload()}, responder, nil
@@ -75,8 +67,8 @@ func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packe
if err := identity.UnmarshalBinary(pk.Metadata()); err != nil {
r.logger.Err(err).Bytes("tracingIdentity", pk.Metadata()).Msg("Failed to unmarshal tracing identity")
} else {
responder.tracedCtx = tracing.NewTracedContext(ctx, identity.String(), r.logger)
responder.serializedIdentity = pk.Metadata()
tracedCtx := tracing.NewTracedContext(ctx, identity.String(), r.logger)
responder.AddTraceContext(tracedCtx, pk.Metadata())
}
return packet.RawPacket{Data: pk.Payload()}, responder, nil
default:
@@ -84,27 +76,27 @@ func (r *PacketRouter) nextPacket(ctx context.Context) (packet.RawPacket, *packe
}
}
func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPacket, responder *packetResponder) {
func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPacket, responder ICMPResponder) {
// ICMP Proxy feature is disabled, drop packets
if r.globalConfig == nil {
if r.icmpRouter == nil {
return
}
icmpPacket, err := r.icmpDecoder.Decode(rawPacket)
icmpPacket, err := r.decoder.Decode(rawPacket)
if err != nil {
r.logger.Err(err).Msg("Failed to decode ICMP packet from quic datagram")
return
}
if icmpPacket.TTL <= 1 {
if err := r.sendTTLExceedMsg(ctx, icmpPacket, rawPacket, r.encoder); err != nil {
if err := r.sendTTLExceedMsg(icmpPacket, rawPacket); err != nil {
r.logger.Err(err).Msg("Failed to return ICMP TTL exceed error")
}
return
}
icmpPacket.TTL--
if err := r.globalConfig.ICMPRouter.Request(ctx, icmpPacket, responder); err != nil {
if err := r.icmpRouter.Request(ctx, icmpPacket, responder); err != nil {
r.logger.Err(err).
Str("src", icmpPacket.Src.String()).
Str("dst", icmpPacket.Dst.String()).
@@ -113,16 +105,9 @@ func (r *PacketRouter) handlePacket(ctx context.Context, rawPacket packet.RawPac
}
}
func (r *PacketRouter) sendTTLExceedMsg(ctx context.Context, pk *packet.ICMP, rawPacket packet.RawPacket, encoder *packet.Encoder) error {
var srcIP netip.Addr
if pk.Dst.Is4() {
srcIP = r.globalConfig.IPv4Src
} else {
srcIP = r.globalConfig.IPv6Src
}
ttlExceedPacket := packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, srcIP)
encodedTTLExceed, err := encoder.Encode(ttlExceedPacket)
func (r *PacketRouter) sendTTLExceedMsg(pk *packet.ICMP, rawPacket packet.RawPacket) error {
icmpTTLPacket := r.icmpRouter.ConvertToTTLExceeded(pk, rawPacket)
encodedTTLExceed, err := r.encoder.Encode(icmpTTLPacket)
if err != nil {
return err
}
@@ -132,22 +117,45 @@ func (r *PacketRouter) sendTTLExceedMsg(ctx context.Context, pk *packet.ICMP, ra
// packetResponder should not be used concurrently. This assumption is upheld because reply packets are ready one-by-one
type packetResponder struct {
datagramMuxer muxer
connIndex uint8
encoder *packet.Encoder
tracedCtx *tracing.TracedContext
serializedIdentity []byte
// hadReply tracks if there has been any reply for this flow
hadReply bool
}
func newPacketResponder(datagramMuxer muxer, connIndex uint8, encoder *packet.Encoder) ICMPResponder {
return &packetResponder{
datagramMuxer: datagramMuxer,
connIndex: connIndex,
encoder: encoder,
}
}
func (pr *packetResponder) tracingEnabled() bool {
return pr.tracedCtx != nil
}
func (pr *packetResponder) returnPacket(rawPacket packet.RawPacket) error {
func (pr *packetResponder) ConnectionIndex() uint8 {
return pr.connIndex
}
func (pr *packetResponder) ReturnPacket(pk *packet.ICMP) error {
rawPacket, err := pr.encoder.Encode(pk)
if err != nil {
return err
}
pr.hadReply = true
return pr.datagramMuxer.SendPacket(quicpogs.RawPacket(rawPacket))
}
func (pr *packetResponder) requestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
func (pr *packetResponder) AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte) {
pr.tracedCtx = tracedCtx
pr.serializedIdentity = serializedIdentity
}
func (pr *packetResponder) RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
if !pr.tracingEnabled() {
return ctx, tracing.NewNoopSpan()
}
@@ -157,14 +165,14 @@ func (pr *packetResponder) requestSpan(ctx context.Context, pk *packet.ICMP) (co
))
}
func (pr *packetResponder) replySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
func (pr *packetResponder) ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
if !pr.tracingEnabled() || pr.hadReply {
return ctx, tracing.NewNoopSpan()
}
return pr.tracedCtx.Tracer().Start(pr.tracedCtx, "icmp-echo-reply")
}
func (pr *packetResponder) exportSpan() {
func (pr *packetResponder) ExportSpan() {
if !pr.tracingEnabled() {
return
}
+8 -7
View File
@@ -19,16 +19,17 @@ import (
)
var (
packetConfig = &GlobalRouterConfig{
ICMPRouter: nil,
IPv4Src: netip.MustParseAddr("172.16.0.1"),
IPv6Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
defaultRouter = &icmpRouter{
ipv4Proxy: nil,
ipv4Src: netip.MustParseAddr("172.16.0.1"),
ipv6Proxy: nil,
ipv6Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
}
)
func TestRouterReturnTTLExceed(t *testing.T) {
muxer := newMockMuxer(0)
router := NewPacketRouter(packetConfig, muxer, &noopLogger)
router := NewPacketRouter(defaultRouter, muxer, 0, &noopLogger)
ctx, cancel := context.WithCancel(context.Background())
routerStopped := make(chan struct{})
go func() {
@@ -53,7 +54,7 @@ func TestRouterReturnTTLExceed(t *testing.T) {
},
},
}
assertTTLExceed(t, &pk, router.globalConfig.IPv4Src, muxer)
assertTTLExceed(t, &pk, defaultRouter.ipv4Src, muxer)
pk = packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("fd51:2391:523:f4ee::1"),
@@ -71,7 +72,7 @@ func TestRouterReturnTTLExceed(t *testing.T) {
},
},
}
assertTTLExceed(t, &pk, router.globalConfig.IPv6Src, muxer)
assertTTLExceed(t, &pk, defaultRouter.ipv6Src, muxer)
cancel()
<-routerStopped
+3 -3
View File
@@ -76,10 +76,10 @@ func CreateConfig(
var file *FileConfig
var rolling *RollingConfig
if rollingLogPath != "" {
rolling = createRollingConfig(rollingLogPath)
} else if nonRollingLogFilePath != "" {
if nonRollingLogFilePath != "" {
file = createFileConfig(nonRollingLogFilePath)
} else if rollingLogPath != "" {
rolling = createRollingConfig(rollingLogPath)
}
if minLevel == "" {
+70
View File
@@ -10,10 +10,13 @@ import (
"sync"
"time"
"github.com/facebookgo/grace/gracenet"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/rs/zerolog"
"golang.org/x/net/trace"
"github.com/cloudflare/cloudflared/diagnostic"
)
const (
@@ -21,8 +24,37 @@ const (
defaultShutdownTimeout = time.Second * 15
)
// This variable is set at compile time to allow the default local address to change.
var Runtime = "host"
func GetMetricsDefaultAddress(runtimeType string) string {
// When issuing the diagnostic command we may have to reach a server that is
// running in a virtual enviroment and in that case we must bind to 0.0.0.0
// otherwise the server won't be reachable.
switch runtimeType {
case "virtual":
return "0.0.0.0:0"
default:
return "localhost:0"
}
}
// GetMetricsKnownAddresses returns the addresses used by the metrics server to bind at
// startup time to allow a semi-deterministic approach to know where the server is listening at.
// The ports were selected because at the time we are in 2024 and they do not collide with any
// know/registered port according https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers.
func GetMetricsKnownAddresses(runtimeType string) []string {
switch runtimeType {
case "virtual":
return []string{"0.0.0.0:20241", "0.0.0.0:20242", "0.0.0.0:20243", "0.0.0.0:20244", "0.0.0.0:20245"}
default:
return []string{"localhost:20241", "localhost:20242", "localhost:20243", "localhost:20244", "localhost:20245"}
}
}
type Config struct {
ReadyServer *ReadyServer
DiagnosticHandler *diagnostic.Handler
QuickTunnelHostname string
Orchestrator orchestrator
@@ -62,9 +94,47 @@ func newMetricsHandler(
})
}
config.DiagnosticHandler.InstallEndpoints(router)
return router
}
// CreateMetricsListener will create a new [net.Listener] by using an
// known set of ports when the default address is passed with the fallback
// of choosing a random port when none is available.
//
// In case the provided address is not the default one then it will be used
// as is.
func CreateMetricsListener(listeners *gracenet.Net, laddr string) (net.Listener, error) {
if laddr == GetMetricsDefaultAddress(Runtime) {
// On the presence of the default address select
// a port from the known set of addresses iteratively.
addresses := GetMetricsKnownAddresses(Runtime)
for _, address := range addresses {
listener, err := listeners.Listen("tcp", address)
if err == nil {
return listener, nil
}
}
// When no port is available then bind to a random one
listener, err := listeners.Listen("tcp", laddr)
if err != nil {
return nil, fmt.Errorf("failed to listen to default metrics address: %w", err)
}
return listener, nil
}
// Explicitly got a local address then bind to it
listener, err := listeners.Listen("tcp", laddr)
if err != nil {
return nil, fmt.Errorf("failed to bind to address (%s): %w", laddr, err)
}
return listener, nil
}
func ServeMetrics(
l net.Listener,
ctx context.Context,
+52
View File
@@ -0,0 +1,52 @@
package metrics_test
import (
"testing"
"github.com/facebookgo/grace/gracenet"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/metrics"
)
func TestMetricsListenerCreation(t *testing.T) {
t.Parallel()
listeners := gracenet.Net{}
listener1, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20241", listener1.Addr().String())
require.NoError(t, err)
listener2, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20242", listener2.Addr().String())
require.NoError(t, err)
listener3, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20243", listener3.Addr().String())
require.NoError(t, err)
listener4, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20244", listener4.Addr().String())
require.NoError(t, err)
listener5, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
assert.Equal(t, "127.0.0.1:20245", listener5.Addr().String())
require.NoError(t, err)
listener6, err := metrics.CreateMetricsListener(&listeners, metrics.GetMetricsDefaultAddress("host"))
addresses := [5]string{"127.0.0.1:20241", "127.0.0.1:20242", "127.0.0.1:20243", "127.0.0.1:20244", "127.0.0.1:20245"}
assert.NotContains(t, addresses, listener6.Addr().String())
require.NoError(t, err)
listener7, err := metrics.CreateMetricsListener(&listeners, "localhost:12345")
assert.Equal(t, "127.0.0.1:12345", listener7.Addr().String())
require.NoError(t, err)
err = listener1.Close()
require.NoError(t, err)
err = listener2.Close()
require.NoError(t, err)
err = listener3.Close()
require.NoError(t, err)
err = listener4.Close()
require.NoError(t, err)
err = listener5.Close()
require.NoError(t, err)
err = listener6.Close()
require.NoError(t, err)
err = listener7.Close()
require.NoError(t, err)
}
+6 -9
View File
@@ -6,9 +6,7 @@ import (
"net/http"
"github.com/google/uuid"
"github.com/rs/zerolog"
conn "github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/tunnelstate"
)
@@ -19,17 +17,16 @@ type ReadyServer struct {
}
// NewReadyServer initializes a ReadyServer and starts listening for dis/connection events.
func NewReadyServer(log *zerolog.Logger, clientID uuid.UUID) *ReadyServer {
func NewReadyServer(
clientID uuid.UUID,
tracker *tunnelstate.ConnTracker,
) *ReadyServer {
return &ReadyServer{
clientID: clientID,
tracker: tunnelstate.NewConnTracker(log),
clientID,
tracker,
}
}
func (rs *ReadyServer) OnTunnelEvent(c conn.Event) {
rs.tracker.OnTunnelEvent(c)
}
type body struct {
Status int `json:"status"`
ReadyConnections uint `json:"readyConnections"`
+44 -74
View File
@@ -1,136 +1,106 @@
package metrics
package metrics_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/google/uuid"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/metrics"
"github.com/cloudflare/cloudflared/tunnelstate"
)
func TestReadyServer_makeResponse(t *testing.T) {
type fields struct {
isConnected map[uint8]tunnelstate.ConnectionInfo
}
tests := []struct {
name string
fields fields
wantOK bool
wantReadyConnections uint
}{
{
name: "One connection online => HTTP 200",
fields: fields{
isConnected: map[uint8]tunnelstate.ConnectionInfo{
0: {IsConnected: false},
1: {IsConnected: false},
2: {IsConnected: true},
3: {IsConnected: false},
},
},
wantOK: true,
wantReadyConnections: 1,
},
{
name: "No connections online => no HTTP 200",
fields: fields{
isConnected: map[uint8]tunnelstate.ConnectionInfo{
0: {IsConnected: false},
1: {IsConnected: false},
2: {IsConnected: false},
3: {IsConnected: false},
},
},
wantReadyConnections: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rs := &ReadyServer{
tracker: tunnelstate.MockedConnTracker(tt.fields.isConnected),
}
gotStatusCode, gotReadyConnections := rs.makeResponse()
if tt.wantOK && gotStatusCode != http.StatusOK {
t.Errorf("ReadyServer.makeResponse() gotStatusCode = %v, want ok = %v", gotStatusCode, tt.wantOK)
}
if gotReadyConnections != tt.wantReadyConnections {
t.Errorf("ReadyServer.makeResponse() gotReadyConnections = %v, want %v", gotReadyConnections, tt.wantReadyConnections)
}
})
func mockRequest(t *testing.T, readyServer *metrics.ReadyServer) (int, uint) {
t.Helper()
var readyreadyConnections struct {
Status int `json:"status"`
ReadyConnections uint `json:"readyConnections"`
ConnectorID uuid.UUID `json:"connectorId"`
}
rec := httptest.NewRecorder()
readyServer.ServeHTTP(rec, nil)
decoder := json.NewDecoder(rec.Body)
err := decoder.Decode(&readyreadyConnections)
require.NoError(t, err)
return rec.Code, readyreadyConnections.ReadyConnections
}
func TestReadinessEventHandling(t *testing.T) {
nopLogger := zerolog.Nop()
rs := NewReadyServer(&nopLogger, uuid.Nil)
tracker := tunnelstate.NewConnTracker(&nopLogger)
rs := metrics.NewReadyServer(uuid.Nil, tracker)
// start not ok
code, ready := rs.makeResponse()
code, readyConnections := mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// one connected => ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
assert.EqualValues(t, 1, readyConnections)
// another connected => still ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 2,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 2, ready)
assert.EqualValues(t, 2, readyConnections)
// one reconnecting => still ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 2,
EventType: connection.Reconnecting,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
assert.EqualValues(t, 1, readyConnections)
// Regression test for TUN-3777
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.RegisteringTunnel,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// other connected then unregistered => not ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Connected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.EqualValues(t, http.StatusOK, code)
assert.EqualValues(t, 1, ready)
rs.OnTunnelEvent(connection.Event{
assert.EqualValues(t, 1, readyConnections)
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Unregistering,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
// other disconnected => not ok
rs.OnTunnelEvent(connection.Event{
tracker.OnTunnelEvent(connection.Event{
Index: 1,
EventType: connection.Disconnected,
})
code, ready = rs.makeResponse()
code, readyConnections = mockRequest(t, rs)
assert.NotEqualValues(t, http.StatusOK, code)
assert.Zero(t, ready)
assert.Zero(t, readyConnections)
}
+15
View File
@@ -254,3 +254,18 @@ func (u *UDP) EncodeLayers() ([]gopacket.SerializableLayer, error) {
udpLayer.SetNetworkLayerForChecksum(ipLayers[0].(gopacket.NetworkLayer))
return append(ipLayers, &udpLayer), nil
}
func FuzzIPDecoder(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
ipDecoder := NewIPDecoder()
ipDecoder.Decode(RawPacket{Data: data})
})
}
func FuzzICMPDecoder(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
icmpDecoder := NewICMPDecoder()
icmpDecoder.Decode(RawPacket{Data: data})
})
}
+4 -2
View File
@@ -1,6 +1,8 @@
package packet
import "github.com/google/gopacket"
import (
"github.com/google/gopacket"
)
var (
serializeOpts = gopacket.SerializeOptions{
@@ -25,7 +27,7 @@ func NewEncoder() *Encoder {
}
}
func (e Encoder) Encode(packet Packet) (RawPacket, error) {
func (e *Encoder) Encode(packet Packet) (RawPacket, error) {
encodedLayers, err := packet.EncodeLayers()
if err != nil {
return RawPacket{}, err
+63 -7
View File
@@ -222,11 +222,11 @@ const (
//
// This method should be used in-place of MarshalBinary which will allocate in-place the required byte array to return.
func MarshalPayloadHeaderTo(requestID RequestID, payload []byte) error {
if len(payload) < 17 {
if len(payload) < DatagramPayloadHeaderLen {
return wrapMarshalErr(ErrDatagramPayloadHeaderTooSmall)
}
payload[0] = byte(UDPSessionPayloadType)
return requestID.MarshalBinaryTo(payload[1:17])
return requestID.MarshalBinaryTo(payload[1:DatagramPayloadHeaderLen])
}
func (s *UDPSessionPayloadDatagram) UnmarshalBinary(data []byte) error {
@@ -239,18 +239,18 @@ func (s *UDPSessionPayloadDatagram) UnmarshalBinary(data []byte) error {
}
// Make sure that the slice provided is the right size to be parsed.
if len(data) < 17 || len(data) > maxPayloadPlusHeaderLen {
if len(data) < DatagramPayloadHeaderLen || len(data) > maxPayloadPlusHeaderLen {
return wrapUnmarshalErr(ErrDatagramPayloadInvalidSize)
}
requestID, err := RequestIDFromSlice(data[1:17])
requestID, err := RequestIDFromSlice(data[1:DatagramPayloadHeaderLen])
if err != nil {
return wrapUnmarshalErr(err)
}
*s = UDPSessionPayloadDatagram{
RequestID: requestID,
Payload: data[17:],
Payload: data[DatagramPayloadHeaderLen:],
}
return nil
}
@@ -284,8 +284,6 @@ const (
ResponseDestinationUnreachable SessionRegistrationResp = 0x01
// Session registration was unable to bind to a local UDP socket.
ResponseUnableToBindSocket SessionRegistrationResp = 0x02
// Session registration is already bound to another connection.
ResponseSessionAlreadyConnected SessionRegistrationResp = 0x03
// Session registration failed with an unexpected error but provided a message.
ResponseErrorWithMsg SessionRegistrationResp = 0xff
)
@@ -372,3 +370,61 @@ func (s *UDPSessionRegistrationResponseDatagram) UnmarshalBinary(data []byte) er
}
return nil
}
// ICMPDatagram is used to propagate ICMPv4 and ICMPv6 payloads.
type ICMPDatagram struct {
Payload []byte
}
// The maximum size that an ICMP packet can be.
const maxICMPPayloadLen = maxDatagramPayloadLen
// The datagram structure for ICMPDatagram is:
//
// 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// 0| Type | |
// +-+-+-+-+-+-+-+-+ +
// . Payload .
// . .
// . .
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
func (d *ICMPDatagram) MarshalBinary() (data []byte, err error) {
if len(d.Payload) > maxICMPPayloadLen {
return nil, wrapMarshalErr(ErrDatagramICMPPayloadTooLarge)
}
// We shouldn't attempt to marshal an ICMP datagram with no ICMP payload provided
if len(d.Payload) == 0 {
return nil, wrapMarshalErr(ErrDatagramICMPPayloadMissing)
}
// Make room for the 1 byte ICMPType header
datagram := make([]byte, len(d.Payload)+datagramTypeLen)
datagram[0] = byte(ICMPType)
copy(datagram[1:], d.Payload)
return datagram, nil
}
func (d *ICMPDatagram) UnmarshalBinary(data []byte) error {
datagramType, err := ParseDatagramType(data)
if err != nil {
return wrapUnmarshalErr(err)
}
if datagramType != ICMPType {
return wrapUnmarshalErr(ErrInvalidDatagramType)
}
if len(data[1:]) > maxDatagramPayloadLen {
return wrapUnmarshalErr(ErrDatagramICMPPayloadTooLarge)
}
// We shouldn't attempt to unmarshal an ICMP datagram with no ICMP payload provided
if len(data[1:]) == 0 {
return wrapUnmarshalErr(ErrDatagramICMPPayloadMissing)
}
payload := make([]byte, len(data[1:]))
copy(payload, data[1:])
d.Payload = payload
return nil
}
+2
View File
@@ -15,6 +15,8 @@ var (
ErrDatagramResponseInvalidSize error = errors.New("datagram response is an invalid size")
ErrDatagramResponseMsgTooLargeMaximum error = fmt.Errorf("datagram response error message length exceeds the length of the datagram maximum: %d", maxResponseErrorMessageLen)
ErrDatagramResponseMsgTooLargeDatagram error = fmt.Errorf("datagram response error message length exceeds the length of the provided datagram")
ErrDatagramICMPPayloadTooLarge error = fmt.Errorf("datagram icmp payload exceeds %d bytes", maxICMPPayloadLen)
ErrDatagramICMPPayloadMissing error = errors.New("datagram icmp payload is missing")
)
func wrapMarshalErr(err error) error {
+97
View File
@@ -160,6 +160,12 @@ func TestTypeUnmarshalErrors(t *testing.T) {
if !errors.Is(err, v3.ErrDatagramHeaderTooSmall) {
t.Errorf("expected invalid length to throw error")
}
d4 := v3.ICMPDatagram{}
err = d4.UnmarshalBinary([]byte{})
if !errors.Is(err, v3.ErrDatagramHeaderTooSmall) {
t.Errorf("expected invalid length to throw error")
}
})
t.Run("invalid types", func(t *testing.T) {
@@ -180,6 +186,12 @@ func TestTypeUnmarshalErrors(t *testing.T) {
if !errors.Is(err, v3.ErrInvalidDatagramType) {
t.Errorf("expected invalid type to throw error")
}
d4 := v3.ICMPDatagram{}
err = d4.UnmarshalBinary([]byte{byte(v3.UDPSessionPayloadType)})
if !errors.Is(err, v3.ErrInvalidDatagramType) {
t.Errorf("expected invalid type to throw error")
}
})
}
@@ -343,6 +355,54 @@ func TestSessionRegistrationResponse(t *testing.T) {
})
}
func TestICMPDatagram(t *testing.T) {
t.Run("basic", func(t *testing.T) {
payload := makePayload(128)
datagram := v3.ICMPDatagram{Payload: payload}
marshaled, err := datagram.MarshalBinary()
if err != nil {
t.Error(err)
}
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(marshaled)
if err != nil {
t.Error(err)
}
require.Equal(t, payload, unmarshaled.Payload)
})
t.Run("payload size empty", func(t *testing.T) {
payload := []byte{}
datagram := v3.ICMPDatagram{Payload: payload}
_, err := datagram.MarshalBinary()
if !errors.Is(err, v3.ErrDatagramICMPPayloadMissing) {
t.Errorf("expected an error: %s", err)
}
payload = []byte{byte(v3.ICMPType)}
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(payload)
if !errors.Is(err, v3.ErrDatagramICMPPayloadMissing) {
t.Errorf("expected an error: %s", err)
}
})
t.Run("payload size too large", func(t *testing.T) {
payload := makePayload(1280 + 1) // larger than the datagram size could be
datagram := v3.ICMPDatagram{Payload: payload}
_, err := datagram.MarshalBinary()
if !errors.Is(err, v3.ErrDatagramICMPPayloadTooLarge) {
t.Errorf("expected an error: %s", err)
}
payload = makePayload(1280 + 2) // larger than the datagram size could be + header
payload[0] = byte(v3.ICMPType)
unmarshaled := &v3.ICMPDatagram{}
err = unmarshaled.UnmarshalBinary(payload)
if !errors.Is(err, v3.ErrDatagramICMPPayloadTooLarge) {
t.Errorf("expected an error: %s", err)
}
})
}
func compareRegistrationDatagrams(t *testing.T, l *v3.UDPSessionRegistrationDatagram, r *v3.UDPSessionRegistrationDatagram) bool {
require.Equal(t, l.Payload, r.Payload)
return l.RequestID == r.RequestID &&
@@ -350,3 +410,40 @@ func compareRegistrationDatagrams(t *testing.T, l *v3.UDPSessionRegistrationData
l.IdleDurationHint == r.IdleDurationHint &&
l.Traced == r.Traced
}
func FuzzRegistrationDatagram(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
unmarshaled := v3.UDPSessionRegistrationDatagram{}
err := unmarshaled.UnmarshalBinary(data)
if err == nil {
_, _ = unmarshaled.MarshalBinary()
}
})
}
func FuzzPayloadDatagram(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
unmarshaled := v3.UDPSessionPayloadDatagram{}
_ = unmarshaled.UnmarshalBinary(data)
})
}
func FuzzRegistrationResponseDatagram(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
unmarshaled := v3.UDPSessionRegistrationResponseDatagram{}
err := unmarshaled.UnmarshalBinary(data)
if err == nil {
_, _ = unmarshaled.MarshalBinary()
}
})
}
func FuzzICMPDatagram(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
unmarshaled := v3.ICMPDatagram{}
err := unmarshaled.UnmarshalBinary(data)
if err == nil {
_, _ = unmarshaled.MarshalBinary()
}
})
}
+52
View File
@@ -0,0 +1,52 @@
package v3
import (
"context"
"github.com/rs/zerolog"
"go.opentelemetry.io/otel/trace"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
"github.com/cloudflare/cloudflared/tracing"
)
// packetResponder is an implementation of the [ingress.ICMPResponder] which provides the ICMP Flow manager the
// return path to return and ICMP Echo response back to the QUIC muxer.
type packetResponder struct {
datagramMuxer DatagramICMPWriter
connID uint8
}
func newPacketResponder(datagramMuxer DatagramICMPWriter, connID uint8) ingress.ICMPResponder {
return &packetResponder{
datagramMuxer,
connID,
}
}
func (pr *packetResponder) ConnectionIndex() uint8 {
return pr.connID
}
func (pr *packetResponder) ReturnPacket(pk *packet.ICMP) error {
return pr.datagramMuxer.SendICMPPacket(pk)
}
func (pr *packetResponder) AddTraceContext(tracedCtx *tracing.TracedContext, serializedIdentity []byte) {
// datagram v3 does not support tracing ICMP packets
}
func (pr *packetResponder) RequestSpan(ctx context.Context, pk *packet.ICMP) (context.Context, trace.Span) {
// datagram v3 does not support tracing ICMP packets
return ctx, tracing.NewNoopSpan()
}
func (pr *packetResponder) ReplySpan(ctx context.Context, logger *zerolog.Logger) (context.Context, trace.Span) {
// datagram v3 does not support tracing ICMP packets
return ctx, tracing.NewNoopSpan()
}
func (pr *packetResponder) ExportSpan() {
// datagram v3 does not support tracing ICMP packets
}
+45
View File
@@ -0,0 +1,45 @@
package v3_test
import (
"context"
"testing"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
)
type noopICMPRouter struct{}
func (noopICMPRouter) Request(ctx context.Context, pk *packet.ICMP, responder ingress.ICMPResponder) error {
return nil
}
func (noopICMPRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
return nil
}
type mockICMPRouter struct {
recv chan *packet.ICMP
}
func newMockICMPRouter() *mockICMPRouter {
return &mockICMPRouter{
recv: make(chan *packet.ICMP, 1),
}
}
func (m *mockICMPRouter) Request(ctx context.Context, pk *packet.ICMP, responder ingress.ICMPResponder) error {
m.recv <- pk
return nil
}
func (mockICMPRouter) ConvertToTTLExceeded(pk *packet.ICMP, rawPacket packet.RawPacket) *packet.ICMP {
return packet.NewICMPTTLExceedPacket(pk.IP, rawPacket, testLocalAddr.AddrPort().Addr())
}
func assertICMPEqual(t *testing.T, expected *packet.ICMP, actual *packet.ICMP) {
if expected.Src != actual.Src {
t.Fatalf("Src address not equal: %+v\t%+v", expected, actual)
}
if expected.Dst != actual.Dst {
t.Fatalf("Dst address not equal: %+v\t%+v", expected, actual)
}
}
+32 -16
View File
@@ -7,20 +7,22 @@ import (
"sync"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/ingress"
)
var (
ErrSessionNotFound = errors.New("session not found")
ErrSessionBoundToOtherConn = errors.New("session is in use by another connection")
// ErrSessionNotFound indicates that a session has not been registered yet for the request id.
ErrSessionNotFound = errors.New("flow not found")
// ErrSessionBoundToOtherConn is returned when a registration already exists for a different connection.
ErrSessionBoundToOtherConn = errors.New("flow is in use by another connection")
// ErrSessionAlreadyRegistered is returned when a registration already exists for this connection.
ErrSessionAlreadyRegistered = errors.New("flow is already registered for this connection")
)
type SessionManager interface {
// RegisterSession will register a new session if it does not already exist for the request ID.
// During new session creation, the session will also bind the UDP socket for the origin.
// If the session exists for a different connection, it will return [ErrSessionBoundToOtherConn].
RegisterSession(request *UDPSessionRegistrationDatagram, conn DatagramWriter) (Session, error)
RegisterSession(request *UDPSessionRegistrationDatagram, conn DatagramConn) (Session, error)
// GetSession returns an active session if available for the provided connection.
// If the session does not exist, it will return [ErrSessionNotFound]. If the session exists for a different
// connection, it will return [ErrSessionBoundToOtherConn].
@@ -33,33 +35,47 @@ type SessionManager interface {
type DialUDP func(dest netip.AddrPort) (*net.UDPConn, error)
type sessionManager struct {
sessions map[RequestID]Session
mutex sync.RWMutex
log *zerolog.Logger
sessions map[RequestID]Session
mutex sync.RWMutex
originDialer DialUDP
metrics Metrics
log *zerolog.Logger
}
func NewSessionManager(log *zerolog.Logger, originDialer DialUDP) SessionManager {
func NewSessionManager(metrics Metrics, log *zerolog.Logger, originDialer DialUDP) SessionManager {
return &sessionManager{
sessions: make(map[RequestID]Session),
log: log,
sessions: make(map[RequestID]Session),
originDialer: originDialer,
metrics: metrics,
log: log,
}
}
func (s *sessionManager) RegisterSession(request *UDPSessionRegistrationDatagram, conn DatagramWriter) (Session, error) {
func (s *sessionManager) RegisterSession(request *UDPSessionRegistrationDatagram, conn DatagramConn) (Session, error) {
s.mutex.Lock()
defer s.mutex.Unlock()
// Check to make sure session doesn't already exist for requestID
_, exists := s.sessions[request.RequestID]
if exists {
if session, exists := s.sessions[request.RequestID]; exists {
if conn.ID() == session.ConnectionID() {
return nil, ErrSessionAlreadyRegistered
}
return nil, ErrSessionBoundToOtherConn
}
// Attempt to bind the UDP socket for the new session
origin, err := ingress.DialUDPAddrPort(request.Dest)
origin, err := s.originDialer(request.Dest)
if err != nil {
return nil, err
}
// Create and insert the new session in the map
session := NewSession(request.RequestID, request.IdleDurationHint, origin, conn, s.log)
session := NewSession(
request.RequestID,
request.IdleDurationHint,
origin,
origin.RemoteAddr(),
origin.LocalAddr(),
conn,
s.metrics,
s.log)
s.sessions[request.RequestID] = session
return session, nil
}
+9 -3
View File
@@ -15,7 +15,7 @@ import (
func TestRegisterSession(t *testing.T) {
log := zerolog.Nop()
manager := v3.NewSessionManager(&log, ingress.DialUDPAddrPort)
manager := v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort)
request := v3.UDPSessionRegistrationDatagram{
RequestID: testRequestID,
@@ -34,8 +34,14 @@ func TestRegisterSession(t *testing.T) {
// We shouldn't be able to register another session with the same request id
_, err = manager.RegisterSession(&request, &noopEyeball{})
if !errors.Is(err, v3.ErrSessionAlreadyRegistered) {
t.Fatalf("session is already registered for this connection: %v", err)
}
// We shouldn't be able to register another session with the same request id for a different connection
_, err = manager.RegisterSession(&request, &noopEyeball{connID: 1})
if !errors.Is(err, v3.ErrSessionBoundToOtherConn) {
t.Fatalf("session should not be able to be registered again: %v", err)
t.Fatalf("session is already registered for a separate connection: %v", err)
}
// Get session
@@ -65,7 +71,7 @@ func TestRegisterSession(t *testing.T) {
func TestGetSession_Empty(t *testing.T) {
log := zerolog.Nop()
manager := v3.NewSessionManager(&log, ingress.DialUDPAddrPort)
manager := v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort)
_, err := manager.GetSession(testRequestID)
if !errors.Is(err, v3.ErrSessionNotFound) {
+90
View File
@@ -0,0 +1,90 @@
package v3
import (
"github.com/prometheus/client_golang/prometheus"
)
const (
namespace = "cloudflared"
subsystem = "udp"
)
type Metrics interface {
IncrementFlows()
DecrementFlows()
PayloadTooLarge()
RetryFlowResponse()
MigrateFlow()
}
type metrics struct {
activeUDPFlows prometheus.Gauge
totalUDPFlows prometheus.Counter
payloadTooLarge prometheus.Counter
retryFlowResponses prometheus.Counter
migratedFlows prometheus.Counter
}
func (m *metrics) IncrementFlows() {
m.totalUDPFlows.Inc()
m.activeUDPFlows.Inc()
}
func (m *metrics) DecrementFlows() {
m.activeUDPFlows.Dec()
}
func (m *metrics) PayloadTooLarge() {
m.payloadTooLarge.Inc()
}
func (m *metrics) RetryFlowResponse() {
m.retryFlowResponses.Inc()
}
func (m *metrics) MigrateFlow() {
m.migratedFlows.Inc()
}
func NewMetrics(registerer prometheus.Registerer) Metrics {
m := &metrics{
activeUDPFlows: prometheus.NewGauge(prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "active_flows",
Help: "Concurrent count of UDP flows that are being proxied to any origin",
}),
totalUDPFlows: prometheus.NewCounter(prometheus.CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "total_flows",
Help: "Total count of UDP flows that have been proxied to any origin",
}),
payloadTooLarge: prometheus.NewCounter(prometheus.CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "payload_too_large",
Help: "Total count of UDP flows that have had origin payloads that are too large to proxy",
}),
retryFlowResponses: prometheus.NewCounter(prometheus.CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "retry_flow_responses",
Help: "Total count of UDP flows that have had to send their registration response more than once",
}),
migratedFlows: prometheus.NewCounter(prometheus.CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "migrated_flows",
Help: "Total count of UDP flows have been migrated across local connections",
}),
}
registerer.MustRegister(
m.activeUDPFlows,
m.totalUDPFlows,
m.payloadTooLarge,
m.retryFlowResponses,
m.migratedFlows,
)
return m
}
+9
View File
@@ -0,0 +1,9 @@
package v3_test
type noopMetrics struct{}
func (noopMetrics) IncrementFlows() {}
func (noopMetrics) DecrementFlows() {}
func (noopMetrics) PayloadTooLarge() {}
func (noopMetrics) RetryFlowResponse() {}
func (noopMetrics) MigrateFlow() {}
+198 -28
View File
@@ -3,29 +3,48 @@ package v3
import (
"context"
"errors"
"fmt"
"sync"
"time"
"github.com/rs/zerolog"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
)
const (
// Allocating a 16 channel buffer here allows for the writer to be slightly faster than the reader.
// This has worked previously well for datagramv2, so we will start with this as well
demuxChanCapacity = 16
logSrcKey = "src"
logDstKey = "dst"
logICMPTypeKey = "type"
logDurationKey = "durationMS"
)
// DatagramConn is the bridge that multiplexes writes and reads of datagrams for UDP sessions and ICMP packets to
// a connection.
type DatagramConn interface {
DatagramWriter
DatagramUDPWriter
DatagramICMPWriter
// Serve provides a server interface to process and handle incoming QUIC datagrams and demux their datagram v3 payloads.
Serve(context.Context) error
// ID indicates connection index identifier
ID() uint8
}
// DatagramWriter provides the Muxer interface to create proper Datagrams when sending over a connection.
type DatagramWriter interface {
// DatagramUDPWriter provides the Muxer interface to create proper UDP Datagrams when sending over a connection.
type DatagramUDPWriter interface {
SendUDPSessionDatagram(datagram []byte) error
SendUDPSessionResponse(id RequestID, resp SessionRegistrationResp) error
//SendICMPPacket(packet packet.IP) error
}
// DatagramICMPWriter provides the Muxer interface to create ICMP Datagrams when sending over a connection.
type DatagramICMPWriter interface {
SendICMPPacket(icmp *packet.ICMP) error
SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error
}
// QuicConnection provides an interface that matches [quic.Connection] for only the datagram operations.
@@ -41,24 +60,43 @@ type QuicConnection interface {
type datagramConn struct {
conn QuicConnection
index uint8
sessionManager SessionManager
icmpRouter ingress.ICMPRouter
metrics Metrics
logger *zerolog.Logger
datagrams chan []byte
readErrors chan error
icmpEncoderPool sync.Pool // a pool of *packet.Encoder
icmpDecoder *packet.ICMPDecoder
}
func NewDatagramConn(conn QuicConnection, sessionManager SessionManager, logger *zerolog.Logger) DatagramConn {
func NewDatagramConn(conn QuicConnection, sessionManager SessionManager, icmpRouter ingress.ICMPRouter, index uint8, metrics Metrics, logger *zerolog.Logger) DatagramConn {
log := logger.With().Uint8("datagramVersion", 3).Logger()
return &datagramConn{
conn: conn,
index: index,
sessionManager: sessionManager,
icmpRouter: icmpRouter,
metrics: metrics,
logger: &log,
datagrams: make(chan []byte, demuxChanCapacity),
readErrors: make(chan error, 2),
icmpEncoderPool: sync.Pool{
New: func() any {
return packet.NewEncoder()
},
},
icmpDecoder: packet.NewICMPDecoder(),
}
}
func (c *datagramConn) ID() uint8 {
return c.index
}
func (c *datagramConn) SendUDPSessionDatagram(datagram []byte) error {
return c.conn.SendDatagram(datagram)
}
@@ -75,6 +113,33 @@ func (c *datagramConn) SendUDPSessionResponse(id RequestID, resp SessionRegistra
return c.conn.SendDatagram(data)
}
func (c *datagramConn) SendICMPPacket(icmp *packet.ICMP) error {
cachedEncoder := c.icmpEncoderPool.Get()
// The encoded packet is a slice to a buffer owned by the encoder, so we shouldn't return the encoder back to the
// pool until the encoded packet is sent.
defer c.icmpEncoderPool.Put(cachedEncoder)
encoder, ok := cachedEncoder.(*packet.Encoder)
if !ok {
return fmt.Errorf("encoderPool returned %T, expect *packet.Encoder", cachedEncoder)
}
payload, err := encoder.Encode(icmp)
if err != nil {
return err
}
icmpDatagram := ICMPDatagram{
Payload: payload.Data,
}
datagram, err := icmpDatagram.MarshalBinary()
if err != nil {
return err
}
return c.conn.SendDatagram(datagram)
}
func (c *datagramConn) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error {
return c.SendICMPPacket(c.icmpRouter.ConvertToTTLExceeded(icmp, rawPacket))
}
var errReadTimeout error = errors.New("receive datagram timeout")
// pollDatagrams will read datagrams from the underlying connection until the provided context is done.
@@ -135,11 +200,12 @@ func (c *datagramConn) Serve(ctx context.Context) error {
c.logger.Err(err).Msgf("unable to unmarshal session registration datagram")
return
}
logger := c.logger.With().Str(logFlowID, reg.RequestID.String()).Logger()
// We bind the new session to the quic connection context instead of cloudflared context to allow for the
// quic connection to close and close only the sessions bound to it. Closing of cloudflared will also
// initiate the close of the quic connection, so we don't have to worry about the application context
// in the scope of a session.
c.handleSessionRegistrationDatagram(connCtx, reg)
c.handleSessionRegistrationDatagram(connCtx, reg, &logger)
case UDPSessionPayloadType:
payload := &UDPSessionPayloadDatagram{}
err := payload.UnmarshalBinary(datagram)
@@ -147,7 +213,16 @@ func (c *datagramConn) Serve(ctx context.Context) error {
c.logger.Err(err).Msgf("unable to unmarshal session payload datagram")
return
}
c.handleSessionPayloadDatagram(payload)
logger := c.logger.With().Str(logFlowID, payload.RequestID.String()).Logger()
c.handleSessionPayloadDatagram(payload, &logger)
case ICMPType:
packet := &ICMPDatagram{}
err := packet.UnmarshalBinary(datagram)
if err != nil {
c.logger.Err(err).Msgf("unable to unmarshal icmp datagram")
return
}
c.handleICMPPacket(packet)
case UDPSessionRegistrationResponseType:
// cloudflared should never expect to receive UDP session responses as it will not initiate new
// sessions towards the edge.
@@ -161,67 +236,162 @@ func (c *datagramConn) Serve(ctx context.Context) error {
}
// This method handles new registrations of a session and the serve loop for the session.
func (c *datagramConn) handleSessionRegistrationDatagram(ctx context.Context, datagram *UDPSessionRegistrationDatagram) {
func (c *datagramConn) handleSessionRegistrationDatagram(ctx context.Context, datagram *UDPSessionRegistrationDatagram, logger *zerolog.Logger) {
log := logger.With().
Str(logFlowID, datagram.RequestID.String()).
Str(logDstKey, datagram.Dest.String()).
Logger()
session, err := c.sessionManager.RegisterSession(datagram, c)
if err != nil {
c.logger.Err(err).Msgf("session registration failure")
c.handleSessionRegistrationFailure(datagram.RequestID, err)
switch err {
case nil:
// Continue as normal
case ErrSessionAlreadyRegistered:
// Session is already registered and likely the response got lost
c.handleSessionAlreadyRegistered(datagram.RequestID, &log)
return
case ErrSessionBoundToOtherConn:
// Session is already registered but to a different connection
c.handleSessionMigration(datagram.RequestID, &log)
return
default:
log.Err(err).Msgf("flow registration failure")
c.handleSessionRegistrationFailure(datagram.RequestID, &log)
return
}
log = log.With().Str(logSrcKey, session.LocalAddr().String()).Logger()
c.metrics.IncrementFlows()
// Make sure to eventually remove the session from the session manager when the session is closed
defer c.sessionManager.UnregisterSession(session.ID())
defer c.metrics.DecrementFlows()
// Respond that we are able to process the new session
err = c.SendUDPSessionResponse(datagram.RequestID, ResponseOk)
if err != nil {
c.logger.Err(err).Msgf("session registration failure: unable to send session registration response")
log.Err(err).Msgf("flow registration failure: unable to send session registration response")
return
}
// We bind the context of the session to the [quic.Connection] that initiated the session.
// [Session.Serve] is blocking and will continue this go routine till the end of the session lifetime.
start := time.Now()
err = session.Serve(ctx)
elapsedMS := time.Now().Sub(start).Milliseconds()
log = log.With().Int64(logDurationKey, elapsedMS).Logger()
if err == nil {
// We typically don't expect a session to close without some error response. [SessionIdleErr] is the typical
// expected error response.
c.logger.Warn().Msg("session was closed without explicit close or timeout")
log.Warn().Msg("flow closed: no explicit close or timeout elapsed")
return
}
// SessionIdleErr and SessionCloseErr are valid and successful error responses to end a session.
if errors.Is(err, SessionIdleErr{}) || errors.Is(err, SessionCloseErr) {
c.logger.Debug().Msg(err.Error())
log.Debug().Msgf("flow closed: %s", err.Error())
return
}
// All other errors should be reported as errors
c.logger.Err(err).Msgf("session was closed with an error")
log.Err(err).Msgf("flow closed with an error")
}
func (c *datagramConn) handleSessionRegistrationFailure(requestID RequestID, regErr error) {
var errResp SessionRegistrationResp
switch regErr {
case ErrSessionBoundToOtherConn:
errResp = ResponseSessionAlreadyConnected
default:
errResp = ResponseUnableToBindSocket
}
err := c.SendUDPSessionResponse(requestID, errResp)
func (c *datagramConn) handleSessionAlreadyRegistered(requestID RequestID, logger *zerolog.Logger) {
// Send another registration response since the session is already active
err := c.SendUDPSessionResponse(requestID, ResponseOk)
if err != nil {
c.logger.Err(err).Msgf("unable to send session registration error response (%d)", errResp)
logger.Err(err).Msgf("flow registration failure: unable to send an additional flow registration response")
return
}
session, err := c.sessionManager.GetSession(requestID)
if err != nil {
// If for some reason we can not find the session after attempting to register it, we can just return
// instead of trying to reset the idle timer for it.
return
}
// The session is already running in another routine so we want to restart the idle timeout since no proxied
// packets have come down yet.
session.ResetIdleTimer()
c.metrics.RetryFlowResponse()
logger.Debug().Msgf("flow registration response retry")
}
func (c *datagramConn) handleSessionMigration(requestID RequestID, logger *zerolog.Logger) {
// We need to migrate the currently running session to this edge connection.
session, err := c.sessionManager.GetSession(requestID)
if err != nil {
// If for some reason we can not find the session after attempting to register it, we can just return
// instead of trying to reset the idle timer for it.
return
}
// Migrate the session to use this edge connection instead of the currently running one.
// We also pass in this connection's logger to override the existing logger for the session.
session.Migrate(c, c.conn.Context(), c.logger)
// Send another registration response since the session is already active
err = c.SendUDPSessionResponse(requestID, ResponseOk)
if err != nil {
logger.Err(err).Msgf("flow registration failure: unable to send an additional flow registration response")
return
}
logger.Debug().Msgf("flow registration migration")
}
func (c *datagramConn) handleSessionRegistrationFailure(requestID RequestID, logger *zerolog.Logger) {
err := c.SendUDPSessionResponse(requestID, ResponseUnableToBindSocket)
if err != nil {
logger.Err(err).Msgf("unable to send flow registration error response (%d)", ResponseUnableToBindSocket)
}
}
// Handles incoming datagrams that need to be sent to a registered session.
func (c *datagramConn) handleSessionPayloadDatagram(datagram *UDPSessionPayloadDatagram) {
func (c *datagramConn) handleSessionPayloadDatagram(datagram *UDPSessionPayloadDatagram, logger *zerolog.Logger) {
s, err := c.sessionManager.GetSession(datagram.RequestID)
if err != nil {
c.logger.Err(err).Msgf("unable to find session")
logger.Err(err).Msgf("unable to find flow")
return
}
// We ignore the bytes written to the socket because any partial write must return an error.
_, err = s.Write(datagram.Payload)
if err != nil {
c.logger.Err(err).Msgf("unable to write payload for unavailable session")
logger.Err(err).Msgf("unable to write payload for the flow")
return
}
}
// Handles incoming ICMP datagrams.
func (c *datagramConn) handleICMPPacket(datagram *ICMPDatagram) {
if c.icmpRouter == nil {
// ICMPRouter is disabled so we drop the current packet and ignore all incoming ICMP packets
return
}
// Decode the provided ICMPDatagram as an ICMP packet
rawPacket := packet.RawPacket{Data: datagram.Payload}
icmp, err := c.icmpDecoder.Decode(rawPacket)
if err != nil {
c.logger.Err(err).Msgf("unable to marshal icmp packet")
return
}
// If the ICMP packet's TTL is expired, we won't send it to the origin and immediately return a TTL Exceeded Message
if icmp.TTL <= 1 {
if err := c.SendICMPTTLExceed(icmp, rawPacket); err != nil {
c.logger.Err(err).Msg("failed to return ICMP TTL exceed error")
}
return
}
icmp.TTL--
// The context isn't really needed here since it's only really used throughout the ICMP router as a way to store
// the tracing context, however datagram V3 does not support tracing ICMP packets, so we just pass the current
// connection context which will have no tracing information available.
err = c.icmpRouter.Request(c.conn.Context(), icmp, newPacketResponder(c, c.index))
if err != nil {
c.logger.Err(err).
Str(logSrcKey, icmp.Src.String()).
Str(logDstKey, icmp.Dst.String()).
Interface(logICMPTypeKey, icmp.Type).
Msgf("unable to write icmp datagram to origin")
return
}
}
+333 -48
View File
@@ -11,23 +11,31 @@ import (
"testing"
"time"
"github.com/google/gopacket/layers"
"github.com/rs/zerolog"
"golang.org/x/net/icmp"
"golang.org/x/net/ipv4"
"github.com/cloudflare/cloudflared/ingress"
"github.com/cloudflare/cloudflared/packet"
v3 "github.com/cloudflare/cloudflared/quic/v3"
)
type noopEyeball struct{}
func (noopEyeball) SendUDPSessionDatagram(datagram []byte) error {
return nil
type noopEyeball struct {
connID uint8
}
func (noopEyeball) Serve(ctx context.Context) error { return nil }
func (n noopEyeball) ID() uint8 { return n.connID }
func (noopEyeball) SendUDPSessionDatagram(datagram []byte) error { return nil }
func (noopEyeball) SendUDPSessionResponse(id v3.RequestID, resp v3.SessionRegistrationResp) error {
return nil
}
func (noopEyeball) SendICMPPacket(icmp *packet.ICMP) error { return nil }
func (noopEyeball) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error { return nil }
type mockEyeball struct {
connID uint8
// datagram sent via SendUDPSessionDatagram
recvData chan []byte
// responses sent via SendUDPSessionResponse
@@ -39,6 +47,7 @@ type mockEyeball struct {
func newMockEyeball() mockEyeball {
return mockEyeball{
connID: 0,
recvData: make(chan []byte, 1),
recvResp: make(chan struct {
id v3.RequestID
@@ -47,6 +56,9 @@ func newMockEyeball() mockEyeball {
}
}
func (mockEyeball) Serve(ctx context.Context) error { return nil }
func (m *mockEyeball) ID() uint8 { return m.connID }
func (m *mockEyeball) SendUDPSessionDatagram(datagram []byte) error {
b := make([]byte, len(datagram))
copy(b, datagram)
@@ -64,9 +76,14 @@ func (m *mockEyeball) SendUDPSessionResponse(id v3.RequestID, resp v3.SessionReg
return nil
}
func (m *mockEyeball) SendICMPPacket(icmp *packet.ICMP) error { return nil }
func (m *mockEyeball) SendICMPTTLExceed(icmp *packet.ICMP, rawPacket packet.RawPacket) error {
return nil
}
func TestDatagramConn_New(t *testing.T) {
log := zerolog.Nop()
conn := v3.NewDatagramConn(newMockQuicConn(), v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(newMockQuicConn(), v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
if conn == nil {
t.Fatal("expected valid connection")
}
@@ -75,7 +92,7 @@ func TestDatagramConn_New(t *testing.T) {
func TestDatagramConn_SendUDPSessionDatagram(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
payload := []byte{0xef, 0xef}
conn.SendUDPSessionDatagram(payload)
@@ -88,7 +105,7 @@ func TestDatagramConn_SendUDPSessionDatagram(t *testing.T) {
func TestDatagramConn_SendUDPSessionResponse(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
conn.SendUDPSessionResponse(testRequestID, v3.ResponseDestinationUnreachable)
resp := <-quic.recv
@@ -109,7 +126,7 @@ func TestDatagramConn_SendUDPSessionResponse(t *testing.T) {
func TestDatagramConnServe_ApplicationClosed(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
@@ -125,7 +142,7 @@ func TestDatagramConnServe_ConnectionClosed(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
quic.ctx = ctx
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
err := conn.Serve(context.Background())
if !errors.Is(err, context.DeadlineExceeded) {
@@ -136,7 +153,7 @@ func TestDatagramConnServe_ConnectionClosed(t *testing.T) {
func TestDatagramConnServe_ReceiveDatagramError(t *testing.T) {
log := zerolog.Nop()
quic := &mockQuicConnReadError{err: net.ErrClosed}
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&log, ingress.DialUDPAddrPort), &log)
conn := v3.NewDatagramConn(quic, v3.NewSessionManager(&noopMetrics{}, &log, ingress.DialUDPAddrPort), &noopICMPRouter{}, 0, &noopMetrics{}, &log)
err := conn.Serve(context.Background())
if !errors.Is(err, net.ErrClosed) {
@@ -171,7 +188,7 @@ func TestDatagramConnServe_ErrorDatagramTypes(t *testing.T) {
log := zerolog.New(logOutput)
quic := newMockQuicConn()
quic.send <- test.input
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, &log)
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
@@ -212,7 +229,7 @@ func TestDatagramConnServe_RegisterSession_SessionManagerError(t *testing.T) {
quic := newMockQuicConn()
expectedErr := errors.New("unable to register session")
sessionManager := mockSessionManager{expectedRegErr: expectedErr}
conn := v3.NewDatagramConn(quic, &sessionManager, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -234,19 +251,12 @@ func TestDatagramConnServe_RegisterSession_SessionManagerError(t *testing.T) {
t.Fatal(err)
}
if resp.RequestID != testRequestID && resp.ResponseType != v3.ResponseUnableToBindSocket {
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseUnableToBindSocket {
t.Fatalf("expected registration response failure")
}
// Cancel the muxer Serve context and make sure it closes with the expected error
cancel(expectedContextCanceled)
err = <-done
if !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), expectedContextCanceled) {
t.Fatal(err)
}
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe(t *testing.T) {
@@ -254,7 +264,7 @@ func TestDatagramConnServe(t *testing.T) {
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -276,7 +286,7 @@ func TestDatagramConnServe(t *testing.T) {
t.Fatal(err)
}
if resp.RequestID != testRequestID && resp.ResponseType != v3.ResponseOk {
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseOk {
t.Fatalf("expected registration response ok")
}
@@ -291,21 +301,160 @@ func TestDatagramConnServe(t *testing.T) {
}
// Cancel the muxer Serve context and make sure it closes with the expected error
cancel(expectedContextCanceled)
err = <-done
if !errors.Is(err, context.Canceled) {
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_RegisterTwice(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
// Send new session registration
datagram := newRegisterSessionDatagram(testRequestID)
quic.send <- datagram
// Wait for session registration response with success
datagram = <-quic.recv
var resp v3.UDPSessionRegistrationResponseDatagram
err := resp.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), expectedContextCanceled) {
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseOk {
t.Fatalf("expected registration response ok")
}
// Set the session manager to return already registered
sessionManager.expectedRegErr = v3.ErrSessionAlreadyRegistered
// Send the registration again as if we didn't receive it at the edge
datagram = newRegisterSessionDatagram(testRequestID)
quic.send <- datagram
// Wait for session registration response with success
datagram = <-quic.recv
err = resp.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseOk {
t.Fatalf("expected registration response ok")
}
// We expect the session to be served
timer := time.NewTimer(15 * time.Second)
defer timer.Stop()
select {
case <-session.served:
break
case <-timer.C:
t.Fatalf("expected session serve to be called")
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_MigrateConnection(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
quic2 := newMockQuicConn()
conn2 := v3.NewDatagramConn(quic2, &sessionManager, &noopICMPRouter{}, 1, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
ctx2, cancel2 := context.WithCancelCause(context.Background())
defer cancel2(errors.New("other error"))
done2 := make(chan error, 1)
go func() {
done2 <- conn2.Serve(ctx2)
}()
// Send new session registration
datagram := newRegisterSessionDatagram(testRequestID)
quic.send <- datagram
// Wait for session registration response with success
datagram = <-quic.recv
var resp v3.UDPSessionRegistrationResponseDatagram
err := resp.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseOk {
t.Fatalf("expected registration response ok")
}
// Set the session manager to return already registered to another connection
sessionManager.expectedRegErr = v3.ErrSessionBoundToOtherConn
// Send the registration again as if we didn't receive it at the edge for a new connection
datagram = newRegisterSessionDatagram(testRequestID)
quic2.send <- datagram
// Wait for session registration response with success
datagram = <-quic2.recv
err = resp.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
if resp.RequestID != testRequestID || resp.ResponseType != v3.ResponseOk {
t.Fatalf("expected registration response ok")
}
// We expect the session to be served
timer := time.NewTimer(15 * time.Second)
defer timer.Stop()
select {
case <-session.served:
break
case <-timer.C:
t.Fatalf("expected session serve to be called")
}
// Expect session to be migrated
select {
case id := <-session.migrated:
if id != conn2.ID() {
t.Fatalf("expected session to be migrated to connection 2")
}
case <-timer.C:
t.Fatalf("expected session migration to be called")
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
// Cancel the second muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx2, done2, cancel2)
}
func TestDatagramConnServe_Payload_GetSessionError(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
// mockSessionManager will return the ErrSessionNotFound for any session attempting to be queried by the muxer
sessionManager := mockSessionManager{session: nil, expectedGetErr: v3.ErrSessionNotFound}
conn := v3.NewDatagramConn(quic, &sessionManager, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -319,15 +468,13 @@ func TestDatagramConnServe_Payload_GetSessionError(t *testing.T) {
datagram := newSessionPayloadDatagram(testRequestID, []byte{0xef, 0xef})
quic.send <- datagram
// Since the muxer should eventually discard a failed registration request, there is no side-effect
// that the registration was failed beyond the muxer accepting the registration request. As such, the
// test can only ensure that the quic.send channel was consumed and that the muxer closes normally
// afterwards with the expected context cancelled trigger.
// Cancel the muxer Serve context and make sure it closes with the expected error
cancel(expectedContextCanceled)
err := <-done
if !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), expectedContextCanceled) {
t.Fatal(err)
}
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_Payload(t *testing.T) {
@@ -335,7 +482,7 @@ func TestDatagramConnServe_Payload(t *testing.T) {
quic := newMockQuicConn()
session := newMockSession()
sessionManager := mockSessionManager{session: &session}
conn := v3.NewDatagramConn(quic, &sessionManager, &log)
conn := v3.NewDatagramConn(quic, &sessionManager, &noopICMPRouter{}, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
@@ -357,14 +504,117 @@ func TestDatagramConnServe_Payload(t *testing.T) {
}
// Cancel the muxer Serve context and make sure it closes with the expected error
cancel(expectedContextCanceled)
err := <-done
if !errors.Is(err, context.Canceled) {
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_ICMPDatagram_TTLDecremented(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
router := newMockICMPRouter()
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, router, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
// Send new ICMP Echo request
expectedICMP := &packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("192.168.1.1"),
Dst: netip.MustParseAddr("10.0.0.1"),
Protocol: layers.IPProtocolICMPv4,
TTL: 20,
},
Message: &icmp.Message{
Type: ipv4.ICMPTypeEcho,
Code: 0,
Body: &icmp.Echo{
ID: 25821,
Seq: 58129,
Data: []byte("test ttl=0"),
},
},
}
datagram := newICMPDatagram(expectedICMP)
quic.send <- datagram
// Router should receive the packet
actualICMP := <-router.recv
assertICMPEqual(t, expectedICMP, actualICMP)
if expectedICMP.TTL-1 != actualICMP.TTL {
t.Fatalf("TTL should be decremented by one before sending to origin: %d, %d", expectedICMP.TTL, actualICMP.TTL)
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func TestDatagramConnServe_ICMPDatagram_TTLExceeded(t *testing.T) {
log := zerolog.Nop()
quic := newMockQuicConn()
router := newMockICMPRouter()
conn := v3.NewDatagramConn(quic, &mockSessionManager{}, router, 0, &noopMetrics{}, &log)
// Setup the muxer
ctx, cancel := context.WithCancelCause(context.Background())
defer cancel(errors.New("other error"))
done := make(chan error, 1)
go func() {
done <- conn.Serve(ctx)
}()
// Send new ICMP Echo request
expectedICMP := &packet.ICMP{
IP: &packet.IP{
Src: netip.MustParseAddr("192.168.1.1"),
Dst: netip.MustParseAddr("10.0.0.1"),
Protocol: layers.IPProtocolICMPv4,
TTL: 0,
},
Message: &icmp.Message{
Type: ipv4.ICMPTypeEcho,
Code: 0,
Body: &icmp.Echo{
ID: 25821,
Seq: 58129,
Data: []byte("test ttl=0"),
},
},
}
datagram := newICMPDatagram(expectedICMP)
quic.send <- datagram
// Origin should not recieve a packet
select {
case <-router.recv:
t.Fatalf("TTL should be expired and no origin ICMP sent")
default:
}
// Eyeball should receive the packet
datagram = <-quic.recv
icmpDatagram := v3.ICMPDatagram{}
err := icmpDatagram.UnmarshalBinary(datagram)
if err != nil {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), expectedContextCanceled) {
decoder := packet.NewICMPDecoder()
ttlExpiredICMP, err := decoder.Decode(packet.RawPacket{Data: icmpDatagram.Payload})
if err != nil {
t.Fatal(err)
}
// Packet should be a TTL Exceeded ICMP
if ttlExpiredICMP.TTL != packet.DefaultTTL || ttlExpiredICMP.Message.Type != ipv4.ICMPTypeTimeExceeded {
t.Fatalf("ICMP packet should be a ICMP Exceeded: %+v", ttlExpiredICMP)
}
// Cancel the muxer Serve context and make sure it closes with the expected error
assertContextClosed(t, ctx, done, cancel)
}
func newRegisterSessionDatagram(id v3.RequestID) []byte {
@@ -402,6 +652,34 @@ func newSessionPayloadDatagram(id v3.RequestID, payload []byte) []byte {
return datagram
}
func newICMPDatagram(pk *packet.ICMP) []byte {
encoder := packet.NewEncoder()
rawPacket, err := encoder.Encode(pk)
if err != nil {
panic(err)
}
datagram := v3.ICMPDatagram{
Payload: rawPacket.Data,
}
payload, err := datagram.MarshalBinary()
if err != nil {
panic(err)
}
return payload
}
// Cancel the provided context and make sure it closes with the expected cancellation error
func assertContextClosed(t *testing.T, ctx context.Context, done <-chan error, cancel context.CancelCauseFunc) {
cancel(expectedContextCanceled)
err := <-done
if !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
if !errors.Is(context.Cause(ctx), expectedContextCanceled) {
t.Fatal(err)
}
}
type mockQuicConn struct {
ctx context.Context
send chan []byte
@@ -454,7 +732,7 @@ type mockSessionManager struct {
expectedGetErr error
}
func (m *mockSessionManager) RegisterSession(request *v3.UDPSessionRegistrationDatagram, conn v3.DatagramWriter) (v3.Session, error) {
func (m *mockSessionManager) RegisterSession(request *v3.UDPSessionRegistrationDatagram, conn v3.DatagramConn) (v3.Session, error) {
return m.session, m.expectedRegErr
}
@@ -465,20 +743,27 @@ func (m *mockSessionManager) GetSession(requestID v3.RequestID) (v3.Session, err
func (m *mockSessionManager) UnregisterSession(requestID v3.RequestID) {}
type mockSession struct {
served chan struct{}
recv chan []byte
served chan struct{}
migrated chan uint8
recv chan []byte
}
func newMockSession() mockSession {
return mockSession{
served: make(chan struct{}),
recv: make(chan []byte, 1),
served: make(chan struct{}),
migrated: make(chan uint8, 2),
recv: make(chan []byte, 1),
}
}
func (m *mockSession) ID() v3.RequestID {
return testRequestID
func (m *mockSession) ID() v3.RequestID { return testRequestID }
func (m *mockSession) RemoteAddr() net.Addr { return testOriginAddr }
func (m *mockSession) LocalAddr() net.Addr { return testLocalAddr }
func (m *mockSession) ConnectionID() uint8 { return 0 }
func (m *mockSession) Migrate(conn v3.DatagramConn, ctx context.Context, log *zerolog.Logger) {
m.migrated <- conn.ID()
}
func (m *mockSession) ResetIdleTimer() {}
func (m *mockSession) Serve(ctx context.Context) error {
close(m.served)
+108 -26
View File
@@ -7,6 +7,7 @@ import (
"io"
"net"
"sync"
"sync/atomic"
"time"
"github.com/rs/zerolog"
@@ -20,10 +21,13 @@ const (
// read 1500 bytes from the origin, we limit the amount of bytes to be proxied to less than
// this value (maxDatagramPayloadLen).
maxOriginUDPPacketSize = 1500
logFlowID = "flowID"
logPacketSizeKey = "packetSize"
)
// SessionCloseErr indicates that the session's Close method was called.
var SessionCloseErr error = errors.New("session was closed")
var SessionCloseErr error = errors.New("flow was closed directly")
// SessionIdleErr is returned when the session was closed because there was no communication
// in either direction over the session for the timeout period.
@@ -32,7 +36,7 @@ type SessionIdleErr struct {
}
func (e SessionIdleErr) Error() string {
return fmt.Sprintf("session idle for %v", e.timeout)
return fmt.Sprintf("flow was idle for %v", e.timeout)
}
func (e SessionIdleErr) Is(target error) bool {
@@ -47,6 +51,11 @@ func newSessionIdleErr(timeout time.Duration) error {
type Session interface {
io.WriteCloser
ID() RequestID
ConnectionID() uint8
RemoteAddr() net.Addr
LocalAddr() net.Addr
ResetIdleTimer()
Migrate(eyeball DatagramConn, ctx context.Context, logger *zerolog.Logger)
// Serve starts the event loop for processing UDP packets
Serve(ctx context.Context) error
}
@@ -55,31 +64,91 @@ type session struct {
id RequestID
closeAfterIdle time.Duration
origin io.ReadWriteCloser
eyeball DatagramWriter
originAddr net.Addr
localAddr net.Addr
eyeball atomic.Pointer[DatagramConn]
// activeAtChan is used to communicate the last read/write time
activeAtChan chan time.Time
closeChan chan error
contextChan chan context.Context
metrics Metrics
log *zerolog.Logger
// A special close function that we wrap with sync.Once to make sure it is only called once
closeFn func() error
}
func NewSession(id RequestID, closeAfterIdle time.Duration, origin io.ReadWriteCloser, eyeball DatagramWriter, log *zerolog.Logger) Session {
return &session{
func NewSession(
id RequestID,
closeAfterIdle time.Duration,
origin io.ReadWriteCloser,
originAddr net.Addr,
localAddr net.Addr,
eyeball DatagramConn,
metrics Metrics,
log *zerolog.Logger,
) Session {
logger := log.With().Str(logFlowID, id.String()).Logger()
closeChan := make(chan error, 1)
session := &session{
id: id,
closeAfterIdle: closeAfterIdle,
origin: origin,
eyeball: eyeball,
originAddr: originAddr,
localAddr: localAddr,
eyeball: atomic.Pointer[DatagramConn]{},
// activeAtChan has low capacity. It can be full when there are many concurrent read/write. markActive() will
// drop instead of blocking because last active time only needs to be an approximation
activeAtChan: make(chan time.Time, 1),
closeChan: make(chan error, 1),
log: log,
closeChan: closeChan,
// contextChan is an unbounded channel to help enforce one active migration of a session at a time.
contextChan: make(chan context.Context),
metrics: metrics,
log: &logger,
closeFn: sync.OnceValue(func() error {
// We don't want to block on sending to the close channel if it is already full
select {
case closeChan <- SessionCloseErr:
default:
}
return origin.Close()
}),
}
session.eyeball.Store(&eyeball)
return session
}
func (s *session) ID() RequestID {
return s.id
}
func (s *session) RemoteAddr() net.Addr {
return s.originAddr
}
func (s *session) LocalAddr() net.Addr {
return s.localAddr
}
func (s *session) ConnectionID() uint8 {
eyeball := *(s.eyeball.Load())
return eyeball.ID()
}
func (s *session) Migrate(eyeball DatagramConn, ctx context.Context, logger *zerolog.Logger) {
current := *(s.eyeball.Load())
// Only migrate if the connection ids are different.
if current.ID() != eyeball.ID() {
s.eyeball.Store(&eyeball)
s.contextChan <- ctx
log := logger.With().Str(logFlowID, s.id.String()).Logger()
s.log = &log
}
// The session is already running so we want to restart the idle timeout since no proxied packets have come down yet.
s.markActive()
s.metrics.MigrateFlow()
}
func (s *session) Serve(ctx context.Context) error {
go func() {
// QUIC implementation copies data to another buffer before returning https://github.com/quic-go/quic-go/blob/v0.24.0/session.go#L1967-L1975
@@ -92,24 +161,29 @@ func (s *session) Serve(ctx context.Context) error {
for {
// Read from the origin UDP socket
n, err := s.origin.Read(readBuffer[DatagramPayloadHeaderLen:])
if errors.Is(err, net.ErrClosed) || errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
s.log.Debug().Msg("Session (origin) connection closed")
}
if err != nil {
if errors.Is(err, io.EOF) ||
errors.Is(err, io.ErrUnexpectedEOF) {
s.log.Debug().Msgf("flow (origin) connection closed: %v", err)
}
s.closeChan <- err
return
}
if n < 0 {
s.log.Warn().Int("packetSize", n).Msg("Session (origin) packet read was negative and was dropped")
s.log.Warn().Int(logPacketSizeKey, n).Msg("flow (origin) packet read was negative and was dropped")
continue
}
if n > maxDatagramPayloadLen {
s.log.Error().Int("packetSize", n).Msg("Session (origin) packet read was too large and was dropped")
s.metrics.PayloadTooLarge()
s.log.Error().Int(logPacketSizeKey, n).Msg("flow (origin) packet read was too large and was dropped")
continue
}
// We need to synchronize on the eyeball in-case that the connection was migrated. This should be rarely a point
// of lock contention, as a migration can only happen during startup of a session before traffic flow.
eyeball := *(s.eyeball.Load())
// Sending a packet to the session does block on the [quic.Connection], however, this is okay because it
// will cause back-pressure to the kernel buffer if the writes are not fast enough to the edge.
err = s.eyeball.SendUDPSessionDatagram(readBuffer[:DatagramPayloadHeaderLen+n])
err = eyeball.SendUDPSessionDatagram(readBuffer[:DatagramPayloadHeaderLen+n])
if err != nil {
s.closeChan <- err
return
@@ -124,12 +198,12 @@ func (s *session) Serve(ctx context.Context) error {
func (s *session) Write(payload []byte) (n int, err error) {
n, err = s.origin.Write(payload)
if err != nil {
s.log.Err(err).Msg("Failed to write payload to session (remote)")
s.log.Err(err).Msg("failed to write payload to flow (remote)")
return n, err
}
// Write must return a non-nil error if it returns n < len(p). https://pkg.go.dev/io#Writer
if n < len(payload) {
s.log.Err(io.ErrShortWrite).Msg("Failed to write the full payload to session (remote)")
s.log.Err(io.ErrShortWrite).Msg("failed to write the full payload to flow (remote)")
return n, io.ErrShortWrite
}
// Mark the session as active since we proxied a packet to the origin.
@@ -137,6 +211,14 @@ func (s *session) Write(payload []byte) (n int, err error) {
return n, err
}
// ResetIdleTimer will restart the current idle timer.
//
// This public method is used to allow operators of sessions the ability to extend the session using information that is
// known external to the session itself.
func (s *session) ResetIdleTimer() {
s.markActive()
}
// Sends the last active time to the idle checker loop without blocking. activeAtChan will only be full when there
// are many concurrent read/write. It is fine to lose some precision
func (s *session) markActive() {
@@ -148,17 +230,11 @@ func (s *session) markActive() {
func (s *session) Close() error {
// Make sure that we only close the origin connection once
return sync.OnceValue(func() error {
// We don't want to block on sending to the close channel if it is already full
select {
case s.closeChan <- SessionCloseErr:
default:
}
return s.origin.Close()
})()
return s.closeFn()
}
func (s *session) waitForCloseCondition(ctx context.Context, closeAfterIdle time.Duration) error {
connCtx := ctx
// Closing the session at the end cancels read so Serve() can return
defer s.Close()
if closeAfterIdle == 0 {
@@ -171,8 +247,14 @@ func (s *session) waitForCloseCondition(ctx context.Context, closeAfterIdle time
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-connCtx.Done():
return connCtx.Err()
case newContext := <-s.contextChan:
// During migration of a session, we need to make sure that the context of the new connection is used instead
// of the old connection context. This will ensure that when the old connection goes away, this session will
// still be active on the existing connection.
connCtx = newContext
continue
case reason := <-s.closeChan:
return reason
case <-checkIdleTimer.C:
+136 -10
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"net"
"net/netip"
"slices"
"sync/atomic"
"testing"
@@ -14,11 +15,16 @@ import (
v3 "github.com/cloudflare/cloudflared/quic/v3"
)
var expectedContextCanceled = errors.New("expected context canceled")
var (
expectedContextCanceled = errors.New("expected context canceled")
testOriginAddr = net.UDPAddrFromAddrPort(netip.MustParseAddrPort("127.0.0.1:0"))
testLocalAddr = net.UDPAddrFromAddrPort(netip.MustParseAddrPort("127.0.0.1:0"))
)
func TestSessionNew(t *testing.T) {
log := zerolog.Nop()
session := v3.NewSession(testRequestID, 5*time.Second, nil, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, 5*time.Second, nil, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
if testRequestID != session.ID() {
t.Fatalf("session id doesn't match: %s != %s", testRequestID, session.ID())
}
@@ -27,7 +33,7 @@ func TestSessionNew(t *testing.T) {
func testSessionWrite(t *testing.T, payload []byte) {
log := zerolog.Nop()
origin := newTestOrigin(makePayload(1280))
session := v3.NewSession(testRequestID, 5*time.Second, &origin, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, 5*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
n, err := session.Write(payload)
if err != nil {
t.Fatal(err)
@@ -64,7 +70,7 @@ func testSessionServe_Origin(t *testing.T, payload []byte) {
log := zerolog.Nop()
eyeball := newMockEyeball()
origin := newTestOrigin(payload)
session := v3.NewSession(testRequestID, 3*time.Second, &origin, &eyeball, &log)
session := v3.NewSession(testRequestID, 3*time.Second, &origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
ctx, cancel := context.WithCancelCause(context.Background())
@@ -103,7 +109,7 @@ func TestSessionServe_OriginTooLarge(t *testing.T) {
eyeball := newMockEyeball()
payload := makePayload(1281)
origin := newTestOrigin(payload)
session := v3.NewSession(testRequestID, 2*time.Second, &origin, &eyeball, &log)
session := v3.NewSession(testRequestID, 2*time.Second, &origin, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
done := make(chan error)
@@ -123,10 +129,125 @@ func TestSessionServe_OriginTooLarge(t *testing.T) {
}
}
func TestSessionServe_Migrate(t *testing.T) {
log := zerolog.Nop()
eyeball := newMockEyeball()
pipe1, pipe2 := net.Pipe()
session := v3.NewSession(testRequestID, 2*time.Second, pipe2, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
done := make(chan error)
eyeball1Ctx, cancel := context.WithCancelCause(context.Background())
go func() {
done <- session.Serve(eyeball1Ctx)
}()
// Migrate the session to a new connection before origin sends data
eyeball2 := newMockEyeball()
eyeball2.connID = 1
eyeball2Ctx := context.Background()
session.Migrate(&eyeball2, eyeball2Ctx, &log)
// Cancel the origin eyeball context; this should not cancel the session
contextCancelErr := errors.New("context canceled for first eyeball connection")
cancel(contextCancelErr)
select {
case <-done:
t.Fatalf("expected session to still be running")
default:
}
if context.Cause(eyeball1Ctx) != contextCancelErr {
t.Fatalf("first eyeball context should be cancelled manually: %+v", context.Cause(eyeball1Ctx))
}
// Origin sends data
payload2 := []byte{0xde}
pipe1.Write(payload2)
// Expect write to eyeball2
data := <-eyeball2.recvData
if len(data) <= 17 || !slices.Equal(payload2, data[17:]) {
t.Fatalf("expected data to write to eyeball2 after migration: %+v", data)
}
select {
case data := <-eyeball.recvData:
t.Fatalf("expected no data to write to eyeball1 after migration: %+v", data)
default:
}
err := <-done
if !errors.Is(err, v3.SessionIdleErr{}) {
t.Error(err)
}
if eyeball2Ctx.Err() != nil {
t.Fatalf("second eyeball context should be not be cancelled")
}
}
func TestSessionServe_Migrate_CloseContext2(t *testing.T) {
log := zerolog.Nop()
eyeball := newMockEyeball()
pipe1, pipe2 := net.Pipe()
session := v3.NewSession(testRequestID, 2*time.Second, pipe2, testOriginAddr, testLocalAddr, &eyeball, &noopMetrics{}, &log)
defer session.Close()
done := make(chan error)
eyeball1Ctx, cancel := context.WithCancelCause(context.Background())
go func() {
done <- session.Serve(eyeball1Ctx)
}()
// Migrate the session to a new connection before origin sends data
eyeball2 := newMockEyeball()
eyeball2.connID = 1
eyeball2Ctx, cancel2 := context.WithCancelCause(context.Background())
session.Migrate(&eyeball2, eyeball2Ctx, &log)
// Cancel the origin eyeball context; this should not cancel the session
contextCancelErr := errors.New("context canceled for first eyeball connection")
cancel(contextCancelErr)
select {
case <-done:
t.Fatalf("expected session to still be running")
default:
}
if context.Cause(eyeball1Ctx) != contextCancelErr {
t.Fatalf("first eyeball context should be cancelled manually: %+v", context.Cause(eyeball1Ctx))
}
// Origin sends data
payload2 := []byte{0xde}
pipe1.Write(payload2)
// Expect write to eyeball2
data := <-eyeball2.recvData
if len(data) <= 17 || !slices.Equal(payload2, data[17:]) {
t.Fatalf("expected data to write to eyeball2 after migration: %+v", data)
}
select {
case data := <-eyeball.recvData:
t.Fatalf("expected no data to write to eyeball1 after migration: %+v", data)
default:
}
// Close the connection2 context manually
contextCancel2Err := errors.New("context canceled for second eyeball connection")
cancel2(contextCancel2Err)
err := <-done
if err != context.Canceled {
t.Fatalf("session Serve should be done: %+v", err)
}
if context.Cause(eyeball2Ctx) != contextCancel2Err {
t.Fatalf("second eyeball context should have been cancelled manually: %+v", context.Cause(eyeball2Ctx))
}
}
func TestSessionClose_Multiple(t *testing.T) {
log := zerolog.Nop()
origin := newTestOrigin(makePayload(128))
session := v3.NewSession(testRequestID, 5*time.Second, &origin, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, 5*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
err := session.Close()
if err != nil {
t.Fatal(err)
@@ -134,18 +255,23 @@ func TestSessionClose_Multiple(t *testing.T) {
if !origin.closed.Load() {
t.Fatal("origin wasn't closed")
}
// Reset the closed status to make sure it isn't closed again
origin.closed.Store(false)
// subsequent closes shouldn't call close again or cause any errors
err = session.Close()
if err != nil {
t.Fatal(err)
}
if origin.closed.Load() {
t.Fatal("origin was incorrectly closed twice")
}
}
func TestSessionServe_IdleTimeout(t *testing.T) {
log := zerolog.Nop()
origin := newTestIdleOrigin(10 * time.Second) // Make idle time longer than closeAfterIdle
closeAfterIdle := 2 * time.Second
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
err := session.Serve(context.Background())
if !errors.Is(err, v3.SessionIdleErr{}) {
t.Fatal(err)
@@ -167,7 +293,7 @@ func TestSessionServe_ParentContextCanceled(t *testing.T) {
origin := newTestIdleOrigin(10 * time.Second)
closeAfterIdle := 10 * time.Second
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, closeAfterIdle, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
err := session.Serve(ctx)
@@ -188,7 +314,7 @@ func TestSessionServe_ParentContextCanceled(t *testing.T) {
func TestSessionServe_ReadErrors(t *testing.T) {
log := zerolog.Nop()
origin := newTestErrOrigin(net.ErrClosed, nil)
session := v3.NewSession(testRequestID, 30*time.Second, &origin, &noopEyeball{}, &log)
session := v3.NewSession(testRequestID, 30*time.Second, &origin, testOriginAddr, testLocalAddr, &noopEyeball{}, &noopMetrics{}, &log)
err := session.Serve(context.Background())
if !errors.Is(err, net.ErrClosed) {
t.Fatal(err)
@@ -249,7 +375,7 @@ func newTestIdleOrigin(d time.Duration) testIdleOrigin {
func (o *testIdleOrigin) Read(p []byte) (n int, err error) {
time.Sleep(o.duration)
return 0, nil
return -1, nil
}
func (o *testIdleOrigin) Write(p []byte) (n int, err error) {
+1 -1
View File
@@ -113,7 +113,7 @@ class PkgCreator:
def create_rpm_pkgs(self, artifacts_path, gpg_key_name):
self._setup_rpm_pkg_directories(artifacts_path, gpg_key_name)
p = Popen(["createrepo", "./rpm"], stdout=PIPE, stderr=PIPE)
p = Popen(["createrepo_c", "./rpm"], stdout=PIPE, stderr=PIPE)
out, err = p.communicate()
if p.returncode != 0:
print(f"create rpm_pkgs result => {out}, {err}")
+6 -3
View File
@@ -7,6 +7,7 @@ import (
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/quic-go/quic-go"
"github.com/rs/zerolog"
@@ -82,12 +83,14 @@ func NewSupervisor(config *TunnelConfig, orchestrator *orchestration.Orchestrato
edgeAddrHandler := NewIPAddrFallback(config.MaxEdgeAddrRetries)
edgeBindAddr := config.EdgeBindAddr
sessionManager := v3.NewSessionManager(config.Log, ingress.DialUDPAddrPort)
datagramMetrics := v3.NewMetrics(prometheus.DefaultRegisterer)
sessionManager := v3.NewSessionManager(datagramMetrics, config.Log, ingress.DialUDPAddrPort)
edgeTunnelServer := EdgeTunnelServer{
config: config,
orchestrator: orchestrator,
sessionManager: sessionManager,
datagramMetrics: datagramMetrics,
edgeAddrs: edgeIPs,
edgeAddrHandler: edgeAddrHandler,
edgeBindAddr: edgeBindAddr,
@@ -116,9 +119,9 @@ func (s *Supervisor) Run(
ctx context.Context,
connectedSignal *signal.Signal,
) error {
if s.config.PacketConfig != nil {
if s.config.ICMPRouterServer != nil {
go func() {
if err := s.config.PacketConfig.ICMPRouter.Serve(ctx); err != nil {
if err := s.config.ICMPRouterServer.Serve(ctx); err != nil {
if errors.Is(err, net.ErrClosed) {
s.log.Logger().Info().Err(err).Msg("icmp router terminated")
} else {
+7 -2
View File
@@ -63,7 +63,7 @@ type TunnelConfig struct {
NamedTunnel *connection.TunnelProperties
ProtocolSelector connection.ProtocolSelector
EdgeTLSConfigs map[connection.Protocol]*tls.Config
PacketConfig *ingress.GlobalRouterConfig
ICMPRouterServer ingress.ICMPRouterServer
RPCTimeout time.Duration
WriteStreamTimeout time.Duration
@@ -176,6 +176,7 @@ type EdgeTunnelServer struct {
config *TunnelConfig
orchestrator *orchestration.Orchestrator
sessionManager v3.SessionManager
datagramMetrics v3.Metrics
edgeAddrHandler EdgeAddrHandler
edgeAddrs *edgediscovery.Edge
edgeBindAddr net.IP
@@ -606,13 +607,17 @@ func (e *EdgeTunnelServer) serveQUIC(
ctx,
conn,
e.sessionManager,
e.config.ICMPRouterServer,
connIndex,
e.datagramMetrics,
connLogger.Logger(),
)
} else {
datagramSessionManager = connection.NewDatagramV2Connection(
ctx,
conn,
e.config.PacketConfig,
e.config.ICMPRouterServer,
connIndex,
e.config.RPCTimeout,
e.config.WriteStreamTimeout,
connLogger.Logger(),
+6
View File
@@ -72,3 +72,9 @@ func TestAddingSpansWithNilMap(t *testing.T) {
// a panic shouldn't occur
tr.AddSpans(nil)
}
func FuzzNewIdentity(f *testing.F) {
f.Fuzz(func(t *testing.T, trace string) {
_, _ = NewIdentity(trace)
})
}
+41 -18
View File
@@ -1,6 +1,7 @@
package tunnelstate
import (
"net"
"sync"
"github.com/rs/zerolog"
@@ -9,54 +10,58 @@ import (
)
type ConnTracker struct {
sync.RWMutex
mutex sync.RWMutex
// int is the connection Index
connectionInfo map[uint8]ConnectionInfo
log *zerolog.Logger
}
type ConnectionInfo struct {
IsConnected bool
Protocol connection.Protocol
IsConnected bool `json:"isConnected,omitempty"`
Protocol connection.Protocol `json:"protocol,omitempty"`
EdgeAddress net.IP `json:"edgeAddress,omitempty"`
}
func NewConnTracker(log *zerolog.Logger) *ConnTracker {
// Convinience struct to extend the connection with its index.
type IndexedConnectionInfo struct {
ConnectionInfo
Index uint8 `json:"index,omitempty"`
}
func NewConnTracker(
log *zerolog.Logger,
) *ConnTracker {
return &ConnTracker{
connectionInfo: make(map[uint8]ConnectionInfo, 0),
log: log,
}
}
func MockedConnTracker(mocked map[uint8]ConnectionInfo) *ConnTracker {
return &ConnTracker{
connectionInfo: mocked,
}
}
func (ct *ConnTracker) OnTunnelEvent(c connection.Event) {
switch c.EventType {
case connection.Connected:
ct.Lock()
ct.mutex.Lock()
ci := ConnectionInfo{
IsConnected: true,
Protocol: c.Protocol,
EdgeAddress: c.EdgeAddress,
}
ct.connectionInfo[c.Index] = ci
ct.Unlock()
ct.mutex.Unlock()
case connection.Disconnected, connection.Reconnecting, connection.RegisteringTunnel, connection.Unregistering:
ct.Lock()
ct.mutex.Lock()
ci := ct.connectionInfo[c.Index]
ci.IsConnected = false
ct.connectionInfo[c.Index] = ci
ct.Unlock()
ct.mutex.Unlock()
default:
ct.log.Error().Msgf("Unknown connection event case %v", c)
}
}
func (ct *ConnTracker) CountActiveConns() uint {
ct.RLock()
defer ct.RUnlock()
ct.mutex.RLock()
defer ct.mutex.RUnlock()
active := uint(0)
for _, ci := range ct.connectionInfo {
if ci.IsConnected {
@@ -69,8 +74,8 @@ func (ct *ConnTracker) CountActiveConns() uint {
// HasConnectedWith checks if we've ever had a successful connection to the edge
// with said protocol.
func (ct *ConnTracker) HasConnectedWith(protocol connection.Protocol) bool {
ct.RLock()
defer ct.RUnlock()
ct.mutex.RLock()
defer ct.mutex.RUnlock()
for _, ci := range ct.connectionInfo {
if ci.Protocol == protocol {
return true
@@ -78,3 +83,21 @@ func (ct *ConnTracker) HasConnectedWith(protocol connection.Protocol) bool {
}
return false
}
// Returns the connection information iff it is connected this
// also leverages the [IndexedConnectionInfo] to also provide the connection index
func (ct *ConnTracker) GetActiveConnections() []IndexedConnectionInfo {
ct.mutex.RLock()
defer ct.mutex.RUnlock()
connections := make([]IndexedConnectionInfo, 0)
for key, value := range ct.connectionInfo {
if value.IsConnected {
info := IndexedConnectionInfo{value, key}
connections = append(connections, info)
}
}
return connections
}
+7
View File
@@ -197,3 +197,10 @@ func createSecureMockServerAndClient(handler http.Handler) (*httptest.Server, *h
return server, client, nil
}
func FuzzNewAccessValidator(f *testing.F) {
f.Fuzz(func(t *testing.T, domain string, issuer string, applicationAUD string) {
ctx := context.Background()
_, _ = NewAccessValidator(ctx, domain, issuer, applicationAUD)
})
}