Files
Naomi Kramer b923c39e2d Update configuration (#78)
* Update on config structure, functionality, and tests

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Extend subnet type to read/write from db, update tests

Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>

* updated read file config test and subnet tests

* fixed config and util tests, updated subnet related functions

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Remove error return from GetDefaultConfig

Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>

* added json tags to database struct

* Updated beacon weights validation for config

* updates to score thresholds validation tags

* changes to config subnet validation and testing

* Update subnet.go

* Write missing host entries to http to populate http_proto

* Updating some fields to uint64

* WIP update some field types

* Update zeek count types and fix tests

* Add clickhouse credentials

* Misc fixes

* Update pointer

* Add ability to mark datasets as sample datasets

* fix column name

* Fix datasets exiting import if hour is empty

* Fix zeek count parsing from TSV files

* Remove storing dns conns in arrays, Fix historical first seen dns lag

* Remove unused columns

* Update config.hjson

* Update config.hjson

* updated impact category score functions to use float64

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Update subnet.go

* Store import version in imports table

* Fix duplicated SNI/IP long connections

* Update subnet_test.go

* Cleanup output

* Rolling files updates (#39)

* Limit number of days to import for rolling datasets

* Fix breaking imports when import was interrupted

* Remove debug output

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>

* Omit parts of env from output

* Set max for threat intel datasize

* Remove SELinux neutering for QA

* Add network size column

* Fix http_proto for missing host, update tests for missing host fixes

* Add online feeds to default config

* Update sshprep (#45)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Update sshprep

Add Bradley's suggestion of using head -1 to limit to a single address.

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Installer Behavior Tweaks (#41)

* Add --yes flag to add-apt-repository command

* Add missing sudo flags, make sure we're using the SUDO variable instead

* Add ability to perform zone transfers (#48)

* Store zone transfer records

Co-Authored-By: moth <25512187+0x6d6f7468@users.noreply.github.com>

* Update config

* Add tests

* Tests, connectivity test

* Update tests

---------

Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>

* Support RedHat/RHEL as a valid target (#47)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Supporrt RedHat/RHEL as a valid target

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* Fix tests (#49)

* Fix tests

* Update WalkFiles to use UTC

* fixed issue with rolling datasets over 24hours old not getting historical first seen timestamp set (#52)

* Change values from float32 to float64 (#50)

* Switch float32 to float64

* Update threat category calculation to match CalculateBucketedScore (#51)

---------

Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>

* Bump max query execution time default value

* Use string instead of error for ZoneTransferConnectivityErrors struct fields (#61)

* Upgrade Golang to version 1.24 (#59) (#60)

* Replace get_url with shell and curl (#58)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Replace get_url with shell and curl

* Use get_url by default, fall back to curl if it fails

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* add automated log transfer, AC-Hunter issue 135 (#62)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* add automated log transfer, PR135

* cron requires non-executable permission

* Specify suggested YAML plugin and config in VSCode workspace

* Linting and light cleanup

* Update generate_installer.sh

Download zeek_log_transport.sh to send to the sensor.

* Create cron file if remote zeek installation

* Only run zeek log import steps for remote sensor installations

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* Temporarily disable RITA/Zeek log transport until installer is modular (#66)

* Uniform -y flag usage for repo management/package installation; Uniform SUDO variable usage (#68)

* Resolve Installer Side Effects and Formalize RHEL Support (#73)

* Add missing necessary wildcards for RHEL versions

* Remove Ansible task replacing python3-requests to avoid RHEL distro installation side effects

* Update supported distros in README

* Update scoring defaults

* Resolve Ansible Reboot Errors (#75)

* Clean up conditionals; Fix reboot step for Ubuntu

* Suppress erroneous error output on RPM systems, ignore errors on reboot necessity checks

* Ignore missing host rows for openhttp (#76)

* Fix integration tests due to prevalence (#77)

---------

Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
Co-authored-by: William Stearns <william.l.stearns@gmail.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>
2025-09-22 16:21:05 -04:00

257 lines
6.8 KiB
Go

package cmd_test
import (
"bytes"
"context"
"fmt"
"log"
"os"
"path/filepath"
"testing"
"github.com/activecm/rita/v5/cmd"
"github.com/activecm/rita/v5/config"
"github.com/activecm/rita/v5/database"
"github.com/activecm/rita/v5/util"
"github.com/google/go-github/github"
"github.com/joho/godotenv"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"github.com/testcontainers/testcontainers-go"
cl "github.com/testcontainers/testcontainers-go/modules/clickhouse"
"github.com/urfave/cli/v2"
)
const ConfigPath = "../integration/test_config.hjson"
const TestDataPath = "../test_data"
type CmdTestSuite struct {
suite.Suite
cfg *config.Config
clickhouseContainer *cl.ClickHouseContainer
clickhouseConnection string
server *database.ServerConn
}
func TestMain(m *testing.M) {
// load environment variables with panic prevention
if err := godotenv.Overload("../.env", "../integration/test.env"); err != nil {
log.Fatalf("error loading .env file: %v", err)
}
// set version
config.Version = ""
// run the tests
os.Exit(m.Run())
}
func TestCmdTestSuite(t *testing.T) {
suite.Run(t, new(CmdTestSuite))
}
// SetupSuite is run once before the first test starts
func (c *CmdTestSuite) SetupSuite() {
t := c.T()
// set up file system interface
afs := afero.NewOsFs()
// load the config file
cfg, err := config.ReadFileConfig(afs, ConfigPath)
require.NoError(t, err, "config should load without error")
// // set version
// config.Version = ""
// start clickhouse container
c.SetupClickHouse(t)
// update the config to use the clickhouse container connection
cfg.Env.DBConnection = c.clickhouseConnection
cfg.RITA.UpdateCheckEnabled = false
c.cfg = cfg
// connect to clickhouse server
server, err := database.ConnectToServer(context.Background(), cfg)
require.NoError(t, err, "connecting to server should not produce an error")
c.server = server
}
// TearDownSuite is run once after all tests have finished
func (c *CmdTestSuite) TearDownSuite() {
if err := c.clickhouseContainer.Terminate(context.Background()); err != nil {
log.Fatalf("failed to terminate clickhouse container: %s", err)
}
}
// SetupTest is run before each test method
// func (d *DatabaseTestSuite) SetupTest() {}
// TearDownTest is run after each test method
// func (d *DatabaseTestSuite) TearDownTest() {}
// SetupSubTest is run before each subtest
// func (c *CmdTestSuite) SetupSubTest() {}
// TearDownSubTest is run after each subtest
// func (c *CmdTestSuite) TearDownSubTest() {}
// SetupClickHouse creates a ClickHouse container using the test.docker-compose.yml and handles taking it down when complete
func (c *CmdTestSuite) SetupClickHouse(t *testing.T) {
t.Helper()
version := os.Getenv("CLICKHOUSE_VERSION")
require.NotEmpty(t, version, "CLICKHOUSE_VERSION environment variable must be set")
// create ClickHouse container
ctx := context.Background()
clickHouseContainer, err := cl.RunContainer(ctx,
testcontainers.WithImage(fmt.Sprintf("clickhouse/clickhouse-server:%s-alpine", version)),
cl.WithUsername("default"),
cl.WithPassword(""),
cl.WithDatabase("default"),
cl.WithConfigFile(filepath.Join("../deployment/", "config.xml")),
)
require.NoError(t, err, "failed to start clickHouse container")
// get connection host
connectionHost, err := clickHouseContainer.ConnectionHost(ctx)
require.NoError(t, err, "failed to get clickHouse connection host")
// set container and connection host
c.clickhouseContainer = clickHouseContainer
c.clickhouseConnection = connectionHost
}
func setupTestApp(commands []*cli.Command, flags []cli.Flag) (*cli.App, context.Context) {
ctx := context.Background()
app := cli.NewApp()
app.Args = true
app.Commands = commands
app.Flags = flags
// custom exit handler to override the default which calls os.Exit
// this prevents the test from exiting when testing for errors
app.ExitErrHandler = func(_ *cli.Context, _ error) {
// add any custom test logic, or assertions or leave it blank
}
return app, ctx
}
func TestCheckForUpdate(t *testing.T) {
// set up file system interface
afs := afero.NewOsFs()
// load the config file
cfg, err := config.ReadFileConfig(afs, ConfigPath)
require.NoError(t, err, "config should load without error")
// get latest release version
latestVersion, err := util.GetLatestReleaseVersion(github.NewClient(nil), "activecm", "rita")
require.NoError(t, err, "latest release version should be retrieved without error")
tests := []struct {
name string
cfg *config.Config
updateCheckEnabled bool
currentVersion string
expectedErr error
expectedOutput string
}{
{
name: "New version available",
updateCheckEnabled: true,
cfg: cfg,
currentVersion: "v0.0.0",
expectedOutput: fmt.Sprintf("\n\t✨ A newer version (%s) of RITA is available! https://github.com/activecm/rita/releases ✨\n\n", latestVersion),
},
{
name: "Error checking for newer version",
updateCheckEnabled: true,
cfg: cfg,
currentVersion: "notaversion",
expectedErr: cmd.ErrCheckingForUpdate,
},
{
name: "Update check disabled",
updateCheckEnabled: false,
cfg: cfg,
currentVersion: "1.0.0",
},
{
name: "Current version is dev",
updateCheckEnabled: true,
cfg: cfg,
currentVersion: "dev",
},
{
name: "Current version is empty",
updateCheckEnabled: true,
cfg: cfg,
currentVersion: "",
},
{
name: "Nil config",
cfg: nil,
expectedErr: cmd.ErrInvalidConfigObject,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
// set update check enabled in config
if test.cfg != nil {
test.cfg.RITA.UpdateCheckEnabled = test.updateCheckEnabled
}
// override global variables and functions
config.Version = test.currentVersion
// capture stdout
output := captureOutput(t, func() {
err := cmd.CheckForUpdate(test.cfg)
// check error
if test.expectedErr != nil {
require.Contains(t, err.Error(), test.expectedErr.Error(), "error should contain expected value")
} else {
assert.NoError(t, err)
}
})
// Assert output
if test.expectedOutput != "" {
assert.Equal(t, test.expectedOutput, output)
}
})
}
}
// captureOutput captures stdout from a function
func captureOutput(t *testing.T, f func()) string {
t.Helper()
// capture stdout
old := os.Stdout
r, w, err := os.Pipe()
require.NoError(t, err)
os.Stdout = w
// run the function
f()
// close and restore stdout
w.Close()
os.Stdout = old
var buf bytes.Buffer
_, err = buf.ReadFrom(r)
require.NoError(t, err)
return buf.String()
}