Files
Naomi Kramer b923c39e2d Update configuration (#78)
* Update on config structure, functionality, and tests

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Extend subnet type to read/write from db, update tests

Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>

* updated read file config test and subnet tests

* fixed config and util tests, updated subnet related functions

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Remove error return from GetDefaultConfig

Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>

* added json tags to database struct

* Updated beacon weights validation for config

* updates to score thresholds validation tags

* changes to config subnet validation and testing

* Update subnet.go

* Write missing host entries to http to populate http_proto

* Updating some fields to uint64

* WIP update some field types

* Update zeek count types and fix tests

* Add clickhouse credentials

* Misc fixes

* Update pointer

* Add ability to mark datasets as sample datasets

* fix column name

* Fix datasets exiting import if hour is empty

* Fix zeek count parsing from TSV files

* Remove storing dns conns in arrays, Fix historical first seen dns lag

* Remove unused columns

* Update config.hjson

* Update config.hjson

* updated impact category score functions to use float64

Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>

* Update subnet.go

* Store import version in imports table

* Fix duplicated SNI/IP long connections

* Update subnet_test.go

* Cleanup output

* Rolling files updates (#39)

* Limit number of days to import for rolling datasets

* Fix breaking imports when import was interrupted

* Remove debug output

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>

* Omit parts of env from output

* Set max for threat intel datasize

* Remove SELinux neutering for QA

* Add network size column

* Fix http_proto for missing host, update tests for missing host fixes

* Add online feeds to default config

* Update sshprep (#45)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Update sshprep

Add Bradley's suggestion of using head -1 to limit to a single address.

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Installer Behavior Tweaks (#41)

* Add --yes flag to add-apt-repository command

* Add missing sudo flags, make sure we're using the SUDO variable instead

* Add ability to perform zone transfers (#48)

* Store zone transfer records

Co-Authored-By: moth <25512187+0x6d6f7468@users.noreply.github.com>

* Update config

* Add tests

* Tests, connectivity test

* Update tests

---------

Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>

* Support RedHat/RHEL as a valid target (#47)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Supporrt RedHat/RHEL as a valid target

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* Fix tests (#49)

* Fix tests

* Update WalkFiles to use UTC

* fixed issue with rolling datasets over 24hours old not getting historical first seen timestamp set (#52)

* Change values from float32 to float64 (#50)

* Switch float32 to float64

* Update threat category calculation to match CalculateBucketedScore (#51)

---------

Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>

* Bump max query execution time default value

* Use string instead of error for ZoneTransferConnectivityErrors struct fields (#61)

* Upgrade Golang to version 1.24 (#59) (#60)

* Replace get_url with shell and curl (#58)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* Replace get_url with shell and curl

* Use get_url by default, fall back to curl if it fails

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* add automated log transfer, AC-Hunter issue 135 (#62)

* Update sshprep

Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>

* add automated log transfer, PR135

* cron requires non-executable permission

* Specify suggested YAML plugin and config in VSCode workspace

* Linting and light cleanup

* Update generate_installer.sh

Download zeek_log_transport.sh to send to the sensor.

* Create cron file if remote zeek installation

* Only run zeek log import steps for remote sensor installations

---------

Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>

* Temporarily disable RITA/Zeek log transport until installer is modular (#66)

* Uniform -y flag usage for repo management/package installation; Uniform SUDO variable usage (#68)

* Resolve Installer Side Effects and Formalize RHEL Support (#73)

* Add missing necessary wildcards for RHEL versions

* Remove Ansible task replacing python3-requests to avoid RHEL distro installation side effects

* Update supported distros in README

* Update scoring defaults

* Resolve Ansible Reboot Errors (#75)

* Clean up conditionals; Fix reboot step for Ubuntu

* Suppress erroneous error output on RPM systems, ignore errors on reboot necessity checks

* Ignore missing host rows for openhttp (#76)

* Fix integration tests due to prevalence (#77)

---------

Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
Co-authored-by: William Stearns <william.l.stearns@gmail.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>
2025-09-22 16:21:05 -04:00

269 lines
9.2 KiB
Go

package config
import (
"net"
"testing"
"github.com/activecm/rita/v5/util"
"github.com/stretchr/testify/require"
)
func TestFilterConnPair(t *testing.T) {
internalSubnetListEmpty := []util.Subnet{}
internalSubnetList := util.NewTestSubnetList(t, []string{
"11.0.0.0/8", "120.0.0.0/8",
})
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
"35.0.0.0/8", "170.0.0.0/8",
})
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
"12.0.0.0/8", "150.0.0.0/8",
})
// load config
cfg := GetDefaultConfig()
// set config filter for external to internal to false
cfg.Filtering.FilterExternalToInternal = false
// AlwaysInclude list tests
t.Run("AlwaysInclude list tests", func(t *testing.T) {
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
checkCases := cfg.Filtering.FilterConnPair(net.IP{35, 0, 0, 0}, net.IP{190, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
checkCases = cfg.Filtering.FilterConnPair(net.IP{190, 0, 0, 0}, net.IP{35, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
// NeverInclude list tests
t.Run("NeverInclude list tests", func(t *testing.T) {
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
checkCases := cfg.Filtering.FilterConnPair(net.IP{12, 0, 0, 0}, net.IP{190, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
checkCases = cfg.Filtering.FilterConnPair(net.IP{190, 0, 0, 0}, net.IP{12, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
})
// InternalSubnets tests
t.Run("InternalSubnets tests", func(t *testing.T) {
cfg.Filtering.InternalSubnets = internalSubnetList
// Both are external
checkCases := cfg.Filtering.FilterConnPair(net.IP{185, 0, 0, 0}, net.IP{16, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
// Both are internal
checkCases = cfg.Filtering.FilterConnPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
// Source is external, destination is internal, FilterExternalToInternal set
cfg.Filtering.FilterExternalToInternal = true
checkCases = cfg.Filtering.FilterConnPair(net.IP{180, 0, 0, 0}, net.IP{11, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
checkCases = cfg.Filtering.FilterDNSPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
// Empty list
cfg.Filtering.InternalSubnets = internalSubnetListEmpty
checkCases = cfg.Filtering.FilterConnPair(net.IP{180, 0, 0, 0}, net.IP{80, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
}
func TestFilterDNSPair(t *testing.T) {
internalSubnetListEmpty := []util.Subnet{}
internalSubnetList := util.NewTestSubnetList(t, []string{
"11.0.0.0/8", "120.0.0.0/8",
})
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
"35.0.0.0/8", "170.0.0.0/8",
})
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
"12.0.0.0/8", "150.0.0.0/8",
})
// load config
cfg := GetDefaultConfig()
// set config filter for external to internal to false
cfg.Filtering.FilterExternalToInternal = false
// AlwaysInclude list tests
t.Run("AlwaysInclude list tests", func(t *testing.T) {
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
checkCases := cfg.Filtering.FilterDNSPair(net.IP{35, 0, 0, 0}, net.IP{190, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
checkCases = cfg.Filtering.FilterDNSPair(net.IP{190, 0, 0, 0}, net.IP{35, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
// NeverInclude list tests
t.Run("NeverInclude list tests", func(t *testing.T) {
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
checkCases := cfg.Filtering.FilterDNSPair(net.IP{12, 0, 0, 0}, net.IP{190, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
checkCases = cfg.Filtering.FilterDNSPair(net.IP{190, 0, 0, 0}, net.IP{12, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
})
// InternalSubnets tests
t.Run("InternalSubnets tests", func(t *testing.T) {
cfg.Filtering.InternalSubnets = internalSubnetList
// Both are external
checkCases := cfg.Filtering.FilterDNSPair(net.IP{185, 0, 0, 0}, net.IP{16, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
// Source is external, destination is internal, FilterExternalToInternal set
cfg.Filtering.FilterExternalToInternal = true
checkCases = cfg.Filtering.FilterDNSPair(net.IP{180, 0, 0, 0}, net.IP{120, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
checkCases = cfg.Filtering.FilterDNSPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
// Empty list
cfg.Filtering.InternalSubnets = internalSubnetListEmpty
checkCases = cfg.Filtering.FilterDNSPair(net.IP{180, 0, 0, 0}, net.IP{80, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
}
func TestFilterSingleIP(t *testing.T) {
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
"35.0.0.0/8", "170.0.0.0/8",
})
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
"12.0.0.0/8", "150.0.0.0/8",
})
// load config
cfg := GetDefaultConfig()
// AlwaysInclude list test
t.Run("AlwaysInclude list test", func(t *testing.T) {
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
checkCases := cfg.Filtering.FilterSingleIP(net.IP{35, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
// NeverInclude list test
t.Run("NeverInclude list test", func(t *testing.T) {
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
checkCases := cfg.Filtering.FilterSingleIP(net.IP{12, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
})
}
func TestFilterDomain(t *testing.T) {
alwaysIncludedDomainList := []string{
"trustmebro-university.com",
"expressbuy.com",
}
neverIncludedDomainList := []string{
"bing.com",
"google.com",
}
// load config
cfg := GetDefaultConfig()
// AlwaysInclude list test
t.Run("AlwaysInclude list test", func(t *testing.T) {
cfg.Filtering.AlwaysIncludedDomains = alwaysIncludedDomainList
checkCases := cfg.Filtering.FilterDomain("trustmebro-university.com")
require.False(t, checkCases, "filter state should match expected value")
})
// NeverInclude list test
t.Run("NeverInclude list test", func(t *testing.T) {
cfg.Filtering.NeverIncludedDomains = neverIncludedDomainList
checkCases := cfg.Filtering.FilterDomain("bing.com")
require.True(t, checkCases, "filter state should match expected value")
})
}
func TestFilterNeverInclude(t *testing.T) {
// load config
cfg := GetDefaultConfig()
t.Run("Value not in NeverInclude list", func(t *testing.T) {
filtered := cfg.Filtering.FilterSingleIP(net.IP{65, 0, 0, 0})
require.False(t, filtered, "filter state should match expected value")
})
t.Run("IPv4 broadcast", func(t *testing.T) {
filtered := cfg.Filtering.FilterSingleIP(net.IPv4bcast)
require.True(t, filtered, "filter state should match expected value")
})
t.Run("IPv4 all zeros address", func(t *testing.T) {
filtered := cfg.Filtering.FilterSingleIP(net.IPv4zero)
require.True(t, filtered, "filter state should match expected value")
})
t.Run("IPv6 unspecified address", func(t *testing.T) {
filtered := cfg.Filtering.FilterSingleIP(net.IPv6unspecified)
require.True(t, filtered, "filter state should match expected value")
})
}
func TestCheckIfInternal(t *testing.T) {
internalSubnetList := util.NewTestSubnetList(t, []string{
"11.0.0.0/8", "120.0.0.0/8",
})
// load config
cfg := GetDefaultConfig()
// set internal subnets
cfg.Filtering.InternalSubnets = internalSubnetList
// internal ip
t.Run("Valid Internal IP", func(t *testing.T) {
checkCases := cfg.Filtering.CheckIfInternal(net.IP{11, 0, 0, 0})
require.True(t, checkCases, "filter state should match expected value")
})
// external ip
t.Run("Valid External IP", func(t *testing.T) {
checkCases := cfg.Filtering.CheckIfInternal(net.IP{110, 0, 0, 0})
require.False(t, checkCases, "filter state should match expected value")
})
// unspecified ip
t.Run("Unspecified IPv6", func(t *testing.T) {
checkCases := cfg.Filtering.CheckIfInternal(net.IPv6unspecified)
require.False(t, checkCases, "filter state should match expected value")
})
// all zeros ip
t.Run("All Zeros IPv4", func(t *testing.T) {
checkCases := cfg.Filtering.CheckIfInternal(net.IPv4zero)
require.False(t, checkCases, "filter state should match expected value")
})
// broadcast ip
t.Run("Broadcast IP", func(t *testing.T) {
checkCases := cfg.Filtering.CheckIfInternal(net.IPv4bcast)
require.False(t, checkCases, "filter state should match expected value")
})
}