mirror of
https://github.com/activecm/rita
synced 2026-06-08 13:02:45 +00:00
b923c39e2d
* Update on config structure, functionality, and tests Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com> * Extend subnet type to read/write from db, update tests Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com> * updated read file config test and subnet tests * fixed config and util tests, updated subnet related functions Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com> * Remove error return from GetDefaultConfig Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com> * added json tags to database struct * Updated beacon weights validation for config * updates to score thresholds validation tags * changes to config subnet validation and testing * Update subnet.go * Write missing host entries to http to populate http_proto * Updating some fields to uint64 * WIP update some field types * Update zeek count types and fix tests * Add clickhouse credentials * Misc fixes * Update pointer * Add ability to mark datasets as sample datasets * fix column name * Fix datasets exiting import if hour is empty * Fix zeek count parsing from TSV files * Remove storing dns conns in arrays, Fix historical first seen dns lag * Remove unused columns * Update config.hjson * Update config.hjson * updated impact category score functions to use float64 Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com> * Update subnet.go * Store import version in imports table * Fix duplicated SNI/IP long connections * Update subnet_test.go * Cleanup output * Rolling files updates (#39) * Limit number of days to import for rolling datasets * Fix breaking imports when import was interrupted * Remove debug output --------- Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com> * Omit parts of env from output * Set max for threat intel datasize * Remove SELinux neutering for QA * Add network size column * Fix http_proto for missing host, update tests for missing host fixes * Add online feeds to default config * Update sshprep (#45) * Update sshprep Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com> * Update sshprep Add Bradley's suggestion of using head -1 to limit to a single address. --------- Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com> Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com> * Installer Behavior Tweaks (#41) * Add --yes flag to add-apt-repository command * Add missing sudo flags, make sure we're using the SUDO variable instead * Add ability to perform zone transfers (#48) * Store zone transfer records Co-Authored-By: moth <25512187+0x6d6f7468@users.noreply.github.com> * Update config * Add tests * Tests, connectivity test * Update tests --------- Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com> * Support RedHat/RHEL as a valid target (#47) * Update sshprep Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com> * Supporrt RedHat/RHEL as a valid target --------- Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com> Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com> Co-authored-by: moth <moth@blackhillsinfosec.com> * Fix tests (#49) * Fix tests * Update WalkFiles to use UTC * fixed issue with rolling datasets over 24hours old not getting historical first seen timestamp set (#52) * Change values from float32 to float64 (#50) * Switch float32 to float64 * Update threat category calculation to match CalculateBucketedScore (#51) --------- Co-authored-by: Liza Tsibur <liza@activecountermeasures.com> * Bump max query execution time default value * Use string instead of error for ZoneTransferConnectivityErrors struct fields (#61) * Upgrade Golang to version 1.24 (#59) (#60) * Replace get_url with shell and curl (#58) * Update sshprep Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com> * Replace get_url with shell and curl * Use get_url by default, fall back to curl if it fails --------- Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com> Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com> Co-authored-by: moth <moth@blackhillsinfosec.com> * add automated log transfer, AC-Hunter issue 135 (#62) * Update sshprep Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com> * add automated log transfer, PR135 * cron requires non-executable permission * Specify suggested YAML plugin and config in VSCode workspace * Linting and light cleanup * Update generate_installer.sh Download zeek_log_transport.sh to send to the sensor. * Create cron file if remote zeek installation * Only run zeek log import steps for remote sensor installations --------- Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com> Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com> Co-authored-by: moth <moth@blackhillsinfosec.com> * Temporarily disable RITA/Zeek log transport until installer is modular (#66) * Uniform -y flag usage for repo management/package installation; Uniform SUDO variable usage (#68) * Resolve Installer Side Effects and Formalize RHEL Support (#73) * Add missing necessary wildcards for RHEL versions * Remove Ansible task replacing python3-requests to avoid RHEL distro installation side effects * Update supported distros in README * Update scoring defaults * Resolve Ansible Reboot Errors (#75) * Clean up conditionals; Fix reboot step for Ubuntu * Suppress erroneous error output on RPM systems, ignore errors on reboot necessity checks * Ignore missing host rows for openhttp (#76) * Fix integration tests due to prevalence (#77) --------- Co-authored-by: Liza Tsibur <liza@activecountermeasures.com> Co-authored-by: moth <moth@blackhillsinfosec.com> Co-authored-by: William Stearns <william.l.stearns@gmail.com> Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com> Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>
269 lines
9.2 KiB
Go
269 lines
9.2 KiB
Go
package config
|
|
|
|
import (
|
|
"net"
|
|
"testing"
|
|
|
|
"github.com/activecm/rita/v5/util"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
func TestFilterConnPair(t *testing.T) {
|
|
internalSubnetListEmpty := []util.Subnet{}
|
|
|
|
internalSubnetList := util.NewTestSubnetList(t, []string{
|
|
"11.0.0.0/8", "120.0.0.0/8",
|
|
})
|
|
|
|
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"35.0.0.0/8", "170.0.0.0/8",
|
|
})
|
|
|
|
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"12.0.0.0/8", "150.0.0.0/8",
|
|
})
|
|
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
// set config filter for external to internal to false
|
|
cfg.Filtering.FilterExternalToInternal = false
|
|
|
|
// AlwaysInclude list tests
|
|
t.Run("AlwaysInclude list tests", func(t *testing.T) {
|
|
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterConnPair(net.IP{35, 0, 0, 0}, net.IP{190, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
|
|
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
|
|
checkCases = cfg.Filtering.FilterConnPair(net.IP{190, 0, 0, 0}, net.IP{35, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// NeverInclude list tests
|
|
t.Run("NeverInclude list tests", func(t *testing.T) {
|
|
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterConnPair(net.IP{12, 0, 0, 0}, net.IP{190, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
|
|
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
|
|
checkCases = cfg.Filtering.FilterConnPair(net.IP{190, 0, 0, 0}, net.IP{12, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// InternalSubnets tests
|
|
t.Run("InternalSubnets tests", func(t *testing.T) {
|
|
cfg.Filtering.InternalSubnets = internalSubnetList
|
|
|
|
// Both are external
|
|
checkCases := cfg.Filtering.FilterConnPair(net.IP{185, 0, 0, 0}, net.IP{16, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
|
|
// Both are internal
|
|
checkCases = cfg.Filtering.FilterConnPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
|
|
// Source is external, destination is internal, FilterExternalToInternal set
|
|
cfg.Filtering.FilterExternalToInternal = true
|
|
checkCases = cfg.Filtering.FilterConnPair(net.IP{180, 0, 0, 0}, net.IP{11, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
|
|
// Empty list
|
|
cfg.Filtering.InternalSubnets = internalSubnetListEmpty
|
|
checkCases = cfg.Filtering.FilterConnPair(net.IP{180, 0, 0, 0}, net.IP{80, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
}
|
|
|
|
func TestFilterDNSPair(t *testing.T) {
|
|
internalSubnetListEmpty := []util.Subnet{}
|
|
|
|
internalSubnetList := util.NewTestSubnetList(t, []string{
|
|
"11.0.0.0/8", "120.0.0.0/8",
|
|
})
|
|
|
|
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"35.0.0.0/8", "170.0.0.0/8",
|
|
})
|
|
|
|
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"12.0.0.0/8", "150.0.0.0/8",
|
|
})
|
|
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
// set config filter for external to internal to false
|
|
cfg.Filtering.FilterExternalToInternal = false
|
|
|
|
// AlwaysInclude list tests
|
|
t.Run("AlwaysInclude list tests", func(t *testing.T) {
|
|
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterDNSPair(net.IP{35, 0, 0, 0}, net.IP{190, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{190, 0, 0, 0}, net.IP{35, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// NeverInclude list tests
|
|
t.Run("NeverInclude list tests", func(t *testing.T) {
|
|
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterDNSPair(net.IP{12, 0, 0, 0}, net.IP{190, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{190, 0, 0, 0}, net.IP{12, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// InternalSubnets tests
|
|
t.Run("InternalSubnets tests", func(t *testing.T) {
|
|
cfg.Filtering.InternalSubnets = internalSubnetList
|
|
|
|
// Both are external
|
|
checkCases := cfg.Filtering.FilterDNSPair(net.IP{185, 0, 0, 0}, net.IP{16, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
|
|
// Source is external, destination is internal, FilterExternalToInternal set
|
|
cfg.Filtering.FilterExternalToInternal = true
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{180, 0, 0, 0}, net.IP{120, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{11, 0, 0, 0}, net.IP{120, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
|
|
// Empty list
|
|
cfg.Filtering.InternalSubnets = internalSubnetListEmpty
|
|
checkCases = cfg.Filtering.FilterDNSPair(net.IP{180, 0, 0, 0}, net.IP{80, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
}
|
|
|
|
func TestFilterSingleIP(t *testing.T) {
|
|
|
|
alwaysIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"35.0.0.0/8", "170.0.0.0/8",
|
|
})
|
|
|
|
neverIncludedSubnetList := util.NewTestSubnetList(t, []string{
|
|
"12.0.0.0/8", "150.0.0.0/8",
|
|
})
|
|
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
// AlwaysInclude list test
|
|
t.Run("AlwaysInclude list test", func(t *testing.T) {
|
|
cfg.Filtering.AlwaysIncludedSubnets = alwaysIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterSingleIP(net.IP{35, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// NeverInclude list test
|
|
t.Run("NeverInclude list test", func(t *testing.T) {
|
|
cfg.Filtering.NeverIncludedSubnets = neverIncludedSubnetList
|
|
checkCases := cfg.Filtering.FilterSingleIP(net.IP{12, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
})
|
|
}
|
|
|
|
func TestFilterDomain(t *testing.T) {
|
|
alwaysIncludedDomainList := []string{
|
|
"trustmebro-university.com",
|
|
"expressbuy.com",
|
|
}
|
|
|
|
neverIncludedDomainList := []string{
|
|
"bing.com",
|
|
"google.com",
|
|
}
|
|
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
// AlwaysInclude list test
|
|
t.Run("AlwaysInclude list test", func(t *testing.T) {
|
|
cfg.Filtering.AlwaysIncludedDomains = alwaysIncludedDomainList
|
|
checkCases := cfg.Filtering.FilterDomain("trustmebro-university.com")
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// NeverInclude list test
|
|
t.Run("NeverInclude list test", func(t *testing.T) {
|
|
cfg.Filtering.NeverIncludedDomains = neverIncludedDomainList
|
|
checkCases := cfg.Filtering.FilterDomain("bing.com")
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
})
|
|
}
|
|
|
|
func TestFilterNeverInclude(t *testing.T) {
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
t.Run("Value not in NeverInclude list", func(t *testing.T) {
|
|
filtered := cfg.Filtering.FilterSingleIP(net.IP{65, 0, 0, 0})
|
|
require.False(t, filtered, "filter state should match expected value")
|
|
})
|
|
|
|
t.Run("IPv4 broadcast", func(t *testing.T) {
|
|
filtered := cfg.Filtering.FilterSingleIP(net.IPv4bcast)
|
|
require.True(t, filtered, "filter state should match expected value")
|
|
})
|
|
|
|
t.Run("IPv4 all zeros address", func(t *testing.T) {
|
|
filtered := cfg.Filtering.FilterSingleIP(net.IPv4zero)
|
|
require.True(t, filtered, "filter state should match expected value")
|
|
})
|
|
|
|
t.Run("IPv6 unspecified address", func(t *testing.T) {
|
|
filtered := cfg.Filtering.FilterSingleIP(net.IPv6unspecified)
|
|
require.True(t, filtered, "filter state should match expected value")
|
|
})
|
|
}
|
|
|
|
func TestCheckIfInternal(t *testing.T) {
|
|
|
|
internalSubnetList := util.NewTestSubnetList(t, []string{
|
|
"11.0.0.0/8", "120.0.0.0/8",
|
|
})
|
|
|
|
// load config
|
|
cfg := GetDefaultConfig()
|
|
|
|
// set internal subnets
|
|
cfg.Filtering.InternalSubnets = internalSubnetList
|
|
|
|
// internal ip
|
|
t.Run("Valid Internal IP", func(t *testing.T) {
|
|
checkCases := cfg.Filtering.CheckIfInternal(net.IP{11, 0, 0, 0})
|
|
require.True(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// external ip
|
|
t.Run("Valid External IP", func(t *testing.T) {
|
|
checkCases := cfg.Filtering.CheckIfInternal(net.IP{110, 0, 0, 0})
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// unspecified ip
|
|
t.Run("Unspecified IPv6", func(t *testing.T) {
|
|
checkCases := cfg.Filtering.CheckIfInternal(net.IPv6unspecified)
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// all zeros ip
|
|
t.Run("All Zeros IPv4", func(t *testing.T) {
|
|
checkCases := cfg.Filtering.CheckIfInternal(net.IPv4zero)
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
// broadcast ip
|
|
t.Run("Broadcast IP", func(t *testing.T) {
|
|
checkCases := cfg.Filtering.CheckIfInternal(net.IPv4bcast)
|
|
require.False(t, checkCases, "filter state should match expected value")
|
|
})
|
|
|
|
}
|