* Added limits to available CPU cores during importing
Also added tests for new functions used
* Update import.go
---------
Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
* Updated http_extensions list for xml
- Added the xsd file extension to the list
- Named application/xml mime type 'xml' as per IANA standard
* Added xml mime types data to test case
---------
Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
* Install Ansible using pipx instead of Linux package manager
* Prepend ~/.local/bin to path if not present (needed on Ubuntu to find pipx-installed Ansible)
* Install pipx in a virtual environment to avoid PEP 668 issues
* Pin ansible-core version 2.15.3 to avoid deprecation warnings and breaking syntax changes
* Pin ansible-core 2.15.13, force installation
* Update on config structure, functionality, and tests
Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>
* Extend subnet type to read/write from db, update tests
Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>
* updated read file config test and subnet tests
* fixed config and util tests, updated subnet related functions
Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>
* Remove error return from GetDefaultConfig
Co-Authored-By: Liza Tsibur <liza@activecountermeasures.com>
* added json tags to database struct
* Updated beacon weights validation for config
* updates to score thresholds validation tags
* changes to config subnet validation and testing
* Update subnet.go
* Write missing host entries to http to populate http_proto
* Updating some fields to uint64
* WIP update some field types
* Update zeek count types and fix tests
* Add clickhouse credentials
* Misc fixes
* Update pointer
* Add ability to mark datasets as sample datasets
* fix column name
* Fix datasets exiting import if hour is empty
* Fix zeek count parsing from TSV files
* Remove storing dns conns in arrays, Fix historical first seen dns lag
* Remove unused columns
* Update config.hjson
* Update config.hjson
* updated impact category score functions to use float64
Co-Authored-By: Naomi Kramer <naomiagoddard@gmail.com>
* Update subnet.go
* Store import version in imports table
* Fix duplicated SNI/IP long connections
* Update subnet_test.go
* Cleanup output
* Rolling files updates (#39)
* Limit number of days to import for rolling datasets
* Fix breaking imports when import was interrupted
* Remove debug output
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
* Omit parts of env from output
* Set max for threat intel datasize
* Remove SELinux neutering for QA
* Add network size column
* Fix http_proto for missing host, update tests for missing host fixes
* Add online feeds to default config
* Update sshprep (#45)
* Update sshprep
Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>
* Update sshprep
Add Bradley's suggestion of using head -1 to limit to a single address.
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
* Installer Behavior Tweaks (#41)
* Add --yes flag to add-apt-repository command
* Add missing sudo flags, make sure we're using the SUDO variable instead
* Add ability to perform zone transfers (#48)
* Store zone transfer records
Co-Authored-By: moth <25512187+0x6d6f7468@users.noreply.github.com>
* Update config
* Add tests
* Tests, connectivity test
* Update tests
---------
Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>
* Support RedHat/RHEL as a valid target (#47)
* Update sshprep
Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>
* Supporrt RedHat/RHEL as a valid target
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
* Fix tests (#49)
* Fix tests
* Update WalkFiles to use UTC
* fixed issue with rolling datasets over 24hours old not getting historical first seen timestamp set (#52)
* Change values from float32 to float64 (#50)
* Switch float32 to float64
* Update threat category calculation to match CalculateBucketedScore (#51)
---------
Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
* Bump max query execution time default value
* Use string instead of error for ZoneTransferConnectivityErrors struct fields (#61)
* Upgrade Golang to version 1.24 (#59) (#60)
* Replace get_url with shell and curl (#58)
* Update sshprep
Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>
* Replace get_url with shell and curl
* Use get_url by default, fall back to curl if it fails
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
* add automated log transfer, AC-Hunter issue 135 (#62)
* Update sshprep
Co-Authored-By: William Stearns <3538265+william-stearns@users.noreply.github.com>
* add automated log transfer, PR135
* cron requires non-executable permission
* Specify suggested YAML plugin and config in VSCode workspace
* Linting and light cleanup
* Update generate_installer.sh
Download zeek_log_transport.sh to send to the sensor.
* Create cron file if remote zeek installation
* Only run zeek log import steps for remote sensor installations
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
* Temporarily disable RITA/Zeek log transport until installer is modular (#66)
* Uniform -y flag usage for repo management/package installation; Uniform SUDO variable usage (#68)
* Resolve Installer Side Effects and Formalize RHEL Support (#73)
* Add missing necessary wildcards for RHEL versions
* Remove Ansible task replacing python3-requests to avoid RHEL distro installation side effects
* Update supported distros in README
* Update scoring defaults
* Resolve Ansible Reboot Errors (#75)
* Clean up conditionals; Fix reboot step for Ubuntu
* Suppress erroneous error output on RPM systems, ignore errors on reboot necessity checks
* Ignore missing host rows for openhttp (#76)
* Fix integration tests due to prevalence (#77)
---------
Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
Co-authored-by: moth <moth@blackhillsinfosec.com>
Co-authored-by: William Stearns <william.l.stearns@gmail.com>
Co-authored-by: William Stearns <3538265+william-stearns@users.noreply.github.com>
Co-authored-by: moth <25512187+0x6d6f7468@users.noreply.github.com>
* Split prep and post tasks into their own playbook
* all playbooks in one command
* Replace ansibles apt-key module (deprecated)
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Co-authored-by: Liza Tsibur <liza@activecountermeasures.com>
* Split prep and post tasks into their own playbook
* all playbooks in one command
* Include install_pre and install_post playbooks in installer generator
---------
Co-authored-by: Naomi Kramer <naomi@activecountermeasures.com>
Naomi Kramer