39d7121dcd
Updating mlmodel to 5.3.x specification as well as the test apparatus
2022-01-25 21:42:16 -05:00
f45d2067da
fix for test key in yml
2022-01-14 14:11:23 -08:00
b928268e90
Merge pull request #1937 from splunk/DockerDetectionUpdatesAndFixes
...
Docker detection updates and fixes
2022-01-14 08:55:34 -08:00
5ecb9f5fbd
Tests were failing on GH Actions that should not have been failing. This is likely due to not enough time waiting for indexing to fail. Doubled the time we wait between indexing complete check. Also moved from hourly tests of all detections to daily.
2022-01-13 15:50:07 -08:00
67ef36117b
Fixed bad printout to console where, if a test failed but --interactive mode was declared, the test would print out as having succeeded. Improved error handling if there is an issue where an uploaded data file generates and exception.
2022-01-13 12:03:49 -08:00
82a85b1305
Branch was auto-updated.
2022-01-13 09:33:10 -08:00
359c3159d0
Branch was auto-updated.
2022-01-13 09:20:09 -08:00
0a7b4d27de
minor fix
2022-01-13 09:17:03 -08:00
860320c367
Branch was auto-updated.
2022-01-12 16:50:15 -08:00
20df3795b3
Merge pull request #1933 from splunk/DockerDetectionUpdatesAndFixes
...
Move Detection Testing job to detection-testing
2022-01-12 16:49:51 -08:00
6a7f917707
Fix to requirements to include psutil. Removed import of deleted modules/aws_service.py
2022-01-12 16:40:56 -08:00
cdc902e052
Removed ansible from requirements and boto3 from requirements. Neither are used anymore
2022-01-12 16:35:11 -08:00
8d669b66f8
Removing previous file that was added accidentally in last commit. Updating the detection-testing.yml file to remove ls of non-existant directory as intended
2022-01-12 16:17:38 -08:00
fc189e92ee
Removed ls or directory that didn't exist anymore
...
in detection-testing.yml
2022-01-12 16:10:21 -08:00
85ac64bb8f
Branch was auto-updated.
2022-01-12 15:19:35 -08:00
dee77e7957
Merge pull request #1932 from splunk/DockerDetectionUpdatesAndFixes
...
Changed summarize_json.py to return success
2022-01-12 15:19:13 -08:00
73998ca5c1
Changed summarize_json.py to return success
...
(0) code even if there are failures or errors found
during testing during the initial testing of CI/CD.
This is because we don't want these to hold up
the build process. Eventually we will report actual
success/failure of all tests after we have the
opportunity to validate the new testing pipeline and
make updates to failing searches.
2022-01-12 14:57:17 -08:00
5d42ca4454
Branch was auto-updated.
2022-01-12 12:17:54 -08:00
52e482c604
Merge pull request #1928 from splunk/DockerDetectionUpdatesAndFixes
...
Docker detection updates and fixes
2022-01-12 12:17:22 -08:00
b12ccf0cd2
Added a readme.md. Need to check that the
...
markdown renders appropriately
2022-01-12 10:54:45 -08:00
3afef5d559
Updated the error for providing both PR and
...
commit_hash to a warning. This was causing
an error during PR triggered testing in github
actions as both of these are provided. Perhaps
we actually can include both of these and can
get old PRs based on commit hash, but until
then we will just test using the PR number and
ignore the commit_hash if we find both.
2022-01-12 10:38:23 -08:00
7b3029b187
Removed splunkbase-only test_config file. Pushing to run a test on all. Will push again for changes only.
2022-01-11 16:35:19 -08:00
23da2f45d2
removing file that was accidentally added.
2022-01-11 16:06:47 -08:00
e3ba9f0074
Finished removing dead code from DataManipulation.
...
Added the new_sysmon as an additional default application.
While it is not ideal to have the old and new sysmons installed
side by side, there are some detection that only work with one
and not the other. We will run both until we have moved
all detections to the new sysmon.
2022-01-11 16:03:04 -08:00
6a6bd04819
Removing some dead code from
...
DataManipulation.py
2022-01-11 15:46:38 -08:00
79d13fccef
Converted DataManipulation from using FileInput to with open, readline, etc. We needed to do this because FileInput with inplace=True remaps the print command, which may be called by several parallel threads running DataManipulation or to output status info, will cause a crash if multiple threads write to it.
2022-01-11 14:18:04 -08:00
81df4e0eec
A bunch of changes to support parallel testing and waiting to ensure data has been properly ingested/indexed before running a test. Still double-checking edge cases for this. Also improved some error printouts and status printouts. When a test completes, it now includes its runtime in its pass/fail/error print.
2022-01-11 11:59:26 -08:00
17d7312bc6
excluding experimental detections from the package
2022-01-10 18:00:59 -05:00
2197a6986c
removed deprecated detections from ssa package
2022-01-10 17:45:57 -05:00
6edaa90519
updated generate to match javiers feedback
2022-01-10 15:12:30 -05:00
871a66ebae
changed to use srs and complex division
2022-01-10 14:59:29 -05:00
1dea66d75c
Added authorize.conf.tar to make it possible to delete data from completed searches properly.
2022-01-05 15:26:43 -08:00
64abbe4458
Significant changes. Most importantly, instead of a hard-coded wait time after uploading data, we poll to make sure that all the data has been indexed. When this is complete, we move ahead with the test. While this costs extra in terms of CPU and Disk access, it usually allows us to go much faster since wait time is usually just a few seconds. Better tracking of time per test and time estimation.
2022-01-05 14:39:13 -08:00
8f0518b5af
spelling fix
2022-01-03 14:39:48 -05:00
7c1025425a
fixed spelling
2022-01-03 14:35:49 -05:00
50ba793358
fixing formatting
2022-01-03 12:20:42 -05:00
1e731ee7af
modify experimental detections to say not supported
2022-01-03 12:03:34 -05:00
eb48e000a8
Updated default app loadout for splunkbase.
...
Added splunk_ta_for_nginz,
splunk_security_essentials,
and ta_for_zeek.
2021-12-22 14:05:55 -08:00
dc3fc8296d
Added a config file to run tests against latest splunkbase apps.
...
Also, added slightly better error output for certain types
of test failures. Finally updated splunkbase app defaults in
validate_args, but they are commented out for now while we
use the S3 versions.
2021-12-22 13:02:21 -08:00
05821f4303
Fixed some parts of the CI that don't give a descriptive error message
...
in the output files when a search generates certain types of errors.
For example, an error where it cannot reach the Splunk endpoint
server. These are rare, but good to have.
2021-12-21 14:46:44 -08:00
47e155b747
Changed back to one container per
...
GH Action Machine config with the
mock option. Also, shuffling detections after they are put into a
list to distribute runtime and load as
much as possible.
2021-12-21 14:22:33 -08:00
5f972f734f
Re-push to test everything with 2 containers per GH Actions machine.
2021-12-21 11:00:11 -08:00
9ba7abd0e8
Fixed a small error. If you specified a PR number and a branch that did not exist, then it would create a branch and that would be bad. Now, when you specify a branch and a PR number, the branch MUST exist. If not, we fail and bail.
2021-12-21 10:44:50 -08:00
63555bc531
Added some more robust error handling to the high level test runner. It looks like we were getting errors pulling the image from docker hub - was it down...
2021-12-20 14:45:01 -08:00
d4aae32b3d
Removed Python 2 which was previously used for splunk packaging toolkit. Replaced with python3
2021-12-20 14:26:57 -08:00
264dbfb7f4
Fixed up and error that could occur where the environment is not properly set up if the users requested PERSIST_SECURITY_CONTENT, but the directory did not exist. Also fixed the default argument for this on GitHub Actions.,
2021-12-20 13:58:58 -08:00
d26032b912
Updated the github_actions config with S3 binary
...
paths.Updated the default args to include web and
experimental as possible folders with Web being a
default.
2021-12-20 13:38:51 -08:00
0e26dc76ff
Changed so that we no longer start a container if
...
we know that we will not have a test for it. For example,
if we try to start 4 containers by have only 2 tests at
the beginning we will only start 2 containers. This saves
a lot of startup time and resources. There is a descriptive
printout for this as well.
Also, bumped the maximum startup time for 6 minutes to
10 minutes. This, combined with the system info from
the previous commit should let us determine if we can
bump the number of containers per GitHub Actions
VM from 1 to 2 or more.
2021-12-20 13:17:46 -08:00
7230b5c63d
Print out some system usage information on each update.
...
This will help users dianose if their systems are overburdend
and also helps us figure out what the appropriate number
of containers to run on cloud infrastructure, like GitHub
Actions, may be without logging directly into the machine
doing the testing. In some cases, we can't log into those
machines by design. If you're running tests at home, you
can also just listen to the volume of your computer's fans.
2021-12-20 12:02:11 -08:00
cc8113ec86
Fixed output of manifest for detection_failure_manifest.json. Previously, it contained mostly default settings and most notably did not container the proper apps configuration. Also updated a few documentation strings.
2021-12-20 08:34:54 -08:00
Michael Hart