it will cause the workflow to fail. In turn, this would
prevent a PR from being merged. However, this
lets a developer know more easily (without needing
to dive into the GitHub Actions logs) whether or
not their detections passed successfully. It also
upload an extra file called DetectionFailureManifest
which is a test file for ONLY the failed tests.
You can run this locally on your machine to
quickly, interactively debug the failures only.
the workflow to fail. An additional file is
also uploaded called "replicate_failures"
which allows the user to easily retest
just the failed tests locally.
(0) code even if there are failures or errors found
during testing during the initial testing of CI/CD.
This is because we don't want these to hold up
the build process. Eventually we will report actual
success/failure of all tests after we have the
opportunity to validate the new testing pipeline and
make updates to failing searches.
commit_hash to a warning. This was causing
an error during PR triggered testing in github
actions as both of these are provided. Perhaps
we actually can include both of these and can
get old PRs based on commit hash, but until
then we will just test using the PR number and
ignore the commit_hash if we find both.
Added the new_sysmon as an additional default application.
While it is not ideal to have the old and new sysmons installed
side by side, there are some detection that only work with one
and not the other. We will run both until we have moved
all detections to the new sysmon.
Also, added slightly better error output for certain types
of test failures. Finally updated splunkbase app defaults in
validate_args, but they are commented out for now while we
use the S3 versions.
in the output files when a search generates certain types of errors.
For example, an error where it cannot reach the Splunk endpoint
server. These are rare, but good to have.
GH Action Machine config with the
mock option. Also, shuffling detections after they are put into a
list to distribute runtime and load as
much as possible.
we know that we will not have a test for it. For example,
if we try to start 4 containers by have only 2 tests at
the beginning we will only start 2 containers. This saves
a lot of startup time and resources. There is a descriptive
printout for this as well.
Also, bumped the maximum startup time for 6 minutes to
10 minutes. This, combined with the system info from
the previous commit should let us determine if we can
bump the number of containers per GitHub Actions
VM from 1 to 2 or more.
This will help users dianose if their systems are overburdend
and also helps us figure out what the appropriate number
of containers to run on cloud infrastructure, like GitHub
Actions, may be without logging directly into the machine
doing the testing. In some cases, we can't log into those
machines by design. If you're running tests at home, you
can also just listen to the volume of your computer's fans.
apps stored on S3. Pointed at the PATCHED linux_sysmon.
Added a slight delay in between starts of containers for
performance reasons. Enabled better handling and ability
to download http_path local apps to a folder instead of
passing them in as string for the container to download.
everything smoother. The largest differences
are improving detection of containers that
take too long to start (or that crash while)
they are starting by adding a timeout. That
timeout is set to 360 seconds and might need
to be tuned in the future to a larger number
since this was tested on a fast machine with
fast network. The other large change is an
initial pass at updating and committing
detections that have passed the test back
to the repo. This still needs a lot of testing and
refinement.
pyth0n1c