lifting-bits-mcsema/tests/integration_tests/README.md

New attempt to create a better test suite. In development process, use at your own peril.

# Use:


1) You can use `populate.py` to copy binaries to be tested from `/bin` or `/usr/bin`.
Directory that contains configs is iterated and each binary that has present at least one config file is searched for.
In case it is not found error is written on standard output, but the script continues with the rest.

    Example:

    ```
    ./populate.py
     > not_exists not found anywhere
     > Found /bin/echo
     > Found /bin/grep
    ```

2) In case you want to try your own sources you can use `compile.py` which compiles all sources from `src` directory. If option `--stub_tags` is used, the corresponding tag files are created.

3) First create a batch of cfg file using `get_cfg.py` -- you can select tags to run only subset of all tests (they are specified in `tags` directory).
    + tag all will get cfg for every file with at least one config present
    + there are several policies that allows modification of already existing batches

    Example:

    ```
    # Every file present in the `bin` directory will be lifted
    # into cfg file.
    # If a batch with name first_batch is present it will be deleted.

    python get_cfg.py --disass dyninst --tags all
                      --batch first_batch --batch_policy D

    # Lifts all missing files that have tags C.
    # In case some file has already present cfg,
    # it is not replaced

    python get_cfg.py --disass dyninst --tags C
                      --batch first_batch --batch_policy C

    # Updates all files with tag echo, leaves rest of the batch untouched

    python get_cfg.py --disass dyninst --tags echo
                      --batch first_batch --batch_policy U
    ```

4) Once batch is created `run_tests.py` can be run.

# Config/Test files (tags/):

Directory tags (name to be changed) contains two types of files (beware, whitespaces are used as delimiters, therefore they matter):

1) `binary.kind.config`, which has following internal structure:

    ```
    TAGS: one two ...
    LIFT_OPTS: +one +two 87 !three
    ```

    `TAGS` specify tags of this config, while `LIFT_OPTS` represent specific lift options. Options prefixed by `+` are added and prefix `!` means that the options is not used even though it would normally be by default.
    One binary can have multiple config files.

    At the moment each config should contain exactly one of the following two tags: `c`, `cpp`. They are later used to determine which compiler to use when recompiling.

2) `binary.test`, which has following internal structure:

    ```
    TEST: cmd_option1 ...
    STDIN: -Fpath/to/file/ or string
    FILES: Not implemented yet
    TEST: ...
    ...
    ```

    For simple test cases it is easier to write this kind of test specification than the one used in python sources.

    `STDIN:` after file is optional and so is `FILES:`. If value of `STDIN:` is prefixed by F corresponding file is loaded (it can be relative to the root of test dir) and used as stdin.

# src/

Files presented in `src` are meant to be compiled by `compile.py` and can have special header:
```
/* TAGS: ... */
/* CC_OPTS: ... */
/* LD_OPTS: ... */
/* LIFT_OPTS: kind1 ... */
/* LIFT_OPTS: kind2 ... */
...
/* TEST: */
/* STDIN: */
...
```
Everything except `CC_OPTS:` and `LD_OPTS:` is used to generated appropriate `.config/.test` files. `CC_OPTS:` and `LD_OPTS:` are forwarded to the compiler.

# Complex tests

Not every test can be described by simple "language" of the `.test` files, therefore it is needed to have a way to define those more complex ones.
In `run_tests.py` a global array `g_complex_test` is present, which is used to store more advanced configurations. It has following data type:

```
{ str : [ TestDetails ] }
```
The structure is rather intuitive -- each binary has one entry and can have multiple test cases stored in array of `TestDetails`. `TestDetails` currently have following options:
```
cmd (set in __init___): array of command line arguments
files: files that are used by the program
       -> it is needed to copy them into appropriate location
check: files that are output of the program
       -> it is needed to compare them
```
As usual, files can be specified by relative paths from root directory of tests.



# Directory structure:

```
get_cfg.py # script to create new cfg files
run_tests.py # runs the selected batches
populate.py # tries to copy binaries from `/bin` or `/usr/bin` based on tags files
bin
  |- base_test
  |- ... rest of tested binaries

tags
  |- base_test.default.config
  |- ... for each binary at least one config file
  |- base.test
  |- .. optional test files

# Folders containing cfg files (there may be a reason you want to have several,
# for example compare frontends or have cfgs of some special flavor)
batch1_cfg
  |- base_test.cfg
  |- ...

batch2_cfg
  |- base_test.cfg
  |- ...

# Symlinks to shared libraries used by original binaries
# Links themselves are created by get_cfg.py
shared_libs
 |- libc.so
 |- ...

# Inputs for the run_tests.py
inputs
|- input1.txt
|- program1
|- ...
```