crash-testing
=============

Crash-testing is a tool used for test crash utility.

Quick start:
------------

The main idea of crash-testing, is to invoke crash utility against a list of vmcores,
as well as a list of crash utility commands, then collect and analyze the output logs
of crash utility, to identify whether there are unexpected errors and regressions.

1) Prepare vmcores.
The more quantity and types of the vmcores, the more chance to find crash utility
bugs. So collect vmcores as many as possible! If you have a list of vmcores collected,
the ideal directory structure to arrange is:

my_vmcores
 |_ x86
 |_ x86_64
 |  |_ 3.13.96
 |  |  |_ vmcore1
 |  |  |_ vmcore2
 |  |  |_ vmlinux
 |  |_ 4.14.2
 |     |_ vmcore
 |     |_ vmlinux.gz
 |_ aarch64
 |_ ppc
 |_ etc

2) Index crash utility commands.
Commandlist is used to index the commands which will be passed to crash utility to generate
output logs. You can index all commands to have a comprehensive testing, or some specific commands
to have a quick testing. Please refer to the files within crash-testing/command_lists, which have
a similar format as above:

COMMANDLIST_START
command/bt
COMMANDLIST_END

command/bt is located in crash-testing/command. It contains 2 parts:
a. file bt itself
b. file bt.awk

The real crash utility commands is in bt file, after they are passed to crash utility, an
log filter, specified in bt.awk will filter out the generated logs, to identify whether
there are invalid logs. You can enrich command and corresponding awk file. For a simple testing,
you don't need to do that.

3) Invoke crash testing.
a. test for backward-compatibility, which will accepts many dumpfiles and do 
   sanity checks.

     $./crash_test.sh  \ 
        -D <path_of_my_vmcores>/x86_64  \
        -c command_lists/overall_short_output_list

   If -f is not specified, then automatically take $(which crash) by default.
   You will have /tmp/crash.log and /tmp/crash_filtered.log generated later.

b. test for patch review, which compares the outputs of the current crash and
   the patched crash.

     $./crash_test.sh  \
        -D <path_of_my_vmcores>/x86_64  \
        -c command_lists/overall_short_output_list  \
        -f ~/crash1  \
        -e ~/crash2
   You will have /tmp/crash.log and /tmp/crash_filtered.log, which are generated
   by -f, and /tmp/crash2.log and /tmp/crash2_filtered.log, which are generated
   by -e, and /tmp/crash_diff.log which is generated by /tmp/crash.log /tmp/crash2.log.

To accerlate crash-testing, you can add -u <NUM> to ./crash_test.sh, to create
NUM processes running concurrently.

5) Result analyzing
These files are crash-testing artifacts:
/tmp/crash.log (gzip compressed, all outputs preserved)
/tmp/crash_filtered.log (plain text, filtered by awk rules)
/tmp/crash2.log (gzip compressed, all outputs preserved)
/tmp/crash2_filtered.log (plain text, filtered by awk rules)
/tmp/crash_diff.log (plain text, generated from /tmp/crash.log and
                     /tmp/crash2.log by analyze_log_diff.sh)

The format of the artifacts is like:

[Test 1]                                  ----> vmcore index number
[Dumpfile dir1/vmlinux.gz dir1/vmcore]    ----> vmcore path  
/usr/bin/crash -s dir1/vmlinux.gz dir1/vmcore   ----> crash command line
[Command command/alias: alias]            ----> testing command 1
....                                      ----> command 1 output
[Command command/ascii: ascii]            ----> testing command 2
....                                      ----> command 2 output

There are 3 stages of result analyzing:

1) View /tmp/crash_filtered.log and /tmp/crash2_filtered.log directly. These logs
didn't pass the awk filter check, are likely to be an indication of crash utility
bugs. Further more, you can diff /tmp/crash_filtered.log and /tmp/crash2_filtered.log,
the differentials are more likely to be a regression.

2) If you have a long list of vmcore dumplists, you can invoke the following cmd to
have a quick view of log diff. The differentials are more likely to be a regression.
$ ./analyze_log_diff.sh -c /tmp/crash_diff.log | less

3) If you have a short list of vmcore dumplists, you can directly view the differentials
by the following command. You can view the detailed differentials to find out the regression.
$ cat /tmp/crash_diff.log | less

Advanced usage options:
-----------------------

A. Specify only one command file for testing, not a list of command files:
    ./crash_test.sh -d dumplist.txt -D <path_of_my_vmcores>/x86_64 -b 
    command/my_command

B. Start multi-threads(-u N) to acceralate testing. (-u 4) will start 4 threads 
   running at the same time. It can shorten the overall time if you have a long
   list of vmcores.
     $./crash_test.sh -d dumplist.txt -D <path_of_my_vmcores>/x86_64 -c
      command_lists/my_command_list -u 4

C. You can exit by Ctrl+c whenever, and no logs will be preserved.

D. Pass extra args(-o) to crash when testing. Also for crash2 as well if -e 
   exits:
     $./crash_test.sh -d dumplist.txt -D <path_of_my_vmcores>/x86_64 -c
      command_lists/my_command_list -o "--kaslr auto"

E. Local live testing:
   1) install kernel-debuginfo first
   2) invoke crash-testing:
     $./crash_test.sh -a -d dump_lists/live_list -c command_lists/overall_short_output_list