UM rose stem xc40 gnu rigorous fail with compiler internal error

[MichaelWhitall]

Hi folks,
I have a UM ticket where all the compiles in rose stem developer are succeeding (and runs look OK) except for:

fcm_make_xc40_gnu_um_rigorous_omp

which fails while compiling atm_step_4a.F90 (a file which I have modified), with a compiler internal error (see output copied below).

Has anyone else seen a problem like this, or know how I might be able to debug it? The only changes I've made in atm_step_4a (since last time I ran the tests on my branch and that compile was succeeding) are to change the argument lists of some subroutines called from atm_step_4a:
https://code.metoffice.gov.uk/trac/um/changeset/126025/main/branches/dev/michaelwhitall/vn13.6_delete_adv_arrays/src/control/top_level/atm_step_4A.F90?old=125987&old_path=main%2Fbranches%2Fdev%2Fmichaelwhitall%2Fvn13.6_delete_adv_arrays%2Fsrc%2Fcontrol%2Ftop_level%2Fatm_step_4A.F90

Cheers!
Mike

[FAIL] ftn -oo/atm_step_4a_mod.o -c -I./include -fdefault-integer-8 -fdefault-real-8 -fdefault-double-8 -std=f2008ts -Wtabs -I/projects/um1/lib/gnu-6.3.0/gcom/gcom-8.3/haswell/meto_xc40_gfortran_mpp/build/include -O0 -Wall -Wextra -ffpe-trap=invalid,zero -fbounds-check -Warray-bounds -fcheck-array-temporaries -finit-real=nan -fimplicit-none -Werror=character-truncation -fopenmp -I/projects/um1/lib/gnu-6.3.0/eccodes/eccodes-2.8.0/ivybridge/include /var/spool/jtmp/2606801.xcf00.H1IMg0/fcm_make_xc40_gnu_um_rigorous_omp.1.vn13.6_delete_adv_arraysCLYaUX/preprocess-atmos/src/um/src/control/top_level/atm_step_4A.F90 # rc=1
[FAIL] f951: internal compiler error: in read_module, at fortran/module.c:5090
[FAIL] 0x69390e read_module
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/module.c:5090
[FAIL] 0x69390e gfc_use_module
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/module.c:6980
[FAIL] 0x694fb6 gfc_use_modules()
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/module.c:7104
[FAIL] 0x6a17dc use_modules
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:114
[FAIL] 0x6a5574 decode_statement
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:332
[FAIL] 0x6a72d4 next_free
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:1143
[FAIL] 0x6a72d4 next_statement
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:1376
[FAIL] 0x6a823c parse_spec
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:3703
[FAIL] 0x6aa743 parse_progunit
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:5487
[FAIL] 0x6aabdc parse_contained
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:5390
[FAIL] 0x6ab879 parse_module
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:5729
[FAIL] 0x6ac249 gfc_parse_file()
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/parse.c:6035
[FAIL] 0x6ed8d2 gfc_be_parse_file
[FAIL] ../../cray-gcc-6.3.0-201701050407.93fe37becc347/gcc/fortran/f95-lang.c:201
[FAIL] Please submit a full bug report,
[FAIL] with preprocessed source if appropriate.
[FAIL] Please include the complete backtrace with any bug report.
[FAIL] See http://gcc.gnu.org/bugs.html for instructions.
[FAIL] compile 0.3 ! atm_step_4a_mod.o <- um/src/control/top_level/atm_step_4A.F90
[FAIL] ! atm_step_4a_mod.mod : depends on failed target: atm_step_4a_mod.o
[FAIL] ! atm_step_4a_mod.o : update task failed

[MichaelWhitall]

In case anyone's following this or hits a similar problem in future; the xc40 gnu rigorous compiler internal error went away after running rose stem from scratch with --new AND -S PREBUILDS=false.

So I guess the build had got stuck with some nasty inconsistency with one of the files it had already compiled, and needed to be forced to recompile from scratch to get out of its rut?