Organizing resources for dev HPRC-only workspace.

[samuelklee]

Given the AoU workspace freeze, I will create a fresh HPRC-only workspace.

@hangsuUNC @fabio-cunial I will gather pointers for the following and update below, can you either verify they are correct or beat me to posting them?

1. short-variant hg38 joint callset (ideally one created with an HPRC-only run, if that exists; otherwise the full AoU+HPRC callset is fine, and I can do the subsetting to HPRC)

2. TPR = 0.7 kanpig integrated SV hg38 callset (again, HPRC-only would be ideal, as long as it's up to date, otherwise happy to subset)

3. any resources or tracks needed for vcfdist, e.g. dipcall truth

It will also be useful to have pointers to

4. HPRC SR BAMs

which will eventually be needed when SR genotyping is integrated into the end-to-end pipeline in the next push (again, let's focus on LR phasing + bubble creation first).

[samuelklee]

1. short-variant hg38 joint callset

   From the joint_gvcf field in the Jointcall_dvcf AoU-HPRC-Integratedcall-hg38_set entity in the https://app.terra.bio/#workspaces/allofus-drc-wgs-lr-prod/AoU_DRC_WGS_LongReads_PacBio data table: gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/outputs/HPRC_8X/Hg38/JointCallGVCFs/Jointcall_dvcf/Jointcall_dvcf.g.vcf.bgz

   @hangsuUNC can you provide a brief description of how this was generated?

2. TPR = 0.7 kanpig integrated SV hg38 callset

   kanpig: gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/cunial_intersample_vcf/grch38/kanpig/cal_sens_07/concat.vcf.gz
   kanpig regenotyped: gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/cunial_intersample_vcf/grch38/kanpig/cal_sens_07/concat_regenotyped_kanpig_10k.vcf.gz

   Now going back through Slack and Terra job histories, I am confused about which of these two callsets we should be using. @hangsuUNC your latest runs (e.g., https://app.terra.bio/#workspaces/allofus-drc-wgs-lr-prod/AoU_DRC_WGS_LongReads_PacBio/job_history/ef713aee-c946-4a9f-bbcb-e179ef51ca70) appear to use the former, and @fabio-cunial it appears that you tried to run the latter but that it was a mistake to do so (https://app.terra.bio/#workspaces/allofus-drc-wgs-lr-prod/AoU_DRC_WGS_LongReads_PacBio/job_history/2605f9a6-3202-4bfc-b4ed-f4b1e9dcfc21)?

   It does look like there are no remaining inconsistencies in the latter! Which is what I would expect from a regenotyping step. So perhaps this changes some conclusions we've reached recently, but I would guess it doesn't completely solve everything given the rate of inconsistencies in the former is already low (hopefully it helps for HiPhase, but I would be skeptical if it completely resolves things for Shapeit). But if it does, then good---we are done!

   Perhaps @fabio-cunial can provide some brief descriptions of these two callsets and their differences here?

3. any resources or tracks needed for vcfdist, e.g. dipcall truth

   {
       "VcfdistEval.bed_file":"gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/hangsu/GRCh38_notinAllTandemRepeatsandHomopolymers_slop5.bed",
       "VcfdistEval.docker":"timd1/vcfdist:2.5.3",
       "VcfdistEval.fai":"gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/references/GRCh38_noalt/GCA_000001405.15_GRCh38_no_alt_analysis_set.fa.fai",
       "VcfdistEval.fasta_file":"gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/references/GRCh38_noalt/GCA_000001405.15_GRCh38_no_alt_analysis_set.fa",
       "VcfdistEval.truth_vcf":"gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/cunial_raw_dipcall_merge/merged.vcf.gz"
   }
   

[hangsuUNC]

it seems that I don't have access to this workspace...

[samuelklee]

Just shared!

[samuelklee]

Fabio has confirmed that gs://fc-bb12940d-f7ba-4515-8a98-42de84f63c34/HPRC_grch38/filtrations/kanpig_07.vcf.gz is indeed the correct kanpig intrasample + truvari intersample HPRC-only callset, see https://docs.google.com/presentation/d/1EYjhJsCeKrNJ0tkiFP18mISb6KP_eGwkIJWfilyTe4U/edit#slide=id.g2c6cb2ea978_1_0 for details. I have copied it to gs://fc-c6ae4a57-f44e-4d6c-befd-c48a2abcecc7/resources/. I believe this is the analog of the AoU+HPRC callset gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/cunial_intersample_vcf/grch38/kanpig/cal_sens_07/concat.vcf.gz

Not sure if we have an analog of the kanpig-intersample-regenotyped AoU+HPRC callset gs://fc-secure-f7d80b48-be60-426f-aa6b-f037a1bf7f34/scratch/cunial_intersample_vcf/grch38/kanpig/cal_sens_07/concat_regenotyped_kanpig_10k.vcf.gz yet.

[samuelklee]

The short variant joint callset run completed (https://app.terra.bio/#workspaces/broad-firecloud-dsde/lrma-aou1-panel-creation-hprc-only/job_history/0cf81519-454e-4714-8f59-2cbfd644bc4b) and the result gs://fc-c6ae4a57-f44e-4d6c-befd-c48a2abcecc7/submissions/0cf81519-454e-4714-8f59-2cbfd644bc4b/LRJointCallGVCFs/e3b31931-2a63-45a4-87ff-33bfd99d188a/call-JointCall/JointCall/711a00f5-e24f-4638-afe7-0887b8f96888/call-ConcatBCFs/HPRC.DV.joint.g.vcf.bgz* has been copied to gs://fc-c6ae4a57-f44e-4d6c-befd-c48a2abcecc7/resources/.

Now that provenance of the files in this resources bucket has been established, I think it would be best to point to them as inputs for any runs, rather than other possible sources. That way 1) we don't have to worry about the state of any finalized outputs, 2) we don't need to rely on the state of the data model, 3) we don't need to refer to files in other workspaces/buckets, and 4) we don't have to worry about blowing away outputs of any submissions.

@hangsuUNC @fabio-cunial please take note and follow this convention.

[hangsuUNC]

Cool! Thanks for organizing this!

[samuelklee]

Scripts for PanGenie bubble/panel creation have been added to gs://fc-c6ae4a57-f44e-4d6c-befd-c48a2abcecc7/resources/pangenie-run-from-callset/. These are from https://github.com/eblerjana/pangenie/releases/tag/v3.1.0.

[samuelklee]

@kvg did I ever get those pointers to the HPRC SR BAMs? Will be needed for #28 and #29.

[samuelklee]

(I have a few of the samples that overlap with HGSVC2 somewhere in a bucket that I can use to start, but would be nice to be organized from the get go!)

[samuelklee]

Comments in the status slides from @kvg:

https://anvil.terra.bio/#workspaces/anvil-datastorage/AnVIL_HPRC
Alternatively (with a bit less noise): https://app.terra.bio/#workspaces/broad-firecloud-dsde-methods/hidive/data
The latter links to the official HPRC AnVIL workspace. The short read CRAMs are in the "sample" table, "child_ilmn" column.
The AnVIL workspace is a requestor-pays bucket. You should be able to use either the dsde-methods or broad-dsp-lrma Google projects to download the files.

Will find an appropriate place for these, update the data table, and then report back.

[samuelklee]

Added the CRAMs in the hidive workspace to the hprc_grch38 table at https://app.terra.bio/#workspaces/broad-firecloud-dsde/lrma-aou1-panel-creation-hprc-only/data. I could not access the raw_data folder in the AnVIL_HPRC workspace.

[samuelklee]

@kvg is the hidive workspace missing HG002?