DeepVariant Step Error: PythonNext() argument variant is not valid: Dynamic cast failed

[weilu1998]

Hi,

I was following the singularity quickstart and encountered an error at the DeepVariant Step. Here is the 5.1_DeepVariant_SNP.log. I used singularity/3.7.1 on a gpu node (operating system: CentOS Linux release 8.4.2105).

I1024 11:40:46.918040 140703382533952 run_deepvariant.py:341] Creating a directory for intermediate results in /home/weilu1/scratch/nanopore/pepper/output/dv_intermediate_outputs/
I1024 11:40:46.918628 140703382533952 run_deepvariant.py:362] You set --customized_model. Instead of using the default model for WGS, call_variants step will load /opt/dv_models/ont_deepvariant_vc/dv_ont_r9_guppy5_sup_vc_model_snp* instead.

***** Intermediate results will be written to /home/weilu1/scratch/nanopore/pepper/output/dv_intermediate_outputs/ in docker. ****


***** Running the command:*****
time seq 0 0 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "/home/weilu1/scratch/nanopore/pepper/input/data/GRCh38_no_alt.chr20.fa" --reads "/home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PHASED.PEPPER_MARGIN.haplotagged.bam" --examples "/home/weilu1/scratch/nanopore/pepper/output/dv_intermediate_outputs/make_examples.tfrecord@1.gz" --add_hp_channel --alt_aligned_pileup "none" --min_base_quality "1" --min_mapping_quality "5" --parse_sam_aux_fields --partition_size "10000" --proposed_variants "/home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PEPPER_VARIANT_OUTPUT_VARIANT_CALLING_SNPs.vcf.gz" --norealign_reads --regions "chr20:1000000-1020000" --sample_name "Sample" --sort_by_haplotypes --variant_caller "vcf_candidate_importer" --task {}

I1024 11:40:48.610073 140135261120320 make_examples_core.py:239] Preparing inputs
I1024 11:40:48.611205 140135261120320 genomics_reader.py:222] Reading /home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PHASED.PEPPER_MARGIN.haplotagged.bam with NativeSamReader
I1024 11:40:48.611673 140135261120320 make_examples_core.py:239] Common contigs are ['chr20']
I1024 11:40:48.612054 140135261120320 make_examples_core.py:239] Reading VCF to skip processing some regions without variants in the --proposed_variants VCF.
I1024 11:40:48.612589 140135261120320 genomics_reader.py:222] Reading /home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PEPPER_VARIANT_OUTPUT_VARIANT_CALLING_SNPs.vcf.gz with NativeVcfReader
2022-10-24 11:40:48.616673: W ./third_party/nucleus/util/proto_clif_converter.h:75] Failed to cast type N6google8protobuf14DynamicMessageE
Traceback (most recent call last):
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 180, in <module>
    app.run(main)
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/absl_py/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/absl_py/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 170, in main
    make_examples_core.make_examples_runner(options)
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 1626, in make_examples_runner
    regions = processing_regions_from_options(options)
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 1492, in processing_regions_from_options
    variant_positions = fetch_vcf_positions([
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 568, in fetch_vcf_positions
    for variant in vcf_reader.query(region):
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/third_party/nucleus/io/clif_postproc.py", line 82, in __next__
    record, not_done = self._raw_next()
  File "/scratch/local/jobs/4725551/Bazel.runfiles__ofnv2t8/runfiles/com_google_deepvariant/third_party/nucleus/io/clif_postproc.py", line 149, in _raw_next
    not_done = self._cc_iterable.PythonNext(record)
RuntimeError: PythonNext() argument variant is not valid: Dynamic cast failed
parallel: This job failed:
/opt/deepvariant/bin/make_examples --mode calling --ref /home/weilu1/scratch/nanopore/pepper/input/data/GRCh38_no_alt.chr20.fa --reads /home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PHASED.PEPPER_MARGIN.haplotagged.bam --examples /home/weilu1/scratch/nanopore/pepper/output/dv_intermediate_outputs/make_examples.tfrecord@1.gz --add_hp_channel --alt_aligned_pileup none --min_base_quality 1 --min_mapping_quality 5 --parse_sam_aux_fields --partition_size 10000 --proposed_variants /home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PEPPER_VARIANT_OUTPUT_VARIANT_CALLING_SNPs.vcf.gz --norealign_reads --regions chr20:1000000-1020000 --sample_name Sample --sort_by_haplotypes --variant_caller vcf_candidate_importer --task 0

real    0m1.905s
user    0m1.588s
sys     0m0.253s
```

[kishwarshafin]

@weilu1998 ,

Sorry for being so late to reply here, can you please look at the /home/weilu1/scratch/nanopore/pepper/output/intermediate_files/PEPPER_VARIANT_OUTPUT_VARIANT_CALLING_SNPs.vcf.gz file to see if there are any variants in there? I think the VCF is empty and that's why you are getting this error.

[weilu1998]

@kishwarshafin Sorry for the late reply. This error is solved by adding --no-home option. I do see different false positive results for the quick start guide.

INFO:    Converting SIF file to temporary sandbox...
2022-11-14 18:52:03,104 WARNING  No reference file found at default locations. You can set the environment variable 'HGREF' or 'HG19' to point to a suitable Fasta file.
Hap.py
[I] Total VCF records:         48
[I] Non-reference VCF records: 48
[I] Total VCF records:         50
[I] Non-reference VCF records: 48
2022-11-14 18:52:05,280 WARNING  No calls for location chr20:1000000-1020000 in query!
2022-11-14 18:52:05,280 WARNING  Creating template for vcfeval. You can speed this up by supplying a SDF template that corresponds to /scratch/midway3/weilu1/nanopore/pepper/input/data/GRCh38_no_alt.chr20.fa
Benchmarking Summary:
Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.Precision  METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
INDEL    ALL            2         2         0            3         0          1      0      0            1.0               1.0        0.333333              1.0                     NaN                     NaN                   0.000000                   0.000000
INDEL   PASS            2         2         0            3         0          1      0      0            1.0               1.0        0.333333              1.0                     NaN                     NaN                   0.000000                   0.000000
  SNP    ALL           39        39         0           45         0          6      0      0            1.0               1.0        0.133333              1.0                1.785714                     2.0                   0.444444                   0.363636
  SNP   PASS           39        39         0           45         0          6      0      0            1.0               1.0        0.133333              1.0                1.785714                     2.0                   0.444444                   0.363636

[kishwarshafin]

@weilu1998 , can can be caused by platform difference as pepper operates on ONNX machine that sometimes causes determinism issues.