Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test updates for NCBI BLAST DB v5 #2863

Closed
peterjc opened this issue May 6, 2020 · 11 comments
Closed

Test updates for NCBI BLAST DB v5 #2863

peterjc opened this issue May 6, 2020 · 11 comments
Labels
Update Update for data files, e.g., codon tables, restriction enzymes etc.

Comments

@peterjc
Copy link
Member

peterjc commented May 6, 2020

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959587

> test_NCBI_BLAST_tools ... /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ blastn and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ blastp and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -ipglist,-negative_ipglist,-negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ blastx and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -ipglist,-negative_ipglist,-negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ deltablast and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ makeblastdb and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -blastdb_version)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ psiblast and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -ipglist,-negative_ipglist,-negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ rpsblast and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -sorthits,-sorthsps,-subject_besthit)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ rpstblastn and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -sorthits,-sorthsps)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ tblastn and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py:398: UserWarning: NCBI BLAST+ tblastx and Biopython out sync. Please update Biopython, or report this issue if you are already using the latest version. (Extra args: ; Missing: -negative_taxidlist,-negative_taxids,-sorthits,-sorthsps,-subject_besthit,-taxidlist,-taxids)
>   warnings.warn("NCBI BLAST+ %s and Biopython out sync. Please "
> FAIL

> ======================================================================
> FAIL: test_fasta_db_nucl (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with nucleotide database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 248, in test_fasta_db_nucl
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.fna.nsd"))
> AssertionError: False is not true
> 
> ======================================================================
> FAIL: test_fasta_db_prot (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with protein database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 213, in test_fasta_db_prot
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.faa.psd"))
> AssertionError: False is not true

@tillea wrote:

per https://www.ncbi.nlm.nih.gov/books/NBK131777/, the upgrade to
2.10.0 changed the default database version (controlled by makeblastdb's
-blastdb_version flag) from 4 to 5, causing some output files to change
names (and formats):

*.nsd -> *.nnd
*.nsi -> *.nni
*.psd -> *.pnd
*.psi -> *.pni

Related to this, there are new switches to add to the command line wrappers (shown as warnings from the tests above)
@peterjc peterjc added the Update Update for data files, e.g., codon tables, restriction enzymes etc. label May 6, 2020
@peterjc peterjc mentioned this issue May 6, 2020
Closed
peterjc added a commit to peterjc/biopython that referenced this issue May 14, 2020
@peterjc
Cope with BLAST DB v5 extensions
e63ced0 
With NCBI BLAST+ v2.10.0 the v5 DB became
the default, and it has slightly different
file extensions to their v4 DB format.

See GitHub issue biopython#2863
@peterjc peterjc mentioned this issue May 14, 2020
Merged
3 tasks
peterjc added a commit that referenced this issue May 15, 2020
@peterjc
Cope with BLAST DB v5 extensions
c52a2b5 
With NCBI BLAST+ v2.10.0 the v5 DB became
the default, and it has slightly different
file extensions to their v4 DB format.

See GitHub issue #2863
@peterjc
Copy link
Member Author

peterjc commented May 15, 2020

Test failure due to new extensions fixed in #2801

Closing issue, see #2184 for the changes to the command line options.

@peterjc peterjc closed this as completed May 15, 2020
@peterjc
Copy link
Member Author

peterjc commented May 16, 2020 •
edited

@tillea reported by email that even with the patch it fails on 32bit systems:

 ======================================================================
> FAIL: test_fasta_db_nucl (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with nucleotide database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 249, in test_fasta_db_nucl
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.fna.nhd"))
> AssertionError: False is not true
>
> ======================================================================
> FAIL: test_fasta_db_prot (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with protein database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 208, in test_fasta_db_prot
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.faa.phd"))
> AssertionError: False is not true

P.S. We're looking into the possibility that makeblastdb itself is failing (since these are the first files we check from the expected output).

@peterjc peterjc reopened this May 16, 2020
@mdehoon
Copy link
Contributor

mdehoon commented May 23, 2020

@chebizarro @peterjc Is this still a blocking issue for release 1.77?

@peterjc
Copy link
Member Author

peterjc commented May 23, 2020

Possibly - I suspect it is a Debian/Blast issue though.

@mdehoon
Copy link
Contributor

mdehoon commented May 23, 2020

Then does it make sense to wait for this? Especially since we are considering to drop Bio.Application.

@peterjc
Copy link
Member Author

peterjc commented May 23, 2020

It works on the major CPUs anyway - seems low risk to proceed with our release.

@tillea
Copy link

tillea commented May 23, 2020 via email

@chebizarro
Copy link
Contributor

I built a Docker container based on the i386/ubuntu 18.04 image and can't seem to replicate the problem using the i386 build of makeblastdb 2.6.0. I can try different base image/blast+ combinations to try and pin down the error but that won't be until next week.

@peterjc
Copy link
Member Author

peterjc commented May 24, 2020

Thanks Chris, that is reassuring - less likely one of our main target demographic will hit this.

@emollier emollier mentioned this issue Jun 20, 2020
Merged
3 tasks
@peterjc
Copy link
Member Author

peterjc commented Nov 10, 2020

This would be solved by #3344

@peterjc
Copy link
Member Author

peterjc commented Nov 10, 2020

Clarification: The warning about the BLAST+ wrappers being out of sync would be fixed by dropping them.

The error about V5 databases was already fixed, as as the -blastdb_version flag.

Closing.

@peterjc peterjc closed this as completed Nov 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Update Update for data files, e.g., codon tables, restriction enzymes etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@peterjc @mdehoon @chebizarro @tillea