URGENT: two of the bdb databases does not download with action

[marvinm2]

From the gene.json file (https://data.bridgedb.org/gene.json), the following two .bridge files do not download:

• Ec_Derby_Ensembl_91.bridge
• Mx_Derby_Ensembl_85.bridge

Because of this, the Docker image does not work.

[egonw]

@tabbassidaloii, can you help out here? What is the status of these files?

@marvinm2, the JSON files have fields to indicate compatibility. I think we should add there the "tested with Docker" option, and just exclude these two files from that for now.

[tabbassidaloii]

@marvinm2 both Ec_Derby_Ensembl_91 and Mx_Derby_Ensembl_85.bridge are on Zenodo (https://zenodo.org/record/3667670 and https://zenodo.org/record/5638771/, respectively), and the links are correctly provided in gene.json file. I just tried and both are working.

[marvinm2]

I experience these issues when starting the service

org.bridgedb.IDMapperException: Could not connect to database: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:64)
	at org.bridgedb.rdb.IDMapperRdb$DriverPgdb.connect(IDMapperRdb.java:57)
	at org.bridgedb.BridgeDb.connect(BridgeDb.java:54)
	at org.bridgedb.rdb.GdbProvider.fromConfigFile(GdbProvider.java:164)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.connectGdbs(RestletService.java:216)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.start(RestletService.java:116)
	at org.restlet.routing.Filter.start(Filter.java:265)
	at org.restlet.routing.Router.start(Router.java:814)
	at org.restlet.Component.startRouters(Component.java:626)
	at org.restlet.Component.start(Component.java:568)
	at org.restlet.Restlet.handle(Restlet.java:306)
	at org.restlet.Component.handle(Component.java:405)
	at org.restlet.Server.handle(Server.java:507)
	at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
	at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
	at org.restlet.engine.connector.HttpServerHelper$1.handle(HttpServerHelper.java:64)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:717)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:689)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:115)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:141)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:252)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2855)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:411)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:660)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:656)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.iapi.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:654)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:300)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:967)
	at org.apache.derby.iapi.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:136)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251)
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:59)
	... 24 more
Caused by: ERROR XJ040: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:170)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:75)
	... 38 more
Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1259)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:733)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:850)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:248)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:191)
	at org.apache.derby.impl.io.JarStorageFactory.doInit(JarStorageFactory.java:111)
	at org.apache.derby.impl.io.BaseStorageFactory.init(BaseStorageFactory.java:88)
	at org.apache.derby.impl.io.JarStorageFactory.init(JarStorageFactory.java:35)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.privGetStorageFactoryInstance(StorageFactoryService.java:222)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.access$400(StorageFactoryService.java:73)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:977)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:971)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.getCanonicalServiceName(StorageFactoryService.java:969)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1515)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:999)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:598)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4042)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4038)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(EmbedConnection.java:4036)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2816)
	... 35 more
Unable to start the Restlet
The org.restlet.Component class has no Restlet defined to process calls. Maybe it wasn't properly started.
Unable to start the Restlet

org.bridgedb.IDMapperException: Could not connect to database: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:64)
	at org.bridgedb.rdb.IDMapperRdb$DriverPgdb.connect(IDMapperRdb.java:57)
	at org.bridgedb.BridgeDb.connect(BridgeDb.java:54)
	at org.bridgedb.rdb.GdbProvider.fromConfigFile(GdbProvider.java:164)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.connectGdbs(RestletService.java:216)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.start(RestletService.java:116)
	at org.restlet.routing.Filter.start(Filter.java:265)
	at org.restlet.routing.Router.start(Router.java:814)
	at org.restlet.Component.startRouters(Component.java:626)
	at org.restlet.Component.start(Component.java:568)
	at org.restlet.Restlet.handle(Restlet.java:306)
	at org.restlet.Component.handle(Component.java:405)
	at org.restlet.Server.handle(Server.java:507)
	at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
	at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
	at org.restlet.engine.connector.HttpServerHelper$1.handle(HttpServerHelper.java:64)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:717)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:689)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:115)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:141)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:252)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2855)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:411)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:660)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:656)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.iapi.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:654)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:300)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:967)
	at org.apache.derby.iapi.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:136)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251)
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:59)
	... 24 more
Caused by: ERROR XJ040: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:170)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:75)
	... 38 more
Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1259)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:733)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:850)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:248)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:191)
	at org.apache.derby.impl.io.JarStorageFactory.doInit(JarStorageFactory.java:111)
	at org.apache.derby.impl.io.BaseStorageFactory.init(BaseStorageFactory.java:88)
	at org.apache.derby.impl.io.JarStorageFactory.init(JarStorageFactory.java:35)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.privGetStorageFactoryInstance(StorageFactoryService.java:222)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.access$400(StorageFactoryService.java:73)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:977)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:971)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.getCanonicalServiceName(StorageFactoryService.java:969)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1515)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:999)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:598)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4042)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4038)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(EmbedConnection.java:4036)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2816)
	... 35 more
Unable to start the Restlet
The org.restlet.Component class has no Restlet defined to process calls. Maybe it wasn't properly started.

[marvinm2]

Are these files valid? They do not have QC notes

[egonw]

For Ec (which seems valid):

$ bash qc.sh  ~/Downloads/Ec_Derby_Ensembl_91.bridge ~/Downloads/Ec_Derby_Ensembl_91.bridge
INFO: old database is Ensembl 91 (build: 20180509)
INFO: new database is Ensembl 91 (build: 20180509)
INFO: Number of ids in T (GeneOntology): 3660 (unchanged)
INFO: Number of ids in En (Ensembl): 4140 (unchanged)
INFO: Number of ids in S (UniProtKB): 8241 (unchanged)
INFO: Attribute provided: Type
INFO: Attribute provided: Description
INFO: Attribute provided: Symbol
INFO: Attribute provided: Chromosome
INFO: new size is 12 Mb (changed +0.0%)
INFO: OLD database has a total number of identifiers of 16041
INFO: OLD database has a total number of mappings of 31933
INFO: OLD database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
INFO: NEW database has a total number of identifiers of 16041
INFO: NEW database has a total number of mappings of 31933
INFO: NEW database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
ERROR: 4140/4140 (100%) ids do not match expected pattern for Ensembl
ERROR: expected pattern is '^ENS[A-Z]*[FPTG]\d{11}$'
ERROR: aberrant ids are e.g. 'b4471', 'b1227', 'b4470', 'b1223', 'b1224', 'b1225', 'b1226', 'b1220', 'b1221', 'b1222'
ERROR: 4103/8241 (49%) ids do not match expected pattern for UniProtKB
ERROR: expected pattern is '^([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])|([O,P,Q][0-9][A-Z, 0-9][A-Z, 0-9][A-Z, 0-9][0-9])(\.\d+)?|([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])$'
ERROR: aberrant ids are e.g. 'UPI000012FD85', 'UPI000012FD86', 'UPI0000168087', 'UPI000012FD87', 'UPI00003B25FD', 'UPI000012FD88', 'UPI000016825D', 'UPI000012FD89', 'UPI000012FD90', 'UPI0001505B8D'

And Mx (also seems valid):

$ bash qc.sh  ~/Downloads/Mx_Derby_Ensembl_85.bridge ~/Downloads/Mx_Derby_Ensembl_85.bridge 
INFO: old database is Ensembl 1 (build: 20161018)
INFO: new database is Ensembl 1 (build: 20161018)
INFO: Number of ids in T (GeneOntology): 2106 (unchanged)
INFO: Number of ids in En (Ensembl): 4018 (unchanged)
INFO: Number of ids in S (UniProtKB): 7955 (unchanged)
INFO: Attribute provided: Type
INFO: Attribute provided: Description
INFO: Attribute provided: Symbol
INFO: Attribute provided: Chromosome
INFO: new size is 11 Mb (changed +0.0%)
INFO: OLD database has a total number of identifiers of 14079
INFO: OLD database has a total number of mappings of 26457
INFO: OLD database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
INFO: NEW database has a total number of identifiers of 14079
INFO: NEW database has a total number of mappings of 26457
INFO: NEW database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
ERROR: 4018/4018 (100%) ids do not match expected pattern for Ensembl
ERROR: expected pattern is '^ENS[A-Z]*[FPTG]\d{11}$'
ERROR: aberrant ids are e.g. 'Rv0753c', 'Rv1176c', 'Rv3716c', 'Rv2919c', 'Rv2918c', 'Rv1103c', 'Rv2349c', 'Rv2129c', 'Rv3432c', 'Rv3047c'
ERROR: 3978/7955 (50%) ids do not match expected pattern for UniProtKB
ERROR: expected pattern is '^([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])|([O,P,Q][0-9][A-Z, 0-9][A-Z, 0-9][A-Z, 0-9][0-9])(\.\d+)?|([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])$'
ERROR: aberrant ids are e.g. 'UPI00000D0FEC', 'UPI000012B714', 'UPI00000C1538', 'UPI00001650EB', 'UPI000012B72A', 'UPI000016527E', 'UPI000013944B', 'UPI00001652CE', 'UPI00000D4FCB', 'UPI00000D1117'

[egonw]

@marvinm2, I also not the Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge in the stacktrace. That suggests something else. Is this the correct stacktrace, or after you removed downloading it, but not before you removed it from the gdb.config?