v496 branch

• Many updates to the faceted composite UI: removed local storage use, began using our server as a proxy for fetching metadata files for hubs on CORS-enabled servers (also local gbdb files), added collapsible facets and UI chips for selected values, and dynamically updated facet counts
• Updated seven big* utilities (bigBedToBed, bigMafToMaf, bigWigToWig, bigWigToBedGraph, bigGenePredToGenePred, bigPslToPsl, bigChainToChain) with support for multiple -range parameters that validate BED and position values via a new callback in bigBedCmdSupport.c. Improved error messages for chromStart/chromEnd boundary violations. Added abort behavior when coordinates exceed chromSize, with a new -skipChromCheck option to bypass both chrom name and size validation. Fixed undetected integer overflow issues; now warns when coordinate values are too large rather than silently failing. Expanded coordinate handling in utils and lib to use the full unsigned integer range for chromStart/chromEnd, consistent with the BED standard and .2bit chrom support. Fixed minor overflow in bigBed.c that could exceed the 4 GB address boundary. Changes originated in bigBedToBed.c and were refactored into the six other utilities
• Added new trackDb statement to run track-specific JavaScript plotting code using data from bigBed extra fields; syntax is detailsScript..
• Calculation of the GC percent track on-the-fly in the browser to eliminate the disk space waste of the bigWig files for this track
• Corresponding changes in hgc and hgTrackUi to tolerate the GC on-the-fly track
• hgSearch merges search results by track name on the client side, preventing a display bug in the category tree and merging results from different findSpecs for the same track into a single result set
• Save position range searches to the recent searches list on hgGateway/hgSearch/hgTracks
• Hide composite when all subtracks are hidden
• Hide superTrack when all subtracks are hidden
• hubCheck: added validation of sequence names in 2bit files for illegal characters; names must match [A-Za-z0-9._-], start with a letter or digit, and be at most 254 characters. Added a -noSeqNameCheck flag to disable the check
• hubCheck: added detection of tracks that list themselves as their own parent, which previously caused an infinite loop
• hgConvert no longer scores all toDb chains; instead, it selects a default one if the toDb CGI variable is already set
• Force Genark autocomplete results to send their genome identifier, like GC[AF]_XXX, as the db field to the hgGateway back end
• hgTracks' "download data in window" feature now always includes track headers
• Fixed a bug in the hgGateway search box to eliminate HTML markup characters in the resulting search answer string that fills the input search form
• Added the ability to read 2bit files for the faSize command, and added test cases to test the faSize command
• Added the ability to read 2bit files for the faCount command, and added test cases to test the faCount command
• Added a 'genome' argument to the 'ucscGenomes' endpoint in hubAPI
• Fixed a crash that occurred when removing the only highlight present, leading to an empty "highlight" cart variable
• When quickLifting decipherSnvs, use source coordinates to query into the supporting table
• Excluded bigRmsk from canWiggle() to prevent a crash when forceWiggle is triggered
• bigGuessDb now supports VCF files; run it with any bigBed, bigWig, or VCF file to output the assembly name
• Fixed mouseOver wiggles not showing up on page load
• Fixed a problem with composite tracks that have only one child and noParentConfig enabled
• hubCheck: added a warning when track names contain periods
• Added refSeqComposite to the quickLift vetted tracks list and fixed the label checkbox cart variable to use trackHubSkipHubName so it matches regardless of hub name prefix
• Fixed quickLiftChain tracks to not wiggle
• Added genePredCompare, a small tool for comparing two genePred files
• Split the varFreqs superTrack docs page into one page per subtrack

v495_branch.2

New display for superTracks in hgTrackUi (#36917). Max
Fixed ideogram range parsing when clicking on the ideogram (#37115). Chris
Removed the limit of 4096 highlights being active at once (#36895). Jonathan
Switching green polyA-color to default off for BAM tracks (#37133). Max
Added an hg.conf-defined 'popular' species list to the search bar as default results on focus, combined with the recents list. Added a chevron indicator next to the search bar. Added popular species support to hgConvert, hgIntegrator, and hgSearch. For hgConvert, filtered the popular genomes list by available liftOver chains and disabled unavailable entries (#36232). Chris
Fixed the hubApi JSON output for columnTypes when more than one track is requested with /getData/track. Added an option in the hgTracks view menu to include column headers when downloading track data. Changed wig table output to behave like bedGraph (chrom, start, stop, value keyed by track name) instead of having a top-level chromosome key. Limited track download requests to 100 tracks to avoid hubApi timeouts (#36858, #36802). Chris
Fixed saving UCSC databases to the 'Genomes' menu bar recents list when adding genomes from outside hgGateway. Corrected URLs for GenArk hubs. Made menu bar assembly link clicks add to the recents list. Improved labels for recent genomes in the menu bar and excluded default assemblies (hg38/hg19/mm10/mm39/hs1) that are already present. Fixed handling of non-GenArk assembly hub links to include hubUrl and hub_id. Handled public assembly hubs without BLAT/PCR servers by submitting the form so the backend can display an appropriate message (#36535). Chris
Refactored chromosome position parsing in hgFind into its own function for reuse. Added a defaultPosition check in hubCheck that verifies the requested chromosome exists for an assembly hub (#37126). Chris
Adding the genePredCompare tool to the kent repo: compare two genePred files and summarize differences, originally for ncbiRefSeq against refGene, but can be used for any genePred file; useful, e.g., to compare lifted genePreds with native genePreds. Max
Libified the API key set/revoke methods in hubSpaceKeys so they can be called from other modules (#36517). Chris
Fixed a crash when the trackDb cache is enabled and one of the tables listed in hg.conf's trackDb setting doesn't exist. Added sqlMayConnect/sqlMayConnectRemote to jksql so hdb.c can gracefully handle a missing database instead of aborting. Brian
Fixed the hgc detail page for ClinVar submissions track items on chrom index 0 in the bigBed numbering scheme. First fixed an off-by-one, then fixed caching of lastChromId so it doesn't incorrectly skip lookups (#37138). Brian
Wrapped the printing of each hub in search results with errCatch so that an error in one hub doesn't derail the entire search results page (#37100). Brian
In the right-click menu, fixed "zoom to transcript" to use the built-in URL parser instead of a regex for parsing coordinates, allowing chromosome names with dots as in GenArk assemblies (#35905). Chris
Added warning in the download dialog when no tracks are selected (#36802). Chris
Made the 'cf' keyboard shortcut on hgTracks work even when no track list is present below the image (#34456). Chris
Fixed species search bugs: removed duplicate hidden form variable in hgConvert, made organism comparison case-insensitive so 'human' matches 'Human' for GenArk results (#36535, #36232). Chris
Fixed up erratic assembly hub connection failures in the API. Hiram
Making tableBrowser off warning show up on Table Browser (#37168). Max
Changed the default trackDb cache directory to /dev/shm/trackDbCache (#35945). Brian
Removed an unused forward declaration in apiUtils.c (#37085). Brian
Added tutorial for building heatmap tracks (#36176). Jonathan
Updated seven big* utilities (bigBedToBed, bigMafToMaf, bigWigToWig, bigWigToBedGraph, bigGenePredToGenePred, bigPslToPsl, bigChainToChain) with support for multiple -range parameters that validate bed and position values, via a new callback function in bigBedCmdSupport.c. Added the ability to quickly check allowed chromosomes so users see an error for incorrect chromosome names, and a new -skipChromCheck option to skip the check. Works with all filter options (#28109, #36925). Galt
All seven toBig and bigTo utilities now properly run their inverse utilities, recreating the original input and verifying correctness. Added the bigWigToWig call to the wigToBigWig test, the bigChainToChain call to the chainToBigChain test, and the genePredToBigGenePred call to the bigGenePredToGenePred test (#28109, #36925). Galt

v494_branch.1

• Added a species search bar to most CGIs and a slight reorganization of hgGateway: new species search bar instead of dropdowns on most cgis. The search bar uses the hubApi to find genomes, allowing connecting to genark assemblies without going through hgGateway or hgHubConnect first. Hide the species tree by default on hgGateway. Save the last 10 used assemblies to localStorage, display them on hgGateway, and set them as default selections in the search bar and the menu bar under 'Genomes'. Chris
• Search changes: special code to add item descriptions to MANE and HGNC search results, need a generic way to do this for bigBeds, fix some bugs with hgSearch as a result of the species selection changes. Chris
• Make users confirm they want to disconnect from a hub on hgTracks, as the button is right next to refresh and can/has been clicked accidentally. Chris
• Fixed assembly hub disconnect bug to also clear the db, org, and genome cart variables, otherwise going back to hgTracks from hgHubConnect results in viewing the genome for the first row of the defaultDb table. Chris
• 'GenArk archive' name changes to 'GenArk repository'. Hiram
• Add bold highlight to new genome search result menu. Hiram
• Adding hub email display to hub error output for public hubs. Max
• Hi-C tracks in arc mode previously included extra whitespace at the top, which became excessive when many arcs were hidden. The whitespace has been removed, and the height of these tracks is now just enough to accommodate the highest arc shown.. Jonathan
• Added HTML page for quickLift alignment differences. Braney
• Fixed an issue with the hubApi that caused a bad cache in trackDb. Brian
• Made a special case for the bigBed loader to only load a bed6 if asked to load a bed7 for backwards compatibility. Braney
• Updates to the faceted UI for composite tracks to support tracks that are on by default and to be more consistent when some subtracks have subsequently been hidden manually (e.g., via right click). Jonathan
• The faceted UI for composites now imposes a limit on the number of subtracks that can be active at once for sanity reasons (the new limit of 1000 is still very high). Jonathan
• When the faceted UI includes data type selectors, it is now possible to provide alternate labels for those checkboxes that don't match the internal names used for subtrack selection. Jonathan
• HubSpace changes: fixed a weird DataTables bug where we double-displayed rows when doing a hubUpload; documented on hgCustom that hubSpace exists; defaulted to the currently open hub name when selecting the upload button; and when viewing a hub's files. Chris
• Fix a Visible Tracks bug when calculating the tracks that are on. Chris
• Updated support for Multi-region exon mode knownGene processing. It now correctly handles the pseudoGenes checkbox in the track settings, all the trackSets including MANE, and also the comma-separated ID-list filtering. Galt
• Restoring static curl in build. Galt
• Adding XSS security fix to hub error printf. Max

v493 branch

New feature: "Visible Tracks" group at the top of the hgTracks group list with the list of currently visible tracks if there are more than 32 tracks displayed (#36609). Chris
Added a strand filter to the short match track (#36984). Max
Allow changes to custom track data, even when uploaded via URL (#35492). Max
Eliminated separate load/draw path for bigBeds with less than 9 fields. Use bigBed header for number of fields ONLY if the number of fields is not specified in trackDb (#36940). Brian
Improved readability of hub error message on hgTracks (#36916). Max
The disconnect button now appears in the group blue bar when a hub encounters an error (#36791). Max
Added hicArcLimit and hicArcLimitEnabled trackDb settings for Hi-C tracks to limit the number of displayed interactions in arc mode. These settings display only the N strongest interactions. This setting is enabled by default for all Hi-C tracks, with a default limit of the top 10000 interactions (#36774). Jonathan
Tweaked the auto-shading scheme for HI-C tracks when more than 1000 interactions are being displayed - ideally, this should result in fewer interactions displaying with a fully-saturated color (#36774). Jonathan
Hubspace: users can overwrite uploaded files and delete hub.txt files without deleting all other files in the hub. warn users about certain actions. Don't add auto-generated trackDb stanzas if the user is uploading a hub.txt and the bigDataUrl for a track in the hub.txt is part of the current upload (hacky reliance on scanning the file, not true trackDb parsing (#36830). Chris
Enabled filters in hgTrackUi for bigBed3 and bigBed4 (#28501). Brian
Creation of a new page to automate chain file requests/delivery (#31811). Hiram
Fixed hubUrl retry button url encoding. Chris
Fixed hubUrl encoding in hubSpace UI (#36936). Chris
Made grey bar icons and hgc popups on by default (#36953). Chris
Made trackHubsCanAddGroups the default setting (#36955). Brian
Encode IDs used by jsOnEventById to prevent cases where IDs contain parentheses, quotes, etc. Now IDs can be used more safely in selectors too (#36841).
Whitelist GenArk in localUrl paths in isValidBigDataUrl() in customFactory.c (#34544). Galt
Added support for vai.pl to accept a GenArk assembly accession as db, and for hgVai to accept a geneTrack without the hub_number_ prefix when db is a hub (#36534). Angie
Moved parsePosition and parseRange routines into common.c. Added callback routines for -bed and -positions shared library bigBedCmdSupport and refactored the utilities to use them. -bed and -positions input are sorted automatically by chrom, start (#28109). Galt
Modified pslMap to remotely access chain psl input file if given a URL. Mark
Added number of aligned bases, alignment span, and strand to pslStats. Mark
CGIs should report when they fail to connect to a bottleneck server, rather than silently dying (#36992). Jonathan
Incorporated changes from Andrew Smith to tidy up the new facetedComposite UI for composite tracks (#36320). Jonathan
Fixed the problem with liftOver with genePreds when the result ends up on a different strand and the exonFrames need to be reversed. Brian
Hide table browser link on hgTrackUi for superTracks as per MarkD request. Max
Increased buffer size for VCF INFO field. Max

v492 Genome Browser Software

This is the v492 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Save HGVS searches onto the search history stack.
• HGVS searches use the refSeqSelect table if the transcript is present.
• Do HGVS searches in the auto-suggest position box on hgTracks.
• Fixed hgTrackUi to properly show filters for bigBeds with less than five fields.
• In bigBed click handling, changed error to warning when trackDb type field count differs from the bigBed header.
• Removed the info link for quickLift groups in hgTracks.
• Fixed gene alleles section of hgGene when doing hgGene in the popup on hgTracks.
• Fixed hgTracks tooltips to store the tooltip information in the 'data-tooltip' attribute instead of in the 'title' attribute to preserve the title strings. wait until DOMContentLoaded to actually parse the titles into 'data-tooltip' so the text size variable is presen.
• Fixed default hidden sections in hgGene to properly open when clicked, and fixed the population button in the Common Gene Haplotype Alleles section.
• Fixed PDF output problem in hgTracks. Previously, we had set hgFindMatches = NULL for PDF output to suppress the item search highlight in the generated PDF. But track loading code in bigBedTrack and squishPackyMode both depend on that search term, those routines that need the original search value.
• Added hgTables Summary and intersection buttons for knownCanonical.
• Fixed missing field descriptions for mm39 knownGene.
• Fixed Table Browser to allow bigGenePred to be used like genePred and provide CDS FASTA output.
• Fixed hgTables Sequence output with CDS casing bug that caused introns to be upper case and has existed since 2002.
• Fixed hgLiftOver min match filter with -multiple fixed. It has been allowing too-small regions in the output. Error was in the system since 2013.
• Beginning to import the TOGA V2 annotations. Added a 'hideToggle' class element to hgc description page to allow hide/show of text on the description page for a track item. Function is on the TOGA V2 details page.

v491 Genome Browser Software

This is the v491 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Single path for loading bigBeds, and auto-recognizing the bigBed type based on the header.
• Initial import and rework of a new faceted UI for composite tracks, intended for situations where there are too many tracks for the existing composite UI to be usable. The feature was graciously contributed by Andrew Smith at USC.
• Added a checkbox that turns on rearrangement mode automatically for all CTs generated by hgBlat.
• Fixed another jquery-upgrade-induced bug on the composite track UI page.
• Fixed a problem quickLifting between human assemblies.
• Fixed link to hs1 browser from hub management page listing track hub.
• hConnectCentralNoCache can now access remote central servers.
• The right-click menu in hgTracks was listing all visibility options for tracks with an onlyVisibility setting (which is supposed to limit options). It has been made aware of the setting.
• Fixed maf frames caching bug.
• Fixed search bug on assembly hubs with supertracks caused by spurious calls to hubCollectTracks.
• Made listExisting endpoint to hubApi/liftOver respect fromGenome and toGenome arguments, taking precedence over the filter argument. If both fromGenome and toGenome arguments are present, they are ANDed together.
• Explicitly send cookies when doing async cart requests.
• Added option to twoBitMask to allow a bigBed file for masking.
• Fixed mafAddIrows and mafSplit to recognize GenArk GCx... assemblies in MAF files where the problem was the assembly name has dots in the name, and MAF files delineate assembly name and sequence name with a dot.

v490 Genome Browser Software

This is the v490 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Added an info icon to the group blue bar of track hubs on hgTracks.
• Added support to the position search box for gnomad style range specifiers, e.g. 1-1234-A-C.
• Clear cart variables related to a hub when you disconnect the hub.
• Added a dash to the allowed chars list.
• Allow mafGene to take in an input file instead of just a database table.
• Users will no longer be nagged about filter* settings being reported as invalid in hubCheck.
• Added cart rewrite 9 for moving ddg2p track out of decipher and into its own track, and move encodeCcreCombined into a super track.
• Fixed a filtering UI bug where we printed br tags that mess up the JavaScript that builds the dropdownchecklist.
• Evaluation of Longdust, FasTAN, and RED as compared to the current masking procedure.
• When API keys are sent, open an hgcentral connection and close it right afterwards, to make sure that no sleeping CGIs clog mysql connections.
• Added bedToBigBed -fixScores to correct scores that are out-of-range or invalid integer values.
• hubCheck can now scan for and report with text valid values for VIEWLIMIT VIEWLIMITMAX and DEFAULTVIEWLIMIT.
• Added hg.conf var to dump all CGI variables (POST and GET) with short value strings to the apache log.
• Verify the returnto argument when doing logins, and add an hg.conf variable of approved returnto hosts.
• More RedHat RHEL-specific fixes to GBiC installer.
• Made hgMenubar not error out when invoked alone.

v489 Genome Browser Software

This is the v489 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Incoming URLs with an "apiKey" parameter on the URL are skipping the CAPTCHA.
• Visibility fixes to the gear and X icons in hgTracks.
• Added two new glyph types for decorators: + and X.
• Fixed error when VCF tree display is a subtrack of a composite.
• Removed 0-length characters from user search strings.
• Restore tab switching on hgHubConnect being saved to the URL.
• Added support for genePred in quickLift.
• "verbose" variable on the URL is not read from the cart anymore.
• Added -bed option to bigWigToWig to selectively output wiggle regions (contributed by Michael Hiller).
• Introduced a function cartOrCfgOption() that will look in the cart and then in hg.conf for a configuration variable, so we can turn on config variables using manual entry into the cart.

v488 Genome Browser Software

This is the v488 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Added a search bar to the track hub definition documentation page.
• Adding "c h" shortcut to hgTracks to quick-connect a track hub.
• Adding a "hide all" button in the group blue bar.
• Make it obvious to users that identifiers, filters, and intersections have been created and will affect the table browser output using color.
• Remove 0-length characters from input when searching on hgGateway,hgTracks, and hgSearch.
• Fixed bug where we didn’t find normalization options for hic files that don't annotate the first chromosome.
• Have gff3ToGenePred indicate which GFF3 errors can be optionally treated as warnings.
• Show proper links to download server for BAM files on the format description page.
• Correct display of gene name on RefSeq gene track items.
• Fixed problem with lrg track nextExon using the wrong handler.
• Fixed bigBeds limitWiggle optimization to work properly with multi-region.
• Upgrading our NodeJS on dev from 0.10 to version 22, after 12 years.
• Upgraded RepeatMasker process to version 4.2.1 and RepeatModeler process to version 2.0.7. New genome assemblies going forward will use these new versions.
• Ongoing work on quickLift.

v486 Genome Browser Software

This is the v486 release of the UCSC Genome Browser. A summary of the relevant code changes includes:

• Add an hg.conf controlled UI change: a gear icon that brings up the context menu when clicked, and an 'x' that hides the track when clicked, that show up when the mouse is hovered on a track.
• Added chromAlias to hgConvert.
• Fix barChart bug, move hgc.js init call a few lines down in the hgc on hgTracks code so normal document.ready will work when the hgc page has rendered.
• Tweaked the setColorWith code to fill in empty values in the wig with the invisible color. This fixes a problem when using summary levels with more values than pixels that resulted in strange coloring effects in regions adjacent to empty places in the wiggle.
• Ongoing work on liftOver, most notably making saved sessions work better.
• Ongoing work on genarkOrg used for the organism pull-downs in hgConvert and hgLiftOver.
• Calls to cartDump from AJAX need to set db or quickLift groups get dropped.