Two-tier storage support

[ww898]

I would be great to support two-tier storage.

[mikem8361]

/cc: @noahfalk

[mikem8361]

We need a lot more on the scenario you are trying to solve.

[ww898]

We have the two-tier storage for 100Gb compressed files. Changing the storage format is a big pain. Or two-tier format is obsolete?

[mikem8361]

The symstore repo was meant for indexing and downloading of symbol artifacts not necessary to implement a symbol store. You could do the decompression on the server/symbol store side.

[noahfalk]

@ww898 I think its still unclear what you are trying to accomplish. I'm not sure what "Two tier storage" means to you in this context. Can you give us an example or a better definition?

Also it sounds like the capability you want is to download symbols that are stored in "two tier storage", as opposed to say uploading to storage, or laying out files on a local filesystem such that they would create a "two tier storage"?

In general this repo set up to make a downloader for the SSQP service. SSQP is intentionally a subset of some of Microsoft's past symbol server efforts that was sufficient to let customers acquire Microsoft's symbols and not too hard to implement across all OS'es. Our short term goals are not trying to offer a complete solution for 3rd parties to host their own symbols, or to fully integrate with pre-existing 3rd party symbol stores based on the symstore tools in the Windows SDK. I suspect that is what you are hoping to do here? Its a reasonable request, its just not something we have the resources to do now. If you were hoping to achieve it soon you'd probably need to collaborate to implement the functionality or help raise the priority by getting lots of users to say they want the same thing. Hope that helps a bit to understand where things are.

[ww898]

@ww898 I think its still unclear what you are trying to accomplish. I'm not sure what "Two tier storage" means to you in this context. Can you give us an example or a better definition?

The symbol storage which has index2.txt in the root. See https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbol-store-folder-tree.

Also it sounds like the capability you want is to download symbols that are stored in "two tier storage", as opposed to say uploading to storage, or laying out files on a local filesystem such that they would create a "two tier storage"?

My target was developing the tool to create the symbol storage with modern format support. Before this moment we used symstore.exe from WinDbg installation, but symstore.exe doesn't support portable PDB and another file formats. It's a bit obsolete. I had been going to develop file parsers by myself and had unexpectedly found your project. Where there are everything which I can dream (thank you very much for this repo). Now my tool based on your library is completely finished. All my questions are based on features which was dropped during implementation. I need to know the is it temporary and you have plans to implement them or some ones are obsolete and won't be supported any more.

In general this repo set up to make a downloader for the SSQP service. SSQP is intentionally a subset of some of Microsoft's past symbol server efforts that was sufficient to let customers acquire Microsoft's symbols and not too hard to implement across all OS'es. Our short term goals are not trying to offer a complete solution for 3rd parties to host their own symbols, or to fully integrate with pre-existing 3rd party symbol stores based on the symstore tools in the Windows SDK. I suspect that is what you are hoping to do here? Its a reasonable request, its just not something we have the resources to do now. If you were hoping to achieve it soon you'd probably need to collaborate to implement the functionality or help raise the priority by getting lots of users to say they want the same thing. Hope that helps a bit to understand where things are.

I can share my sources with you on GitHub.