huggingface interface

[cmatKhan]

This is going to be a new repo, but it will be very closely based on this one. In particular, we're going to need a modified AbstractAPI (maybe called HfAbstractAPI() ), which provides an interface for the huggingface hub API to manage downloading and querying data.

methods:

• parse_datacard()
  Each repo has a datacard that provides the structure, factor levels, etc. Right now, in particular, I'm interested in ways of reducing how much is in the datacard -- is there a better way of extracting the factor levels, especially those available at each level of partitioning, programmatically?

• downoad()
  this will wrap huggingface_hub.snapshot_download

• query()
  An interface to submit sql cmds to duckdb

Things to think about:

1. What if you have multiple datasets and what to combine them? Eg, a callingcards hops record and a promoter set?

[MackLiao]

After reviewing the base api abstract class, I found out it's worth decoupling two concerns: http request execution and the authentication. Specifically, we can create a separate wrapper interface for the http library (e.g. request()), so that customize the response more flexibly. We can also move the token injection into another module for clarity.

[cmatKhan]

http request execution and the authentication

currently the authentication is via token, and stored in the .env. See here

tfbpapi/tfbpapi/AbstractAPI.py

Line 45 in 7366239

self._token = token or os.getenv("TOKEN", "")

and here

tfbpapi/tfbpapi/AbstractAPI.py

Line 56 in 7366239

def header(self) -> dict[str, str]:

the token is authenticated by _is_valid_url().

tfbpapi/tfbpapi/AbstractAPI.py

Line 86 in 7366239

def token(self, value: str) -> None:

that functionality works well currently with the django API. I suppose it will need some adjustment to handle huggingface. However, huggingface is also authenticated via a token.

Right now I am inclined towards keeping the authentication in the .env (global variables, configured by user). Can you explain why we'd change this approach?

[cmatKhan]

More generally, my current thinking is not that we're going to get rid of AbstractAPI(), AbstractRecordsOnlyAPI() AbstractRecordsAndFilesAPI(). Those work well for the django API, which we will still use (although possibly just for user related data).

What I think we'll do is make another class, eg AbstractHfAPI(), which defines the required functionality of all concrete instances which inherit from it.

Another note here -- this is object oriented design. This has worked well so far as a framework and I'm inclined towards continuing with it.

Finally, I am interested in using pydantic and logfire. I'd like to incorporate these into the current AbstractAPI and its derivatives. But let's start by thinking about how to use them as we develop the huggingface interface

[MackLiao]

Yes -- the current implementation works. I am just wondering in respect to the design of this abstract class, it might be better if we split these two functionalities into separate files for loose coupling -- not necessary tho. We can definitely follow the same convention to implement AbstractHfAPI().

[cmatKhan]

it might be better if we split these two functionalities into separate files for loose coupling

At the moment, I am not anticipating that there will be much shared code between the current AbstractAPI() and the new AbstractHfAPI(), though they might share some vocabulary (eg, I could imagine using a function called _is_valid_url()).

I do not think that we'll gain anything by adding an additional layer of abstraction over AbstractAPI() and AbstractHfAPI(), but let's keep it in mind as we code.