[RFC] Create a new datachain pattern to describe an AI model or algorithm

[pichot]

Context

DTPR has now been used in dozens of cities to bring transparency to data collection in public space. It has proven itself effective in communicating to citizens the who, how, where, why of data collection, increasing citizens’ understanding and trust of the sensors deployed. For organizations that manage public spaces, the process of deploying DTPR has proven to break internal silos by bringing together disparate departments–public works, smart cities, IT, public engagement, etc.–getting all staff on the same page in understanding the purpose of a sensor deployment, its risks, its value, and the lifecycle of the data it will produce.

Problem

DTPR was designed to describe sensors collecting data in public space. As deployers of DTPR have grown in their ambitions to develop comprehensive digital transparency programs for their constituencies, public data collection is only one part.

Today, a comprehensive digital transparency program would be incomplete if it did not include transparency on the role AI or algorithms used in public decision making. DTPR is not currently suited to describe these.

Proposal

Grow and adapt DTPR’s framework, methodology, and taxonomy to describe an AI model or algorithms.

The current DTPR datachain, what I’ll call the Sensor datachain is designed to describe a sensor collecting data in public space, and the story of the data collected–how it is processed, who has access to it, where and for how long is it stored.

An AI datachain would describe an AI or algorithm: who is accountable for it and its purpose, the primary AI/algorithm technologies being used, what decisions will be made using it and what level of autonomy it has in those decisions, what risks it may pose, who has access to it, and where the AI is run and its results stored.

Datachain Categories

	Sensor datachain	AI datachain	Contextual color
hexagon	Accountable	Accountable
hexagon	Purpose	Purpose
hexagon		Decision Making 🆕	Level of autonomy in decision-making
hexagon	(Data-collection) Technology		Data collected is personally identifiable
circle	Data Type [deprecate] 🗑️
circle		Input Datasets 🆕
circle	Processing (Technology)	Processing (Technology) ♻️ with location context
circle	Output Datasets 🆕	Output Datasets 🆕
square	Access	Access
square	Storage with location context	Storage with location context
octagon		Risks & Mitigation 🆕
octagon	Rights 🆕	Rights 🆕

“Decision Making” taxonomy category 🆕

What decisions does this AI/algorithm inform?

Possible Elements

• Allocation of resources
  • For example, optimization of trash pickup routes, traffic plans
• Accept or deny
  • For example, civil service application, or a child’s acceptance to a school
• Ranking of priority

Contextual color

Level of Autonomy

• Orange: The AI/Algorithm processes, decides, and executes without human involvement
• Yellow: The AI/Algorithm processes and generates a decision, but a human required to execute
• Blue: The AI/Algorithm system processes and flags info for human to evaluate, decide, and action

“Input/Output Datasets” taxonomy category 🆕

What datasets are inputs into the algorithm/AI/data processing pipeline?
What datasets are produced by the algorithm/AI/data processing pipeline?

Possible Elements

This category would reuse the Data Type icons, but require a user to name and define a dataset based on its type.

Example Usage

Let's take the example of an algorithm or model that assigns children what elementary school they will attend in a city. Potential input datasets might include school capacity and current class sizes, the locations of students' homes, the locations of schools, school bus routes, public transportation routes, etc. Potential output datasets might be the list of children and their assigned schools, optimization analysis showing how the algorithm optimized for class size, or for transportation distance, etc.

Note, input datasets are not meant to define training data, but instead an input into an already trained model or algorithm. Training data and its potential risks and mitigations should be defined in the "Risks & Mitigation" category.

“Processing Technology” category 🆕

What type of AI/Algorithm is being used?

Possible Elements

This category expand on the existing elements from the "Processing" category, adding more specific types of AI and algorithmic systems such as:

• Regression Analysis
• Large Language Model
• Natural Language Processing
• Image recognition algorithms

Some questions I still have:

• What level of detail and precision would should we aim for to find the balance between technical accuracy that builds trust vs explanatory simplicity that helps with understanding?
  • For example, there are many different types of image recognition algorithms (Convolutional Neural Networks, Residual Networks, Scale-Invariant Feature Transform, etc.). Should we be that precise, or does “image recognition algorithm” suffice?

Location Context

Like the "Storage" category, which already has elements that define where data is stored, "Processing Technology" should have what I'm calling "Location Context" elements that allow a user to define where the processing is taking place. Examples include:

• Processed locally
• Processed internationally
• Processed on a 3rd party cloud

“Risks & Mitigation” taxonomy category 🆕

What identified risks does this AI/algorithm have?

A team at MIT conducted a meta-review of AI risks. In their own words,

The AI Risk Repository has three parts:

• The AI Risk Database captures 1000+ risks extracted from 56 existing frameworks and classifications of AI risks
• The Domain Taxonomy of AI Risks classifies these risks into 7 domains (e.g., “Misinformation”) and 23 subdomains (e.g., “False or misleading information”)

You can see the taxonomies here.

Leveraging this existing, open source work, we could generate elements for each risk in the MIT taxonomy.

Possible Elements

A user of these elements would outline the mitigation to the given risks the DTPR elements “additional description” field.

• False or misleading information
• Over-reliance and unsafe use
• Unequal performance across groups
• Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information
• Power centralization and unfair distribution of benefits
• AI system security vulnerabilities and attacks

“Rights” taxonomy category 🆕

What rights do I have in relation to this AI/Algorithm?

See the open proposal on adding a Rights category to DTPR here: #230

What’s missing and why

Define what training data was used to

I'd argue it should not be its own category. Here are my reasons:

• not all algorithms use training data
• it can be difficult to even know what training was used (ex. some LLMs)
• the general, non-technical public understanding of the risks and benefits of an AI/algorithm is, I would argue, not helped much by knowing what data a given model was trained on. A much more effective way to communicate this information would be to document what risks the training data may introduce, and explain the mitigation strategies used to avoid that risk.

Request for Comments

This RFC is meant to spark a discussion amongst users of DTPR and other interested parties around how to adapt DTPR to AI. This is just the beginning. There is a lot more to define and decide. Consider this a jumping off point.

The first questions I would appreciate thoughts on are:

• Do the AI datachain categories capture what the general public wants to know about an AI/algorithm? Are we missing anything?
• What other research and resources might we learn from?

[pichot]

Civic Tech Filed Guide has some good high-level typologies of AI technologies. Could help feed the Technology (AI/Algorithm) category: https://civictech.guide/were-getting-more-specific-about-ai/

[jeffyactive]

Excellent initiative @pichot, thanks for the detailed context of the thought process. The Open Data Institute recently created an AI Data Transparency Index and this report which I feel could complement the DTPR initiative and validate that it captures the general public's concerns.

[pichot]

@jyotisinghdhillon, here's a list of icons to draft to experiment with the AI datachain:

Decision Making

Hexagon Icons

Name	Description	Icon notes
Allocation of resources	A decision-making process where the AI/algorithm determines how to distribute limited resources (such as time, money, personnel, or materials) across different needs or requests. The system analyzes factors like availability, demand, and priority to optimize distribution. Examples include determining trash pickup routes, assigning staff schedules, or distributing public services across neighborhoods.	Three different sized blocks. Make a yellow example
Accept or deny	A binary decision-making process where the AI/algorithm evaluates information against specific criteria to produce a yes/no outcome. The system processes inputs through predetermined rules or learned patterns to determine if a request, application, or condition meets the required threshold for approval. Examples include civil service applications, school admissions, loan approvals, or benefit eligibility determinations.	Split circle with checkmark on one side, X on the other. Make a red example
Priority ranking	A decision-making process where the AI/algorithm evaluates multiple items or requests and arranges them in order of importance or urgency based on specific criteria. The system assigns relative values to each item and sorts them accordingly. Examples include emergency response triage, customer service ticket ordering, maintenance request scheduling, or public project funding priorities.	Stacked horizontal bars of decreasing length (like a ranked list). Make a green example
Matching	A decision-making process where the AI/algorithm pairs entities (people, resources, opportunities) based on compatibility or optimal fit. The system evaluates multiple variables to determine the most appropriate connections. Examples include housing assistance placement, job applicant matching, or pairing service providers with community needs.	- Two puzzle pieces connecting together
Personalization	A decision-making process where the AI/algorithm tailors content, recommendations, or services to specific individuals based on their preferences, behaviors, or characteristics. The system analyzes personal data to customize experiences. Examples include content recommendations, personalized learning paths, or targeted communications from public services.	- Person silhouette with customization settings gear

Contextual color

Level of Autonomy

• Red: The AI/Algorithm processes, decides, and executes without human involvement
• Yellow: The AI/Algorithm processes and generates a decision, but a human required to execute
• Green: The AI/Algorithm system processes and flags info for human to evaluate, decide, and take action

Processing Technology

Circle Icons

Name	Description	Icon notes
Large Language Model	A type of AI system trained on vast amounts of text data that can understand, generate, and manipulate human language. These models can perform tasks like writing text, answering questions, summarizing content, and translating languages. They work by predicting what words are likely to come next in a sequence.	- Speech bubble containing letter patterns (A, B, C...)
Optimization Algorithm	A mathematical process that finds the best solution from all possible solutions for a given problem, such as determining the most efficient route, schedule, or resource allocation. These algorithms work by systematically evaluating different possibilities to maximize or minimize specific outcomes like cost, time, or resources.	- Curved line with an arrow pointing to peak/optimal point
Text-to-Speech	Technology that converts written text into spoken voice output. These systems analyze text and generate synthetic speech that mimics human voice patterns, intonations, and pronunciations. Text-to-Speech is commonly used in accessibility tools, virtual assistants, and automated customer service systems.	- Text characters flowing into a sound wave
Time Series Forecasting	A method that analyzes historical data points collected over time to predict future values. This technology identifies patterns and trends in time-ordered data to make predictions about future events or behaviors. Common applications include weather forecasting, stock market prediction, demand planning, and resource management.	- Line graph with solid line becoming dotted (past to future)

Risks & Mitigation

Octagon Icons

Name	Description	Icon notes
Unequal performance across groups	The risk that the AI system performs differently (typically worse) for certain demographic groups based on characteristics such as race, gender, age, disability status, or socioeconomic background. This can lead to biased or unfair outcomes that disproportionately impact vulnerable communities. Mitigations may include diverse training data, routine fairness audits, and regular performance monitoring across different demographic groups.	- Two person silhouettes, one larger than the other
Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information	The risk that the AI system could access, expose, or deduce private information about individuals without their consent. This includes risks of data breaches, misuse of collected data, or the system's ability to infer sensitive characteristics from seemingly non-sensitive inputs. Mitigations may include data minimization practices, robust security measures, differential privacy techniques, and rigorous access controls.	- Shield with lock that has a crack
Over-reliance and automation bias	The risk that people place excessive trust in AI systems, leading to insufficient human oversight or inability to question algorithmic decisions. This can result in uncritical acceptance of AI outputs even when they are incorrect or harmful. Mitigations may include clear communication about system limitations, training for users on appropriate reliance, and maintaining meaningful human involvement in critical decisions.	- Person silhouette blindly following robot

Rights

Octagon Icons

Name	Description	Icon notes
Right to Access	The right to request and receive information about what personal data an AI system has collected about you, how this data is being used, and what decisions have been made using this information. This includes the right to obtain a copy of your data in a readable format.	- Key unlocking a data document
Right to Purpose Limitation	The right to ensure your data is only used for the specific purposes that were clearly stated when it was collected. This prevents organizations from using your data for new, unrelated purposes without your knowledge or consent. The data collector must specify and document the intended purposes before collection begins, and adhere to these limitations.	- Document with clearly defined boundary lines
Right to Contest	The right to challenge decisions made by an AI system that affect you, particularly when these decisions have legal or similarly significant effects. This includes the right to request human review of automated decisions, provide additional information, express your point of view, and have the decision reconsidered based on your input.	- Raised hand or flag symbol next to document
Right to Algorithmic Transparency	The right to understand how an AI system makes decisions that affect you, including meaningful information about the logic involved, the significance of the processing, and the likely consequences. This information should be provided in clear, plain language that allows you to understand how the system works and how it arrived at a particular decision.	- Magnifying glass over robot
Right to Non-discrimination	The right to be free from discriminatory treatment by AI systems based on protected characteristics such as race, gender, age, religion, disability, or sexual orientation. Organizations must implement and demonstrate appropriate technical and organizational measures to prevent discriminatory outcomes from their AI systems.	- Person silhouettes of equal size, maybe on a scale?
Right to Be Forgotten	The right to request the deletion of your personal data from an AI system under certain conditions, such as when the data is no longer necessary for its original purpose, when you withdraw consent, or when there is no legitimate interest in continuing to process it. This includes the right to have the system "unlearn" information derived from your data where technically feasible.	- Eraser over document

[pichot]

@jyotisinghdhillon with those icons an elements, here's a datachain we could use as an example:

AI System: Municipal Housing Assistance Matching Algorithm

Datachain Elements:

Accountable

• Organization: City Housing Department
• Description: The municipal housing department is responsible for this AI system and its outcomes

Purpose

• Planning & Decision-making
• Description: To efficiently match residents with appropriate affordable housing options and assistance programs

Decision Making

• Matching
• Level of Autonomy: Yellow - The AI processes and generates matches, but a human caseworker is required to review and execute
• Description: Matches housing applicants to available units and programs based on need, eligibility, preferences, and availability

Input Datasets

• Personal: Applicant financial and household information
• Tabular: Housing inventory database
• Spatial: Geographic information about housing locations
• Description: Combines applicant data, housing inventory, and location data to inform matching decisions

Processing Technology

• Optimization Algorithm
• Processing Location: Processed locally on secure government servers
• Description: Uses constrained optimization to maximize housing placements while respecting eligibility requirements and preferences

Output Datasets

• Tabular: Ranked potential matches for each applicant
• Description: Generates a prioritized list of housing options for each applicant with justification for the recommendations

Access

• Available to the accountable organization
• Available to me (the applicant, limited to their own data)
• Not available to vendor
• Description: Housing department staff can access all data; applicants can only access their own information and matches

Storage

• Stored primarily locally
• Retained 2 years
• Description: Data is stored within the city's secure data infrastructure and retained for 24 months after last activity

Risks & Mitigation

• Unequal performance across groups
  • Mitigation: Regular fairness audits and demographic performance analysis
• Over-reliance and automation bias
  • Mitigation: Required human review of all matches and clear explanation of system limitations

Rights

• Right to Access
  • Description: Applicants can request all personal data and factors that influenced matching decisions
• Right to Contest
  • Description: Applicants can challenge matching decisions and request human review with additional information
• Right to Non-discrimination
  • Description: System designed to prevent discrimination based on protected characteristics

[pichot]

@jyotisinghdhillon another proposed datachain inspired by the SOPA DCM system

AI System: Crowd Mood Analysis for Event Safety

Datachain Elements:

Accountable

• Organization: Event Security Department
• Description: The event security team is responsible for this crowd monitoring system

Purpose

• Safety & Security
• Description: Measuring the mood of crowds to keep major events safe by detecting potential security issues before they escalate

Decision Making

• Priority ranking
• Level of Autonomy: Yellow - The AI processes and identifies potential concerns, but security personnel are required to evaluate and take action
• Description: Analyzes crowd emotions and flags areas that may require security attention based on detected mood patterns

Input Datasets

• De-identified Image: Facial images processed on-device without storing identifiable data
• Description: Video feeds from security cameras positioned throughout event venue

Processing Technology

• Facial characterization
• Computer Vision
• Processing Location: Processed locally on-site
• Description: Analyzes facial expressions to detect emotional states without identifying specific individuals

Output Datasets

• Spatial: Heatmaps showing emotional patterns across venue areas
• Values/Time: Real-time mood trend analysis
• Description: Generates crowd mood visualizations for security team monitoring

Access

• Available to the accountable organization
• Not available to me (individual attendees)
• Not available for resale
• Description: Only authorized security personnel can access the mood analysis dashboard

Storage

• No personal data retained
• Aggregated data retained 3 months
• Description: No individual facial images stored; only anonymized, aggregated crowd sentiment data

Risks & Mitigation

• Compromise of privacy
  • Mitigation: On-device processing only, no individual identification, no storage of raw images
• Unequal performance across groups
  • Mitigation: System calibrated across diverse demographic groups with regular bias testing

Rights

• Right to Notice
  • Description: Clear signage informs event attendees about the crowd monitoring technology
• Right to Purpose Limitation
  • Description: System used exclusively for safety monitoring, not for marketing or other purposes
• Right to Non-discrimination
  • Description: Technical measures prevent demographic bias in emotion detection

[ChasFinley]

Perhaps explore colour scheme other th an red, amber and green - too much light street light - are there situations where it is ok that AI acts on its own without human intervention? If so, red implies this is always "bad" or "stop". Perhaps explore a different colour scheme.

[jiahao]

Sharing a few other resources that talk about classifications of AI and risks

• OECD Framework for the Classification of AI systems - OECD's definition of AI has been adopted in several laws, including the US. OECD also maintains a catalogue of tools and metrics for trustworthy AI.
• National AI Initiative Act
• CY 2024 Summary of Agency Compliance Reporting of Algorithmic Tools - has significant overlap (though not exclusive) with AI.
• AI Risk Ontology (AIRO) - designed to align to EU AI Act
• TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI - focused on the pyramid leading to x-risk
• San Jose GovAI Coalition's sample AI Policy and other resources
• IBM Risk Atlas
• AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policies