enh: consider development history criteria similar to what JOSS has

[lwasser]

JOSS is implementing a development history criteria

https://blog.joss.theoj.org/2026/01/preparing-joss-for-a-generative-ai-future#what-well-look-for

which I think is a nice way to avoid the - overnight package creation and submit for review.

We should consider implementing something similar and even referencing JOSS' value of long standing development practices. This is not to say that we won't review new packages or single maintainer packages but we do want to see some history that extends beyond the "package overnight" and potential "gamification" that could turn into that would burden our volunteers.

[erik-whiting]

I like how JOSS says "very recent projects (particularly those with commit histories of only weeks) may warrant additional scrutiny to ensure they represent genuine collaborative scholarship rather than rapid AI-assisted code generation" but I don't like that there aren't explicit numbers given for number of collaborators or time spent in development. I hope that if pyOpenSci implements a history criteria, the criteria is explicitly stated.

I think the spirit of this criteria is to maintain the "openness" of OSS development. That is, paths for community contribution and a public record of development, and a good development history criteria would allow for either of those things. For example, any repo that has at least 3 months worth of publicly viewable commits OR at least 3 different non-bot contributors automatically meets the development history criteria. Either of those things show that the project maintainers are open to collaboration and public opinion.

This is just an example of something I think honors the spirit of the criteria, there are surely other and better ways. My main point though is that if pyOpenSci does introduce a history criteria, that criteria be explicit.

[sneakers-the-rat]

Tough balance between predictable criteria for authors and retaining editor discretion, good point. Rules like this always should be framed in terms of the values they reflect (as you say!). The values here are both (at least) maintaining open and social contributorship and also a human-on-human review process - avoiding spending human reviewer time reviewing code that the authors did not spend.

So perhaps we have values framing, provide a guideline like "approximately this much time, contributors, etc." While still saying editors retain discretion. Want to a) give authors clear expectations, b) allow editors to act in the best interest of the venue and reviewers, c) avoid throwing editors under the bus if (a) and (b) are in conflict, where authors say "we meet the benchmarks!" But editors see some problem that violates the underlying values of the rule, even if the rule is technically satisfied. That goes both ways too - allow editors discretion to review a package that is under the thresholds, e.g. if a team of two works hard and in the open for a year, they probably shouldnt get turned away just because there werent 3 of them.

[lwasser]

I love this discussion. One of the things that I found difficult with JOSS' language (i mentioned this in Slack) was that it might be challenging to enforce (which I think begins to align with your comment @erik-whiting ), but I also REALLY love this idea of articulating our values related to the policy - which essentially is the underpinning of what open source is.

i'm going to pull out some bullets based on the JOSS blog post and the discussion here so far!

JOSS

• clear evidence of open source software practices (and i would add thoughtful development and sustained maintenance)
• <Extends existing ecosystems thoughtfully - genai might spit out pure matplotlib when the user could be building on top of holoviz or some other existing tool>We particularly value work that thoughtfully builds upon or extends existing software ecosystems rather than reinventing solutions where quality alternatives already exist. We recommend this information live in your user-facing documentation to help would-be users understand what your software does, and how it does it.
• <i love this paragraph - we want to embody the spirit of open source however> Meaningful research software typically emerges from sustained collaborative effort over time. We will examine the project’s development timeline and commit history as indicators of genuine community engagement and iterative refinement. Projects with development histories spanning multiple months or years, showing contributions from multiple developers, and demonstrating evolution through community feedback are more likely to represent substantial scholarly contributions. While we don’t impose strict temporal requirements, very recent projects (particularly those with commit histories of only weeks) may warrant additional scrutiny to ensure they represent genuine collaborative scholarship rather than rapid AI-assisted code generation.

I'd be okay with using JOSS' third bullet above verbatim and referencing it too.

---

So maybe something like (we can continue to co-develop this its just a start at an idea)

• pyOpenSci values open source principles, and we want to see those values manifested in the package we review. Our values include open development, clear pathways for new contributors, and a history of thoughtful development of the package prior to the package being submitted for review. We also value and believe it's critical that all code in a package we review has been reviewed and tested by humans before being submitted.

Below are some guidelines to help enforce these values in our volunteer peer review program. These values are not discrete and may be evaluated on a case-by-case basis by the editor leading the review. ...:

• As a baseline, we will look for 3 months of thoughtful development time
• We will look at a package's commit history for scoped commits and pull requests that show thoughtful and deliberate development that could be evaluated and reviewed by a human in its entirety.
• We will review contributor base and contributor documentation <i don't know about this one - there are some great 1-person maintained tools out there, so i don't think we should require a contributor base, but how do we write this language>
• We do not place value on stars as a metric for package maturity or adoption (there is a ton of star farming happening and we have even seen it in our Slack community - I've moderated it)

---

Just an idea that we can start with and then evolve to a final policy as more feedback and comments come in here!! keep the thoughts coming!

[hamogu]

We have a few accepted 1-person projects on our list that were written mostly before the age of AI. We thought they were so good that we accepted them so we shouldn't make a policy that would no longer allows those packages. But @erik-whiting comment above is more open than that. He says "x months of history or (not "and"!) y contributors". That means a one-person package is fine, as long as it was not written overnight.

However, I think the editor discretion is important. Without that, it's too easy to game by exactly hitting the quota - just launch Y AI agents and tell them to co-develop that package. That's exactly the advertised use case for the most recent Claude model (see, e.g. this blog, so "at least Y contributors" is, right now, only a restriction for people who can't afford their MAX or PRO package.
I get the idea of quantifiable criteria, but in this case we want criteria for humans, but still need the editor discretion for cases where people command their bots to just circumvent them in obvious ways (commit with different ids, backdate commits).