Skip to content

Conversation

@rerpha
Copy link
Contributor

@rerpha rerpha commented Dec 3, 2025

Closes ISISComputingGroup/DataStreaming#23
This PR documents what we are proposing to do for data streaming going forwards and removes the description of what the ESS were going to do several years ago.

If there's anything that we decided in the meeting that isn't in here I will add it.

@rerpha rerpha marked this pull request as ready for review December 17, 2025 14:48


## Live view, spectra plots etc.
These will be provided by a soft IOC (`azawakh`) which effectively consumes from event and histogram topics (and possibly run starts?) which will serve areaDetector and other PVs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will serve areaDetector and other PVs

Hmm - I was thinking it will serve spectra and spectra-maps, but AreaDetector PVs would be best done by having an AreaDetector 'source' plugin I think which is a bit of a different responsibility in some ways. To discuss...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will it be configured via PVs or from kafka as to what binning to use for the spectra etc

Copy link
Member

@Tom-Willemsen Tom-Willemsen Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure yet.

My initial instinct is that it makes more sense for those parameters to be in kafka, so that other consumers can use them if they wish (e.g. filewriter might be run in a mode where it histograms monitors? in which case it may want to use these same settings?). This way "serving spectra" and "areadetector" can be independent processes but use the same underlying ToF binning settings. But I am open to being persuaded either way on this...

Copy link
Member

@Tom-Willemsen Tom-Willemsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but wait for feedback from others in yesterday's meeting before merging.

@FreddieAkeroyd
Copy link
Member

On the diagram there is a link from the FPGA boards to kafka - i don't believe the FPGA can write kakfa directly, so is there a missing UDP2KAFKA component?

@rerpha
Copy link
Contributor Author

rerpha commented Dec 22, 2025

OK - let's get this in as a first draft, we can update when we come up with more sensible names for stuff.

@rerpha rerpha merged commit 1ef5821 into master Dec 22, 2025
2 checks passed
@rerpha rerpha deleted the ds_docs branch December 22, 2025 15:35
@GRyall
Copy link
Member

GRyall commented Dec 23, 2025

I'm late to the party, but LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update the dev wiki with the proposed system architecture

6 participants