Think about dynamic link instead of static dict in queue_entry and Qt4_queue_item

[IvarsKarpics]

Implement more dynamic way to represent needed queue_entry methods:

1. add list of methods to queue_entry.xml. Dynamicaly populated for QueueModel

[pkeller]

IIRC this refers to classes defined in queue_entry.py and their mapping to queue model objects. It would be good to also have a way of querying which queue model objects are available that returns true when a subtype of the queried type has been loaded. This would allow for generic/abstract types for some of these, with specific variants implemented by subtyping.

[IvarsKarpics]

As I explainded before my suggestion would be:

1. Split queue_entry in several files: queue_entry stays as a dynamic container and imports generic_queue_entry with basic collection methods and site_specific_queue_entry which containes each site specific methods.
2. Expand static dict with all available methods and add methods that checks if available queue_entry is available.
   This is a major change and needs an agreement between all developers. I can prepare a working solution of this idea and post it here with some examples.

[IvarsKarpics]

Hi,

Coming back to this topic. A solution could be to use at least 3 files:

• queue_entry as a mapping point for all entry classes
• base_queue_entry defines basic queue entries (DataCollection, Characterisation, etc)
• site_specific_queue_entry

I tried something like this:

queue_entry:

import logging
from collections import namedtuple

from base_queue_entry import *
import EMBL_queue_entry
import queue_model_objects_v1 as queue_model_objects

status_list = ['SUCCESS','WARNING', 'FAILED']
QueueEntryStatusType = namedtuple('QueueEntryStatusType', status_list)
QUEUE_ENTRY_STATUS = QueueEntryStatusType(0,1,2,)

MODEL_QUEUE_ENTRY_MAPPINGS = \
    {queue_model_objects.DataCollection: DataCollectionQueueEntry,
     queue_model_objects.Characterisation: CharacterisationGroupQueueEntry,
     queue_model_objects.EnergyScan: EnergyScanQueueEntry,
     queue_model_objects.XRFSpectrum: XRFSpectrumQueueEntry,
     queue_model_objects.SampleCentring: SampleCentringQueueEntry,
     queue_model_objects.Sample: SampleQueueEntry,
     queue_model_objects.Basket: BasketQueueEntry,
     queue_model_objects.TaskGroup: TaskGroupQueueEntry,
     queue_model_objects.Workflow: GenericWorkflowQueueEntry,
     queue_model_objects.XrayCentering: EMBL_queue_entry.EMBLXrayCenteringQueueEntry}

This MODEL_QUEUE_ENTRY_MAPPINGS could be something more advanced istead of a static dictionary.

base_queue_entry:

• All classes previously located in queue_entry moved to base_queue_entry

EMBL_queue_entry:

• Site specific classes
• Inherited from BaseQueueEntry

from base_queue_entry import BaseQueueEntry

class EMBLXrayCenteringQueueEntry(BaseQueueEntry):
    """
    Defines the behaviour of an Advanced scan
    """
    def __init__(self, view=None, data_model=None,
                 view_set_queue_entry=True):

        BaseQueueEntry.__init__(self, view, data_model, view_set_queue_entry)
        self.mesh_qe = None
        self.helical_qe = None
        self.in_queue = False

    def execute(self):
        BaseQueueEntry.execute(self)
    ...

After some adjustments this might work.

[IvarsKarpics]

Hi all,

I played a little bit with a dynamic mapping of data model and queue entry. The idea would be similar as described previously and a code example could look like:

queue_entry.py

import logging

import queue_model_objects_v1
import generic_queue_entry
import EMBL_queue_entry

MODEL_QUEUE_ENTRY_MAPPINGS = {}

logging.getLogger("HWR").debug("Initializing queue entries:")
for attr in dir(generic_queue_entry):
    if "QueueEntry" in attr:
        queue_entry_name = attr
        queue_model_name = queue_entry_name.replace("QueueEntry", "")
        if hasattr(queue_model_objects_v1, queue_model_name):
            logging.getLogger("HWR").debug("- %s : %s" % (queue_model_name, queue_entry_name))
            MODEL_QUEUE_ENTRY_MAPPINGS[getattr(queue_model_objects_v1, queue_model_name)] = \
                  getattr(generic_queue_entry, queue_entry_name)
queue_model_name)


logging.getLogger("HWR").debug("Initializing site specific queue entries:")
for attr in dir(EMBL_queue_entry):
    if "QueueEntry" in attr:
        queue_entry_name = attr
        queue_model_name = queue_entry_name.replace("QueueEntry", "")
        if hasattr(queue_model_objects_v1, queue_model_name):
            logging.getLogger("HWR").debug("- %s : %s" % (queue_model_name, queue_entry_name))
            MODEL_QUEUE_ENTRY_MAPPINGS[getattr(queue_model_objects_v1, queue_model_name)] = \
                  getattr(EMBL_queue_entry, queue_entry_name)

generic_queue_entry.py in principle is like the current version of queue_entry.py and
EMBL_queue_entry looks like:

from generic_queue_entry import BaseQueueEntry

class EMBLXrayCenteringQueueEntry(BaseQueueEntry):
    """
    Defines the behaviour of an Advanced scan
    """
    def __init__(self, view=None, data_model=None,
                 view_set_queue_entry=True):

        BaseQueueEntry.__init__(self, view, data_model, view_set_queue_entry)
        ...

When MXCuBE is started the mapping is based on looking through generic_queue_entry and site_specific_queue_entry and searching for attributes that contain "QueueEntry". Of course this could be something more pretty, but this works:

2016-10-27 16:17:16,849 |HWR    |DEBUG  | Initializing queue entries:
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - Basket : BasketQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - Characterisation : CharacterisationQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - DataCollection : DataCollectionQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - EnergyScan : EnergyScanQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - SampleCentring : SampleCentringQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - Sample : SampleQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - TaskGroup : TaskGroupQueueEntry
2016-10-27 16:17:16,849 |HWR    |DEBUG  | - XRFSpectrum : XRFSpectrumQueueEntry
2016-10-27 16:17:16,850 |HWR    |DEBUG  | - XrayCentering : XrayCenteringQueueEntry
2016-10-27 16:17:16,850 |HWR    |DEBUG  | Initializing site specific queue entries:
2016-10-27 16:17:16,850 |HWR    |DEBUG  | - EMBLXrayCentering : EMBLXrayCenteringQueueEntry

Maybe this is to difficult to discuss via issues, but I think it would be useful to implement a more generic mapping mechanism. In principle each site could work on their specific, custom queue entries and others would not care (from the code merge point if view), because they are in separate modules. Further we could make queue_models_objects_v1 also more generic.

[marcus-oscarsson]

Hey !

Thanks for the initiative Ivars :)

This is a interesting discussion and I remembered that you mentioned this last time we met in Lund. I wanted to discuss this a bit more but we didn't have enough time. The idea from the beginning was to provide a plug-in like behavior for the the queue entries. It was not considered to be a priority at the moment and we didn't have enough time to complete the implementation.

The idea was simply that one could have the possibility load queue entries with a corresponding interface from a certain folder (or folders, maybe even configurable in the UI). A new queue entry would often have a UI to go with it, otherwise it would just be a variant of a already existing one. The idea was that it was supposed to be fairly straightforward to create a widget in the qt designer and write the entry in a sequence like way. A data model would be created to represent the actual data, this model would be automatically bound to the widget through a explicit naming scheme or a configuration file. This was roughly the original idea which I believe still could be of use.

I'm really happy to see that you guys are thinking in a similar way.

We are now working intensively on the web version so some of these things have to be replaced by web technologies, which maybe makes it even easier to create the "widget" and the configuration for the data model. However we are still in quite a early stage of the development and I think it will take a little while before we get to a point were this can be realized.

I think it would be good to take some time to improve the data model and the implementation of the queue before we realize this kind of plug-in feature. I'm personally thinking about it as a background activity and would be really happy to continue the discussion so that we a more well developed idea once we are ready to start to deal with it.

Thanks,
Marcus