Dataset handling

In ManiVault, all data that is available to plugins is managed by the mv::data() manager. It helps us to make data sets from our plugin available for other to use.

When operating on data sets in a plugin, we need to notify other plugins about these changes.

General dataset operations

Creating data

Datasets can be added to the core by using the mv::data().createDataset<DatasetType>("DatasetType", "DatasetName") function.

For example, to create a Points type dataset:

auto points =  mv::data().createDataset<Points>("Points", "Dataset Name");

The signal events().notifyDatasetAdded(dataset) is emitted automatically and will trigger an EventType::DatasetAdded event.

We may want to create derived datasets or new data sets from a given subset like this:

mv::Dataset<Points> datasetDerived = mv::data().createDerivedDataset<Points>("Derived name", /*source dataset*/ dataset, /*data hierarchy parent*/, dataset);

mv::Dataset<Points> datasetDerived = mv::data().createSubsetFromSelection<Points>("Derived name", /*data hierarchy parent*/ dataset);

The notification of a newly added data set may be delayed, see the full function signature for more details.

Adding data to the dataset

We can then add data to the dataset by using the setData(Data, Number of Points, Number of Dimensions) function.

For our points data, we can generate some 1000 random points, each with two dimensions and set that example data in the dataset:

// Create 2D example data by randomly generating 1000 points
std::default_random_engine generator;
std::uniform_real_distribution<float> distribution;

int numPoints = 1000;
int numDimensions = 2;

std::vector<float> exampleData;
for (int i = 0; i < numPoints * numDimensions; i++)
    exampleData.push_back(distribution(generator));

// Set the data in the dataset
points->setData(exampleData.data(), numPoints, numDimensions);

events().notifyDatasetDataChanged(dataset);

The signal events().notifyDatasetDataDimensionsChanged(dataset) is automatically emitted if numDimensions is different from the current number of dimensions.

The setData function expects the data as a flat array in row-major order. The following matrix would therefore have to be fed in as [5, 2, 3, 1].

[5 2]
[3 1]

Changing data selections

When defining a new selection, we need to trigger the EventType::DatasetDataSelectionChanged event or Dataset<DatasetImpl>::dataSelectionChanged signal manually:

std::vector<std::uint32_t> selectionIndices = {0, 1};
dataset->setSelectionIndices(selectionIndices);
events().notifyDatasetDataSelectionChanged(dataset);

// equivalent but leaving selectionIndices empty
auto selection = dataset->getSelection<Points>();
selection->indices = std::move(selectionIndices);
events().notifyDatasetDataSelectionChanged(dataset);

Point data

The Point data set type is a container that can hold a vector of a certain data type like float or int8 and many more. It follows the visitor pattern and implements several generic functions for accessing the data that it holds.

Iterate over all data elements with visitFromBeginToEnd:

mv::Dataset<Points> dataset;
std::vector<float> samples = {0.5, 2, 6, 7.25, 1, 4, 12, 14.5};
std::uint32_t numPoints = 2;
std::uint32_t numDimensions = 4;
dataset->setData(std::move(samples), numPoints, numDimensions);

// Access all data set elements individually
dataset->visitFromBeginToEnd([](auto begin, auto end)
{
    std::cout <<"Values: ";
    for(auto it = begin; it != end; it++)
        std::cout << *it << " ";
    std::cout << std::endl;
});
// Values: 0.5, 2, 6, 7.25, 1, 4, 12, 14.5

std::vector<float> transformedDataA;
values.resize(dataset.getNumPoints() * dataset.getNumPoints());
dataset->visitFromBeginToEnd([&newValues](auto begin, auto end)
{
    std::transform(begin, end, samples.begin(), [](float num) {
        return std::sqrt(num);
    });
});
// New values: 0.5, 2, 6, 7.25, 1, 4, 12, 14.5

// return an object after visiting
auto transformedDataB = dataset->visitFromBeginToEnd<std::vector<float>>([dataset](auto begin, auto end) {
    std::vector<float> newValues;

    values.resize(dataset.getNumPoints() * dataset.getNumPoints());
    std::transform(begin, end, std::back_inserter(newValues), [](float num) {
        return std::sqrt(num);
    });

    return newValues;
});

// const overload: constVisitFromBeginToEnd

Iterate over all data points with visitData:

mv::Dataset<Points> dataset;
std::vector<float> samples = {0.5, 2, 6, 7.25, 1, 4, 12, 14.5};
dataset->setData(std::move(samples), 2, 4);

// Access data set elements individually or point-wise 
dataset->visitData([dataset](auto pointData) {

    // iterate over all points
    std::cout <<"All values: " << std::endl;;
    for (const auto& point : pointData) {
        for (const auto& dimVal : point)
            std::cout << dimVal << " ";
        std::cout << std::endl;
    }
    std::cout << std::endl;

    std::uint32_t dimIndex = 2;
    // iterate over all point indices
    std::cout <<"Values of dimension 2: " << std::endl;;
    for (std::uint32_t pointIndex = 0; pointIndex < dataset->getNumPoints(); pointIndex++) {
        std::cout << pointData[pointIndex][dimIndex] << " ";
    }
    std::cout << std::endl;
});
// All values:
// 0.5, 2, 6, 7.25,
// 1, 4, 12, 14.5,
// Values of dimension 2:
// 6, 12

Assign new data and convert it automatically to the internal Dataset type with convertData:

std::vector<int> samples = {0, 2, 6, 7, 1, 4, 12, 14};
// Updates the internal dataset data type to int
dataset->setData(std::move(samples), 2, 4);
mv::Dataset<Points> dataset;
std::vector<float> newValues = {1.1, 2.4, 6.9, 7.5, 1.2, 7.4, 12.2, 12.5};
// Converts float to int and changes number of points and dimensions
dataset->convertData(newValues, 4, 2);

Access source data of current data set with visitSourceData:

mv::Dataset<Points> sourceDataset;
std::vector<float> samplesSource = {0.5, 2, 6, 7.25};
newOutput->setData(std::move(samplesSource), 4, 1);

auto currentDataset = Dataset<Points>(mv::data().createDerivedDataset("Example Data", sourceDataset, sourceDataset));
std::vector<float> samplesDerived = {1, 4, 12, 14.5};
newOutput->setData(std::move(samplesDerived), 4, 1);

// access sourceDataset and not currentDataset
currentDataset->visitSourceData([](auto pointData) {
    std::cout <<"Values: ";
    for (const auto& point : pointData) {
        for (const auto& val : point)
            std::cout << val << ", ";
    }
    std::cout << std::endl;
});
// Values: 0.5, 2, 6, 7.25,

// access currentDataset
currentDataset->visitData([](auto pointData) {
    std::cout <<"Values: ";
    for (const auto& point : pointData) {
        for (const auto& val : point)
            std::cout << val << ", ";
    }
    std::cout << std::endl;
});
// Values: 1, 4, 12, 14.5,