Technical Concept
Digital Twins or Cyber-Physical systems require a variety of different input methods. MQTT is just one of them. Hence, this framework is designed to enable an easy/quick switch between different communication methods. (E.g. Changing between MQTT, Serial, ROS# or plain UDP datagrams)
In an ideal situation, only the communication module needs to be exchanged, while the data pipeline itself remains identical within unity.
Centered around the aforementioned philosophy, the 3-Stage pipeline is used (DATA -> Subsciber -> Handler -> Writer(s)):
In essence, the data entry/exit point is three stages away from the actuator. Looking at the figure above, this means our data enters via the subscriber module, which is only designed to establish the connection to the MQTT broker, receive new messages, serialize it to a predefined message (FeedData() to a C# class) and forward this message (C# class) to a handler.
The handler is where the message is actually processed, e.g. if the message contains information for more than a single writer (Actuator), the distribution of this message will happen in the handler.
As a final step, the prepared information is applied to an object using one or more Writers.
More information regarding the individual steps are provided below
For example, we create a new sample Message including a publisher and subscriber for it:
Note: Anything publishing or subscribing to an MQTT Broker is referred to as a client. In the following, a subscriber will be used to explain the concept of the Digital Twinning Stacks.
The resultant Subscriber which will be created will look like this:
using UnityEngine;
using System.Collections;
using DTStacks.UnityComponents.Communication.MQTT;
using DTStacks.DataType.Generic.Custom;
namespace DTStacks.UnityComponents.Generic.Subscriber.Custom
{
public class SomeMessageSubscriber : DTS_MQTTSubscriber // Derived from the DTS_MQTTSubscriber
{
public SomeMessage Msg; // The new message
public void FeedData(string s) // Feeds the data into the message
{
Msg.FeedDataFromJSON(s);
}
public override void ProcessMessage(string msg) // This void will be called when data was received
{
FeedData(msg);
}
// Use this for extras during initialization
public override void ExtendedStart()
{
}
// Extended Update function, will be executed after regular update
public override void ExtendedUpdate()
{
}
}
}As a new user, we do not have to care about what is going on under the hood, but essentially, the new data package is pre-processed and forwarded to the
void ProcessMessage(string msg). From here, the data is simply forwarded to the FeedData(string msg) void, which feeds the data to the actual message C# class.
Next, within the message, the FeedDataFromJSON(string msg) is triggered within the actual Message class:
using System;
using UnityEngine;
using DTStacks.DataType.Templates; //Provides the message template
using DTStacks.DataType.Generic.Geometry;
// Automatically includes all DTStacks namespaces to prevent errors.
using DTStacks.DataType.Generic.Helpers;
using DTStacks.DataType.Generic.Math;
using DTStacks.DataType.Generic.Navigation;
// Automatically includes all ROS namesapces, since it was enabled in the message-creator window
using DTStacks.DataType.ROS.Messages.std_msgs;
using DTStacks.DataType.ROS.Messages.nav_msgs;
using DTStacks.DataType.ROS.Messages.geometry_msgs;
using DTStacks.DataType.ROS.Messages.sensor_msgs;
namespace DTStacks.DataType.Generic.Custom
{
[Serializable]
public class SomeMessage : Message // New custom message class is derived from the base message class
{
public string myString; // The created string
public float myFloat; // The created float
public SomeMessage() { } // Constructor of the message
}
}Where the Message class we derive our custom messages from looks like this:
using System;
using UnityEngine;
namespace DTStacks.DataType.Templates
{
[Serializable]
public class Message
{
/// <summary>
/// Overwrite the properties of the derived Message of type <c>DTStacks.DataType.Templates.Message</c>
/// </summary>
/// <param name="s">The JSON string</param>
public void FeedDataFromJSON(string s)
{
JsonUtility.FromJsonOverwrite(s, this);
}
/// <summary>
/// Generates a JSON-string based on the properties of the class derived from <c>DTStacks.DataType.Templates.Message</c>
/// </summary>
/// <returns name = "jsonString">The JSON-string containing all data in hierachical order</returns>
public string CreateJSONFromData()
{
string jsonString = JsonUtility.ToJson(this);
return jsonString;
}
}
}The base Message class only has two dedicated functions, Overriding itself with a JSON-String and creating a JSON-String from itself. This functionality is copied to our custom message since it is derived from the base class.
A JavaScript-Object Notation or short JSON is a standardized way of transferring information using a string. It is easy to read by humans while it remains easy to parse/"understand" by a computer. As an example, our created message would look like this when being sent/received:
{
"myString": "Hello DTStacks World",
"myFloat": 2021
}Since, JSONs are a powerful tool, for the most commonly used languages, a library is available to interpret the information received. Unity has an inbuild JSONUtility which handles this parsing and encoding of JSON strings. In essence, when asked to interpret a JSON object, this parser checks the class (in this case the custom message) for any coinciding objects and tries to overwrite their properties accordingly. This also works if the object itself consists of multiple objects. E.g. if our message looks like this:
public class SomeMessage : Message
{
public string[] myStrings;
public float myFloat;
public ClusterObject clusterObject;
public SomeMessage() { }
}With the clusterobject defined as:
public class ClusterObject
{
public string oneString;
public string anotherString;
public float oneFloat;
public float anotherFloat;
}Our resultant JSON string could look like this:
{
"myStrings": [
"Hello",
"DTStacks",
"World"
],
"myFloat": 20.209999084472656,
"clusterObject": {
"oneString": "Hello",
"anotherString": "Again",
"oneFloat": 20.209999084472656,
"anotherFloat": 1337
}
}Once the subscriber receives this JSON and overwrites the objects in the message, the values are available in Unity just as usual. E.g. the Inspector displays them as expected.
With the message class being overwritten with the new information, this message object is forwarded to a handler. As described at the beginning of this page, the handler's function is simply to distribute the information to where it is needed (or from where to get it, if it shall be published). Primarily, to reduce the efforts of implementing/changing the desired communication method.
In some cases (e.g. a simple bool) it might be unnecessary but for others (e.g. a manipulator joint positions) this is very handy. The DemoScene of DTStacks uses this principle. Where within the message the name and position (among other information) of all joints are transferred in a single JSON.
To keep it simple, here is the relevant part of this JSON:
"name": [
"shoulder_link",
"upper_arm_link",
"forearm_link",
"wrist_1_link",
"wrist_2_link",
"wrist_3_link"
],
"position": [
0,
-0.3385932445526123,
0,
0,
0,
0
],In this case, we can observe how the "upper_arm_link" object has a position value of "-0.3385..." while all other links remain at position = 0. The designed processor (in this case a JointStateProcessor) simply forwards to (or reads these values from) the subsequent Actuator module (in this case a JointStateActuator).
namespace DTStacks.UnityComponents.ROS.Helpers
{
public class JointStateProcessor : Processor
{
[Tooltip("The latest jointStateMsg")]
public JointStateMsg jointStateMsg;
[Tooltip("List of all joint state controllers found within the object tree below the robot parent.")]
public JointStateActuator[] JointStateActuators;
private void Start()
{
jointStateMsg.SetNumberOfJoints(JointStateActuators.Length);
for (int i = 0; i < JointStateActuators.Length; i++)
{
jointStateMsg.name[i] = JointStateActuators[i].name;
}
}
public JointStateMsg GetJointStateMsg()
{
for (int i = 0; i < JointStateActuators.Length; i++)
{
jointStateMsg.name[i] = JointStateActuators[i].name;
jointStateMsg.position[i] = JointStateActuators[i].GetJointState();
}
return jointStateMsg;
}
/// <summary>
/// Initiates all JointStateActuators to update their joint angle based on the JointStateMessage. Indicate if it is a ROS-Message.
/// </summary>
/// <param name="jsa">List of <c>JointStateActuators</c></param>
/// <param name="jsm">The <c>JointStateMessage</c> with updated paramters</param>
/// <param name="ros">Bool indicating if it is a ros-message and additional steps have to be taken.</param>
public void UpdateJointStates(JointStateActuator[] jsa, JointStateMsg jsm, bool ros)
{
for ( int i = 0; i < jsm.name.Length; i++)
{
var foundJoinstStateActuator = jsa.SingleOrDefault(item => item.name == jsm.name[i]);
foundJoinstStateActuator.UpdateJoint(i, jsm, ros);
}
}
}
}Since the handlers and reader/writer modules are highly dependent on their actual purpose, these have to be created individually. However, these are supposed to be simple forwarding and writing tools. Looking at this JointStateProcessor, it only forwards information or gathers them. For which the references Actuator modules (JointStateActuators) have a dedicated function as well:
namespace DTStacks.UnityComponents.ROS.Helpers
{
[RequireComponent(typeof(UnityEngine.HingeJoint))]
public class JointStateActuator : Actutor
{
[Tooltip("The name of the joint in the relevant data message.")]
public string name;
[Tooltip("The current angular state of the joint in radian.")]
public float angleInRad;
[Tooltip("The current angular state of the joint in degree.")]
public float angle;
[Tooltip("The current angular velocity of the joint.")]
public float velocity;
[Tooltip("The current effort.")]
public float effort;
[Tooltip("The angular offset to apply. Relevant if the used model is not in home-pose by default.")]
public float angleOffset;
private float targetState;
[Tooltip("If the robot is publishing, the joint rotation will not be manipulated with every frame.")]
public bool isPublishing = false;
private bool isConvertingNecessary;
[Range(1f, 0.001f)]
public float interpolationSpeed = 0.1f;
[HideInInspector]
public HingeJoint joint;
/// <summary>
/// Get the <c>UnityEngine.HingeJoint</c> at the start of the program
/// </summary>
private void Start()
{
joint = this.GetComponent<HingeJoint>();
}
/// <summary>
/// Update the joint state at which the <c>JointStateActuator</c> is attached based on the index
/// </summary>
/// <param name="index">Hierachical position of the joint</param>
/// <param name="msg"><c>JointStateMsg</c> containing the information</param>
/// <param name="isConvertingNecessary"> Indicate if a conversion from left to right-handed coordinate system is necessary</param>
public void UpdateJoint(int index, JointStateMsg msg, bool isConvertingNecessary)
{
name = msg.name[index];
this.isConvertingNecessary = isConvertingNecessary;
//angle = UnWrapAngle(msg.position[index]);
angleInRad = msg.position[index];
angle = Mathf.Rad2Deg *msg.position[index];
velocity = msg.velocity[index];
effort = msg.effort[index];
targetState = (angle + angleOffset);
}
/// <summary>
/// Get the current JointState based on the joint.angle and the angular Offset
/// </summary>
/// <returns> Returns the current JointState in radians</returns>
public float GetJointState()
{
angle = -(joint.angle + angleOffset) * Mathf.Deg2Rad;
return angle;
//return WrapAngle(angle);
}
public void Update()
{
if (!isPublishing)
{
MoveJoint();
}
}
/// <summary>
/// Moves the joint by lerping from the current position to target position based on the interpolation speed
/// </summary>
private void MoveJoint()
{
Vector3 angleV;
angleV = joint.axis * (targetState);
if (isConvertingNecessary)
{
//angleV = angleV.ROS2Unity();
}
Quaternion rot = Quaternion.Lerp(this.transform.localRotation, Quaternion.Euler(-angleV), interpolationSpeed);
this.transform.localRotation = rot;
}
/// <summary>
/// Get the current JointState based on the joint.angle and the angular Offset
/// </summary>
/// <returns> Returns the current JointState in degrees</returns>
public float GetJointStateInDegree()
{
angle = -(joint.angle + angleOffset);
return angle;
}
}
}This is the last (or first) step for the 3-Stage-Pipeline. While any conversion between values (e.g. radians to degree) can be conducted in previous steps. In practice, it was considered handier to combine these steps with the Reader/Writer module directly.