-
Notifications
You must be signed in to change notification settings - Fork 0
CallingBot
Every time the bot receives a call, the from the CallingController, it will create a new instance of this script in its constructor where it will go through a few phases depending which event was received:
public simplecallbot(ICallingBotService callingBotService)
{
if (callingBotService == null)
throw new ArgumentNullException(nameof(callingBotService));
this.CallingBotService = callingBotService;
CallingBotService.OnIncomingCallReceived += OnIncomingCallReceived;
CallingBotService.OnPlayPromptCompleted += OnPlayPromptCompleted;
CallingBotService.OnRecordCompleted += OnRecordCompleted;
CallingBotService.OnHangupCompleted += OnHangupCompleted;
}This will only be run through Once in every call. So it can be conveniently used to first initialize some variables here. In here, use the incomingCallEvent to gather the details of the participants in the call and use it to clean the user data variables that we will use later.
participant = incomingCallEvent.IncomingCall.Participants;
var id = Guid.NewGuid().ToString();
//remove the persistent variables first
int retries = 0;
BotStateEdit.removeUserData(participant, "activeMode", ref retries);
BotStateEdit.removeUserData(participant, "activeAcc", ref retries);Then lets greet the user by calling GetPromptForText function from Replies class and return the Task with the actions to the next function, which is OnPlayPromptCompleted. GetPromptForText takes in a string and vocalize it using bot framework's default synthetic voice.
incomingCallEvent.ResultingWorkflow.Actions = new List<ActionBase>
{
new Answer { OperationId = id },
Replies.GetPromptForText("Top of the day to you! What would you like to do today?",2)
};
return Task.FromResult(true);In this function, we determine if the user wanted to record his/her voice or talk to a profile. So it sends a text message (Hero Card) to prompt the user, and thereafter prompt for name of profile if response was received from the Hero card.
//get click input here
Replies.GenResponseCard(participant);
activeMode = GetActiveProperty(activeMode, "activeMode", 50);
Debug.WriteLine($"ACTION: {activeMode}");
//Debug.WriteLine($"equal: {activeMode.Equals("None")}");
if (activeMode != null)
{
await SendRecordMessage($"Who would you like to {activeMode} as? 'as ::name::'");
activeAcc = GetActiveProperty(activeAcc, "activeAcc", 50);
Debug.WriteLine($"WHO: {activeAcc}");
}If record was selected, the function will initiate record where GetRecordForText vocalizes string input and creates a prompt object for recording the caller's voice.
playPromptOutcomeEvent.ResultingWorkflow.Actions = new List<ActionBase>
{
Replies.GetRecordForText("Recording. Please read out the sentence after you received the message and heared the beep.", silenceTimeout: 2)
};If call was selected, it will go into conversation mode:
playPromptOutcomeEvent.ResultingWorkflow.Actions = new List<ActionBase>
{
Replies.GetPromptForText($"Transferring call to {activeAcc}!",2),
getVoiceAsync("Hello...", 10).Result,
Replies.GetRecordForText(string.Empty, mode: -1)
};- OnRecordCompleted
The function receives recordOutComeEvent object which contains the recorded voice and pass it on to the appropriate processing function.
- RecordOnRecordCompleted - text message user sentences to record after every text
- CallOnRecordCompleted - retrieves audio clip from resourcespace / use built-in synthetic voice to reply the caller
if (activeMode == "record")
{
await RecordOnRecordCompleted(recordOutcomeEvent);
}
else
{
await CallOnRecordCompleted(recordOutcomeEvent);
}- OnHangupCompleted The bot will only touch this function when new Hangup() function is called, it lets the bot knows that the conversation should be terminated.
This class is the API for the ResourceSpace. ResourceSpace is Open Source Digital Asset Management (DAM) software. The most important function of this class are uploading recording clips to ResourceSpace and searching ResourceSpace.
The clips will firstly uploading to the ResourceSpace server by UploadSFTPFile.
client.Connect();
client.ChangeDirectory(@destinationPath);
foreach (var file in dir.GetFiles("*.wav"))
{
string sourcefile = sourceFileFolder + "\\" + file.ToString();
using (FileStream fs = new FileStream(sourcefile, FileMode.Open))
{
client.BufferSize = 40 * 1024;
client.UploadFile(fs, Path.GetFileName(sourcefile));
}
}
client.Disconnect();Then the file will be add in to the ResourceSpace database as in UploadResource
string resourceId = await CallAPI("create_resource", parameter.CreateResource(resourceType));
string uploadSuccess = await CallAPI("upload_file", parameter.UploadFile(resourceId, filePath));
if (uploadSuccess.Equals("true"))
{
await CallAPI("update_field", parameter.UpdateField(resourceId, "8", title));
await CallAPI("update_field", parameter.UpdateField(resourceId, "12"));
await CallAPI("add_resource_to_collection", parameter.AddResourceToCollection(resourceId, collectionId));
}This class stores functions that manipulate audio files. Functions vital for interacting with our ResourceSpace server.
Bot framework record records in memory stream in simplecallbot, but to upload, we need to convert it to .wav format. Which is simply writing it to a file stream as below.
public void ConvertWavStreamToWav(ref MemoryStream ms, string savetofilename)
{
FileStream file = new FileStream(savetofilename, FileMode.Create, FileAccess.Write);
ms.WriteTo(file);
file.Close();
ms.Close();
}And a function that uploads our .wav file to ResourceSpace. First create a CloudBlobClient so that we can interact with the blob container that is already in our Azure Storage service.
public string azureFunc(string localPath)
{
// Implement the account, set true for https for SSL.
StorageCredentials creds = new StorageCredentials(<AZURE_STORAGE_ID>, <AZURE_STORAGE_PASS>);
CloudStorageAccount strAcc = new CloudStorageAccount(creds, true);
CloudBlobClient blobClient = strAcc.CreateCloudBlobClient();Then direct it to open the desired blob container and delete it.
//Setup our container we are going to use and create it.
CloudBlobContainer container = blobClient.GetContainerReference("<BLOB_CONTAINER_NAME>");
container.CreateIfNotExistsAsync();In order to void a condition 412 error of accessing the same resource, use GUID as file name where it will always be unique and
// use GUID to generate a unique file name every time
fileName = string.Format(@"{0}.txt", Guid.NewGuid()); Then create the blob is it does not exist.
appBlob = container.GetAppendBlobReference(fileName);
// Now we are going to check if file exists and if it doesn't we create it.
if (!appBlob.Exists())
{
appBlob.CreateOrReplace();
}And upload it to Azure using .uploadFromFile and grab the absolute URI so that we can use it in RSAPI to move it to ResourceSpace.
// Save blob contents to a file.
try {
appBlob.UploadFromFile(localPath);
azureUrl = appBlob.Uri.AbsoluteUri;
}
catch {
Console.WriteLine("======Failed to save on Azure");
}BingSpeech will allow us to transcribe the caller's voice to words that we can interpret programmatically in ElizaDialog.
Firstly, set up variables with locale and subscription key, we are using "en-US" in this case to transcribe english audio.
public string DefaultLocale { get; } = "en-US";
public string SubscriptionKey { get; } = ConfigurationManager.AppSettings["BingKey"];Before we are able to use the BingSpeech subscription, we have to create the client with the necessary credentials to access it. And make it such that if there are response from the service, pass it to local function OnDataShortPhraseResponseReceivedHandler.
public void CreateDataRecoClient()
{
this.dataClient = SpeechRecognitionServiceFactory.CreateDataClient(
SpeechRecognitionMode.ShortPhrase,
this.DefaultLocale,
this.SubscriptionKey);
this.dataClient.OnResponseReceived += this.OnDataShortPhraseResponseReceivedHandler;
}The recorded audio was received in a stream object so in order to send it to BingSpeech service, iterate through the bytes and send it.
public void SendAudioHelper(Stream recordedStream)
{
int bytesRead = 0;
byte[] buffer = new byte[1024];
try
{
do
{
// Get more Audio data to send into byte buffer.
bytesRead = recordedStream.Read(buffer, 0, buffer.Length);
// Send of audio data to service.
this.dataClient.SendAudio(buffer, bytesRead);
}
while (bytesRead > 0);
}
catch (Exception ex)
{
Debug.WriteLine("Exception ------------ " + ex.Message);
}And when the transmission is done, tell it so by calling .EndAudio().
finally
{
// We are done sending audio. Final recognition results will arrive in OnResponseReceived event call.
this.dataClient.EndAudio();
}
}The response passed back by BingSpeech is a list of sentences it think its appropriate. Each element has some useful variables like length of string result, confidence score and most importantly the text itself.
private async void OnDataShortPhraseResponseReceivedHandler(object sender, SpeechResponseEventArgs e)
{
if (e.PhraseResponse.Results.Length == 0)
{
this.WriteLine("No phrase response is available.");
}
else
{
this.WriteLine("********* Final n-BEST Results *********");
for (int i = 0; i < e.PhraseResponse.Results.Length; i++)
{
this.WriteLine(
"[{0}] Confidence={1}, Text=\"{2}\"",
i,
e.PhraseResponse.Results[i].Confidence,
e.PhraseResponse.Results[i].DisplayText);We would always want the answer with the highest confident level therefore return index 0's transcribed text back to simplecallbot.
_bingresponse(e.PhraseResponse.Results[0].DisplayText);