Dependent VNFs receive single set of cfg values from multiple instances

[wittling]

This is a race condition issue. It affects descriptors where a VNF ("VNF D") has:

• a dependency with two or more VNFs of same type (VNF 1 and VNF 2),
• VNF 1 and VNF 2 are of same type (different names), and each uses same set of parameters (confkey names), even if the configurations themselves have different names.

To re-create this issue:

1. Create 1st VNF (name=l3gw1, type=l3gw)
   Create a configuration (name=l3gw1cfg), and include 3 parameters. Initialize these parameters to numerics i.e. { "confKey":"parm1", "value":"1" }, { "confKey":"parm2", "value":"2" }, { "confKey":"parm3", "value":"3" }

2. Create 2nd VNF (name=l3gw2, type=l3gw), and include the same 3 parameters. Initialize these parameters to alphabetics i.e. { "confKey":"parm1", "value":"a" }, { "confKey":"parm2", "value":"b" }, { "confKey":"parm3", "value":"c" }

3. Pass all of these parameters to a SINGLE dependent VNF ("VNF D") by configuring dependency in NSD. Important: Make sure you have a single VNF that depends on both VNF 1 and VNF 2 above.

4. In the INSTANTIATE scripts of VNF 1 and VNF 2, dump the environment variables to a file. (at runtime, these will print out properly as initialized in the descriptor).

5. In the CONFIGURE scripts of the VNF ("VNF D") that depends on VNF 1 and VNF 2, make sure you dump the parameters out to a file that ends in ".$$" so that you don't overwrite or append the same file. At runtime, you will actually get two files - one file as VNF 1 calls in, and one for VNF 2 as it calls in.

6. After the CONFIGURE life cycle event of VNF D, proceed to examine the parameter values that are prefixed with the type prefix ("l3gw_").
   i.e. cat vnf1.env.3601 | sort
   i.e. cat vnf2.env.2743 | sort

You will notice that although the script ran TWICE - once for each VNF, you only see values of ONE of the VNFs in BOTH env files. Whether you see the numerics or the alphabetics depends on who wins the race condition. I believe it is the FIRST VNF that wins, based on my testing. Because the VNF with numerics ran its INSTANTIATE 3 seconds prior to the VNF using alphabetics and the scripts in my test were printing the numerics (this only my theory based on that observation).

Not having studied the source code on this, it appears to me as though the orchestrator has a single dictionary with ConfKeys, when it probably needs to maintain a dictionary per VNF instance.

[wittling]

I just did some further debugging on this. In this test, I went back to what I had done originally, which was to define a single VNFD, with a scale-in-out parm of 2, and define two connection points. This effectively launches two VNFs of same type, each with the same set of confKeys, initialized to default values (so each VNF gets this dictionary set of parms and parm values). The only differences between the two are the connection points (they are on different networks, and need to be).

I noticed that the ems package is what is managing the parms. I see in utils.py that the parms are received from orchestrator via message queue. I also see that the VNF gets events from all of its "peer" VNFs (from same NSD). So I see that the initial set of JSON values are pumped into the VNF. Then later, when CONFIGURE is run, I see the VNF get a new event, but the parameters are still the original JSON values.

So now I see the issue. If a VNF changes an environment variable, there is no way for the orchestrator to get updated. What would need to happen, I think, is this:
a. an INSTANTIATE script would update a config parameter (environment variable)
b. the INSTANTIATE script would use ems package to notify the orchestrator that the config parm has been changed.

Unfortunately the ems package is only programmed to "receive" events from the orchestrator / queue. It is not set up to pro-actively send anything to the orchestrator in the other direction. Of course, there could be a timing and synchronization issue here. If you send something to an asynchronous queue, there is no guarantee that the message will be delivered by the time the next life cycle event (i.e. CONFIGURE) comes along.

I do know that the life cycle won't advance until the script returns 0. So a INSTANTIATE script could place a message on the queue, and the orchestrator could make sure it processes all queues and sends these updates before the CONFIGURE script gets kicked off. It may work that way already? I don't know. But that is the only way I can think of to do this. Because there is no inherent Linux OS "event" per se when a shell variable gets updated. It must be the job of the script programmer to fire off updates that would get back to the orchestrator.

At least now I think I understand why this is happening. It needs to be addressed, I think, because this solution cannot orchestrate properly and dynamically without something like this IMO.