Support copy to production and delete destination data in CI/CD environments

[j-fischer]

Hi @eltoroit ,

I've been playing around with your plugin quite a bit and I can see the potential for it to be useful as a deployment tool for some data driven configuration. In order to use it as part of a CI/CD process, however, it would be great to provide a way to flag the plugin invocation as a production deployment without getting another prompt in the command line. In addition, there are cases where a complete removal of the configuration records prior to the load might be desired, so it would be good if the following condition could be "overruled" through a command line flag of some sort.

I understand why those guards are in place, given that this plugin was designed to load training/test data in lower environments. However, would you be open to consider the alternative use case?

I could also see that an explicit deletion on an object level could be useful (clear some object records but not others - see upsert PR), and I'd be happy to look into the PR for that.

Please let me know if you have any questions.

Thanks for considering those changes.

Cheers

// ETCopyData.ts, line 391, 
if (data.settings.deleteDestination) {
	const msg = "Destination Org can not be production because this app deletes data! (2)";
	Util.writeLog(msg, LogLevel.FATAL);
	Util.throwError(msg);
	reject(msg);
} else { //...

[eltoroit]

With great power comes great responsibility. How to prevent accidental using this to wipe out the data in your production, when you actually meant to do it on a sandbox or a scratch org?

[j-fischer]

I understand that, but the use case for CI/CD is legit: the set up would not be accidental and the build server cannot respond to the prompt. My recommendation would be to use a command line flag like --no-prompt or --force like it is used with common destructive commands such as del or rm.

What are your thoughts?

[j-fischer]

I missed to mention that the delete functionality would be guarded by a prompt to ask the user for confirmation under normal conditions unless the --no-prompt is set.

I am all for preventing simple mistakes by adding gates and choosing the proper default values, which you have certainly done here. Just completely deactivating a feature would be a pity, given the potential here. Hope that makes sense?

[j-fischer]

Hi @eltoroit ,

Just wanted to see if you have any thoughts on what I proposed?

Thanks,
Johannes

[pbabalis1]

The potential to use this pluging to seed data is very useful. I installed and tried to export a simple object w/o success Attachments included,,,,,, for export command : sfdx ETCopyData:export -c ./data -s mrs --loglevel TRACE --json

Please let me know any glaring issue of my setup...I would like to have a live session if it is possible

for it to be useful as a deployment tool for some data driven configuration. II've been playing around with your plugin quite a bit and

[j-fischer]

Hi @pbabalis1 ,

Remove the "Account." prefix when referencing the fields. The two pass reference fields in the root should also be empty as those should only be set per object.

Pleases note that while this project is a great tool and I have used a few times in projects already, there is also a more stable and actively maintained project that does the same work: https://github.com/forcedotcom/SFDX-Data-Move-Utility

I would recommend to give this a try since you are at the early stages of playing with these plugins.

Hope this helps,
Johannes

[pbabalis1]

Johannes Thanks for the pointers and the introduction to the move utility which I will give a try and let you know if I run into any issues.... My use case is to export from a full sandbox a complete data subset to fit in a Dev Org

…

On Thu, Jul 27, 2023 at 8:49 PM Johannes Fischer ***@***.***> wrote: Hi @pbabalis1 <https://github.com/pbabalis1> , Remove the "Account." prefix when referencing the fields. The two pass reference fields in the root should also be empty as those should only be set per object. Pleases note that while this project is a great tool and I have used a few times in projects already, there is also a more stable and actively maintained project that does the same work: https://github.com/forcedotcom/SFDX-Data-Move-Utility I would recommend to give this a try since you are at the early stages of playing with these plugins. Hope this helps, Johannes — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6A7MBP3F4RAD2W4BNQVDXSMZC3ANCNFSM4RIMNXMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Phone: 831.207.4268

[pbabalis1]

Joahannes I installed the sfdmu utility and was able to export few files like Account. I tried to import the Account.csv file by issuing the command sfdx sfdmu:run --sourceusername Account.csv --targetusername ***@***.*** and received this error ...Any hints? [image: image.png]

…

On Thu, Jul 27, 2023 at 8:49 PM Johannes Fischer ***@***.***> wrote: Hi @pbabalis1 <https://github.com/pbabalis1> , Remove the "Account." prefix when referencing the fields. The two pass reference fields in the root should also be empty as those should only be set per object. Pleases note that while this project is a great tool and I have used a few times in projects already, there is also a more stable and actively maintained project that does the same work: https://github.com/forcedotcom/SFDX-Data-Move-Utility I would recommend to give this a try since you are at the early stages of playing with these plugins. Hope this helps, Johannes — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6A7MBP3F4RAD2W4BNQVDXSMZC3ANCNFSM4RIMNXMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Phone: 831.207.4268

[j-fischer]

I can't see the image, so won't be able to help. In the meanwhile, I came across this great talk by the LFDG. Hope you enjoy it.

https://www.youtube.com/watch?v=Z4ka2cRuUks

[pbabalis1]

Thx for the link ... Pretty informative and helped resolve the issue I was facing ... I added this "excludeIDsFromCSVFiles" :"True", in the export.json which resolved the issue and allowed me to complete the import!

…

On Sat, Aug 5, 2023 at 10:21 AM Johannes Fischer ***@***.***> wrote: I can't see the image, so won't be able to help. In the meanwhile, I came across this great talk by the LFDG. Hope you enjoy it. https://www.youtube.com/watch?v=Z4ka2cRuUks — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6A7LC7LJJOKL2G4GNWE3XTZ6HZANCNFSM4RIMNXMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Phone: 831.207.4268

[j-fischer]

@pbabalis1 Awesome, glad I could help! Cheers

[pbabalis1]

Is there a way to add comments in the export.json file .... I tried this // Some comments but sfdmu aborts ...Any alternatives ?

…

On Mon, Aug 7, 2023 at 10:18 AM Johannes Fischer ***@***.***> wrote: @pbabalis1 <https://github.com/pbabalis1> Awesome, glad I could help! Cheers — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEB6A7JK4BSW4N5HSWQLJUTXUEPNBANCNFSM4RIMNXMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Phone: 831.207.4268

[j-fischer]

JSON does not support comments. Please see this for help: https://www.freecodecamp.org/news/json-comment-example-how-to-comment-in-json-files/