This repository was archived by the owner on Sep 6, 2023. It is now read-only.
Dataflow partitioning #108
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Removing tracked deleted records should be treated as separate from the export process (#79) * First draft * Adding notable change * update app.json * adding tooltip help Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * TryFunction should not make DB calls (#82) * first draft * Further changes * Correcting the telemetry IDs Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Separate changelog (#85) Added separate changelog to shorten readme * Added ADLS Run API page (#90) * Merge branch 'main' of https://github.com/microsoft/bc2adls * Adjusted version * Improvements to logging (#89) * Improvements to logging * LockTable in Try function --------- Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Access denied issue on spark notebook (#92) * Added step * minor * Update SharedMetadataTables.md Clarified instructions reg. naming of the managed identity and reason for adding the permissions --------- Co-authored-by: Soumya Dutta <soudutta@microsoft.com> Co-authored-by: Henri Schulte <77101781+HenriSchulte-MS@users.noreply.github.com> * Warn user before makign schema changes if data already exported. (#96) Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Internal Fields cannot be exported (#98) Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Only start export for Enabled tables (#97) Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Skip global trigger event subscriber on missing license or permissionset (#100) * Skip event subscribers when no license or permissions * Increase version --------- Co-authored-by: Ron Koppelaar <Ron.Koppelaar@cegeka-dsa.nl> * Update Execution.md Adding link to Microsoft documentation to consume ADLS Gen 2 resources * Allow telemetry to be logged at all outputs. (#102) Co-authored-by: Soumya Dutta <soudutta@microsoft.com> * Adding the file path to the telemetry * Add the testimonials received (#103) * Add the testimonials received * remove logos --------- Co-authored-by: Soumya Dutta <soudutta@microsoft.com> --------- Co-authored-by: Soumya Dutta <38040179+DuttaSoumya@users.noreply.github.com> Co-authored-by: Soumya Dutta <soudutta@microsoft.com> Co-authored-by: Bert Verbeek <71499421+Bertverbeek4PS@users.noreply.github.com> Co-authored-by: Ron Koppelaar <33791875+RonKoppelaar@users.noreply.github.com> Co-authored-by: Ron Koppelaar <Ron.Koppelaar@cegeka-dsa.nl>
…FlowPartitioning
|
This sounds promising, may I ask a question about this?
Is this a warning we should consider? And if so, what would be the best setting for the Broadcast options?
|
Contributor
Author
|
@Arthurvdv The custom expression in the "Remove Deleted" step does not involve any non-quality comparisons, so I have not paid any mind to this warning. |
|
@HenriSchulte-MS, thank you for sharing. I'll update our pipeline ahead of the merge of this PR. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, data is not deliberately partitioned in the dataflow. Partitioning based on a unique identifier (systemid + company) can reduce data shuffling between worker nodes and reduce execution time.