32.0.0

What's Changed

• Feat/theodw 2813 cur-main-convert all managed to external tables by @roytalari in #2426
• Feat/theodw 2851 additionof group b discontinuance notice appeals by @BandlamudiNaveen in #2430
• Feat/pipeline modularisation final by @Fred83200 in #2359
• Feat/theodw 2813 cur main convert all managed to external tables by @roytalari in #2431
• Feat/theodw 2852 update mi pins include discontinuance notice appeals by @BandlamudiNaveen in #2432
• Updating notebook: nb_template_notebook by @stef-solirius in #2405
• Feat/dqf 870 create data model documentation for green cases and local plans by @RohitShuklaPINS in #2435
• Mpesc data changes by @Amerali786 in #2421
• Fix/theodw 2908 improve imports by @HarrisonBoyleThomas in #2433
• Feat/theodw 2863 nsip document entity refactor by @Fred83200 in #2428
• Upgrade spark to spark3.5 by @HarrisonBoyleThomas in #2438

Full Changelog: 31.0.0...32.0.0

31.0.0

What's Changed

• Updating pipeline: pln_copy_green_specialist_cases_mipins - In correct names by @prathapA100 in #2397
• Feat/theodw 2811 convert std managed tables to external tables in the standardised layer by @roytalari in #2396
• Feat/theodw 2881 std convert managed tables to external standardised custom notebooks by @roytalari in #2400
• Feat/theodw 2328 etl by @HarrisonBoyleThomas in #2351
• Feat/theodw 2520 coding standards nfr sta 06 by @stef-solirius in #2398
• Feat/theodw 2328 synapse integration by @HarrisonBoyleThomas in #2352
• Feat/dqf 824 the source table for live dim inspector should be changed to the weekly sap hr load table by @RohitShuklaPINS in #2399
• Fix missing caseCreatedDate mapping for Horizon S78 cases in standardised layer by @Imran-23 in #2404
• Feat/theodw 2109 upgrade spark pool to 3.5 by @EvangelosMagklaras in #2384
• Updating SBFunctionApp MD file Documentation by @Fred83200 in #2403
• Feat/theodw 2328 etl testing framework by @HarrisonBoyleThomas in #2353
• Feat/2870 listed buildings dlq 05 03 26 by @KranthiRayipudi in #2401
• Feat/theodw 2328 only run tests in build by @HarrisonBoyleThomas in #2409
• Feat/theodw 2812 convert(hrm-main) managed tables to external tables in the harmonised layer by @roytalari in #2406
• THEODW-2891-Part-1-Convert-Hrm-Managed-Tables-to-External-Tables-in-the-Harmonised-Layer by @anushaveerapalli68 in #2410
• Feat/2870 spark pool35 listed buildings dlq 05 03 26 by @KranthiRayipudi in #2412
• Mpesc additions am by @Amerali786 in #2408
• Updating notebook: py_sb_raw_to_std by @EvangelosMagklaras in #2415
• Fix undefined end_exec_time in Horizon harmonised notebooks by @Imran-23 in #2418
• Updating pipeline: pln_master by @roytalari in #2407
• Feat/dqf 897 migration pinsdatalab from uk north location to uk south for unmanaged data files by @RohitShuklaPINS in #2422
• Feat/dqf 898 review align mwr legacy tables odw standardised by @prathapA100 in #2423
• Theodw 2915 logging failure name error due to undefined variable error message in fail and log by @anushaveerapalli68 in #2420
• Updating notebook: py_sb_raw_to_std by @EvangelosMagklaras in #2425
• Feat/theodw 2795 prevent spark datetime failures in curated mipins 24 02 26 by @KranthiRayipudi in #2382
• Revert spark3.5 upgrade by @HarrisonBoyleThomas in #2419

Full Changelog: 30.0.0...31.0.0

30.0.0

What's Changed

• Feat/dqf 814 bug historic inspector specialisms data valid from date is not accurately updating by @RohitShuklaPINS in #2368
• Feat/dqf 847 alter webhook activity in sap hr and unmanaged master pipelines to use new teams channels by @RohitShuklaPINS in #2390
• Feat/Theodw-2846: Part 2 LDC & ELB attributes by @NHAL22 in #2394
• Feat/theodw 2785 3 update mi pins pipelines for group a attributes by @KranthiRayipudi in #2392
• Horizon migration am by @Amerali786 in #2383
• Updating pipeline: pln_copy_green_specialist_cases_mipins by @prathapA100 in #2389

Full Changelog: 29.0.02...30.0.0

29.0.2

• Revert the pln_publish_to_sb to use the auto-resolve runtime, in order to fix connection issues with the production appeals and back office service buses

29.0.1

What's Changed

• Revert "Updating notebook: appeal_s78_curated_mipins (#2376)" by @KalyaniNik in #2388

Full Changelog: 29.0.0...29.0.1

29.0.0

What's Changed

• Revert/theodw 2726 by @EvangelosMagklaras in #2364
• Updating notebook: py_create_spark_schema by @anushaveerapalli68 in #2363
• Theodw 2728 add ldc attributes for appeal s78 across odw layers by @anushaveerapalli68 in #2366
• Feat/theodw 2788 update group b harmonised and curated layers by @KalyaniNik in #2365
• Theodw 2784 update group a harmonised & curated layers by @anushaveerapalli68 in #2370
• Updating notebook: nsip_meeting by @KalyaniNik in #2360
• FEAT/THEODW-2723: Harmonised & Curated ELB changes (re-applied changes) by @NHAL22 in #2373
• Feat/theodw 2715 pins inspector final release by @stef-solirius in #2367
• Updating notebook: appeal_s78 by @KalyaniNik in #2375
• Fix incorrect HAS/S78 status history when multiple same-day transitions occur (HAS → S78 → HAS) by @Imran-23 in #2369
• Updating pipeline: pln_service_bus by @Fred83200 in #2377
• Updating notebook: appeal_s78_curated_mipins by @EvangelosMagklaras in #2376
• Updating notebook: appeal_s78 by @NHAL22 in #2381
• Fix Throttling Issues in Service Bus Publishing for Listed Building and PINS Inspector by @stef-solirius in #2378
• Feat/theodw 2784 revert update group a and group b changes by @KalyaniNik in #2386

Full Changelog: 28.0.0...29.0.0

28.0.0

What's Changed

• Feat/theodw 2500 logging and tracking service bus by @Imran-23 in #2333
• Updating notebook: py_logging_from_app_insights by @roytalari in #2335
• Update tests to use an az cli credential by @HarrisonBoyleThomas in #2341
• Fix Delta table schema shrink caused by narrow batches in Service Bus ingestion by @Imran-23 in #2339
• Repalce AutoResolvingIntegrationRuntime to PinsIntegrationRuntime dev… by @EvangelosMagklaras in #2313
• Data model version Update by @anushaveerapalli68 in #2338
• FEAT/THEODW-2769: Remove dart partitioning by @NHAL22 in #2343
• Feat/theodw 2499 nk logging notebook py sb std to hrm in pln service bus by @NagaPINS in #2340
• Theodw 2728 add ldc attributes for appeal s78 across odw layers by @anushaveerapalli68 in #2348
• feat(nsip): added nsip-document markdown by @KalyaniNik in #2332
• AIE, EntraID ODW Data Model by @roytalari in #2337
• Updating notebook: appeals_has_curated_mipins by @Imran-23 in #2355
• Updating notebook: vw_hr_measures by @prathapA100 in #2344
• Feat/dqf 864 green specialist case implement teams notification for post validation data quality issues as failure by @RohitShuklaPINS in #2356
• Feat/dqf 863 bug no cost centres are coming through to odw curated db dbo pbi hr measures by @prathapA100 in #2357
• Updating notebook: py_hzn_standardized_s78_tables by @anushaveerapalli68 in #2358
• FEAT/THEODW-2724: Harmonised ELB changes by @NHAL22 in #2346
• FEAT/THEODW-2725: Curated ELB changes by @NHAL22 in #2347
• Updating pipeline: pln_master by @roytalari in #2354
• FEAT/THEODW-2455: Update appeal-has.md file by @NHAL22 in #2336
• Updating linkedService: ls_prod_high_court by @HarrisonBoyleThomas in #2361
• Add classes to handle read/write operations by @HarrisonBoyleThomas in #2350
• Updating notebook: py_logging_from_app_insights by @roytalari in #2362

New Contributors

• @KalyaniNik made their first contribution in #2332

Full Changelog: 27.0.0...28.0.0

27.0.0

What's Changed

• Feat/theodw 2345 new nsip invoice orchestration existing tables metadata by @KranthiRayipudi in #2254
• fix(tooling): removing codeql check for feat push and pull request by @Medini-Aithal in #2264
• Feat/dqf 735 fix curated layer error handling pipeline messages by @RohitShuklaPINS in #2268
• Add Enforcement attributes for Appeal S78 across ODW layers by @Imran-23 in #2273
• FIX/THEODW-2622: Updating notebook: dart_api by @NHAL22 in #2266
• Updating pipeline: pln_horizon_to_odw by @KranthiRayipudi in #2262
• Feat/theodw 2495 rework logging and tracking in pln horizon to odw notebook py hzn standardised s78 tables by @KranthiRayipudi in #2272
• Theodw 2344 1c nsip project entity create nsip meeting table in odw curated db by @roytalari in #2277
• Fix Enforcement appeal grounds schema alignment in Appeal S78 Harmonised layer by @Imran-23 in #2280
• Bug fix/theodw 2638 bug fix notebooks py sb raw to std py sb std to hrm by @roytalari in #2278
• Dqf 755 fixing cosmetic issues in green specialist and local plans pipeline by @RohitShuklaPINS in #2283
• Updating notebook: py_sap_hr_protected_data_history_one_off by @prathapA100 in #2285
• Feat/theodw 2562 new horizoan pipeline failing at harmonised notebook but pipeline run shows as successful by @KranthiRayipudi in #2281
• Feat/theodw 2343 nsip meeting table in odw harmonised db by @roytalari in #2217
• Feat/theodw 2494 rework logging and tracking in pln horizon to odw notebooks by @anushaveerapalli68 in #2287
• Bugfix/theodw 2643 handle duplicate entra ids by @HarrisonBoyleThomas in #2289
• Updating notebook: pins_inspectors by @NagaPINS in #2282
• Updating notebook: py_sb_horizon_harmonised_appeal_has by @HarrisonBoyleThomas in #2251
• Add new users to PR reviewers group by @stef-solirius in #2292
• Feat/dqf 762 create post deployment pipeline for the green specialist case scd type 2 data migration by @RohitShuklaPINS in #2293
• FIX/THEODW-2127: appeal matrix notebook fixes by @NHAL22 in #2294
• Feat/theodw 2640 enhance nsip meeting notebook to support curated by @roytalari in #2286
• Fix/theodw 2632 appeal has s78 by @Fred83200 in #2295
• Fix/theodw 2659 nsip meeting curated failed by @roytalari in #2297

New Contributors

• @NagaPINS made their first contribution in #2282

Full Changelog: 25.0.0...26.0.0

What's Changed

• Feat/theodw 2619 create harmonised table for casework inspector by @stef-solirius in #2267
• Feat/theodw 2628: Enforcement appeals78 by @NHAL22 in #2301
• Updating notebook: appeal_s78_curated_mipins by @BandlamudiNaveen in #2303
• Feat/theodw 2619 create harmonised table for casework inspector3 by @stef-solirius in #2304
• Theodw 2665 pln master pipeline fails in py sb harmonised nsip meeting notebook new count is not defined by @roytalari in #2298
• Feat/theodw 2273 s78/Has data deletion script by @Imran-23 in #2302
• Feat/theodw 2420 inspector data create curated layer table and notebook by @stef-solirius in #2291
• Feat/dqf 778 incremental load testing for migrated data by @RohitShuklaPINS in #2308
• Add raamvar and KalyaniNik to CODEOWNERS by @stef-solirius in #2307
• Updating notebook: py_create_spark_schema by @NHAL22 in #2309
• Fix/theodw 2709 horizon pins iinspector failed in harmonised layer test and prod by @stef-solirius in #2312
• FEAT/THEODW-2492: pln_aie_document_data by @NHAL22 in #2305
• Updating pipeline: pln_curated by @stef-solirius in #2316
• Feat/theodw 2479 rework logging and tracking in pln service bus notebook py sb raw to std by @anushaveerapalli68 in #2306
• Updating pipeline: pln_pins_inspector_main by @stef-solirius in #2317
• Fix/theodw 2709 fix horizon harmonised pins inspector by @stef-solirius in #2318
• Updating pipeline: pln_aie_document_data by @NHAL22 in #2319
• Theodw 2479 implement logging and tracking in py sb raw to std fix by @anushaveerapalli68 in #2321
• Feat/theodw 2433 logging py horizon harmonised aie document by @NagaPINS in #2314
• Theodw 2733 s62a view cases dim notebook rework by @anushaveerapalli68 in #2322
• Updating pipeline: pln_rest_api by @NHAL22 in #2323
• Updating notebook: py_sb_raw_to_std by @anushaveerapalli68 in #2324
• merge main to feat/THEODW-2491-v1-rework-logging-and-tracking-in-pln-rest-api-notebook-py-raw-to-std by @roytalari in #2328
• Updating pipeline: pln_application_run_stage_service_bus by @NagaPINS in #2315
• S78 and HAS Load Test Cleanup by @Imran-23 in #2320
• Added pins-inspector documentation by @stef-solirius in #2326
• Feat/theodw 2491 v1 rework logging and tracking in pln rest api notebook py raw to std by @roytalari in #2325
• Theodw 2493 implement logging and tracking in pln listed buildings main sub pipeline and rework of notebooks by @anushaveerapalli68 in #2327
• Theodw 2490 implement logging and tracking in pln rest api sub pipeline 0 raw entra id and rework of entraid notebook by @anushaveerapalli68 in #2330
• Updating notebook: dart_api by @NHAL22 in #2331
• Updating notebook: py_sb_raw_to_std by @anushaveerapalli68 in #2334

Full Changelog: 26.0.0...27.0.0

What's Changed

• Feat/theodw 2345 new nsip invoice orchestration existing tables metadata by @KranthiRayipudi in #2254
• fix(tooling): removing codeql check for feat push and pull request by @Medini-Aithal in #2264
• Feat/dqf 735 fix curated layer error handling pipeline messages by @RohitShuklaPINS in #2268
• Add Enforcement attributes for Appeal S78 across ODW layers by @Imran-23 in #2273
• FIX/THEODW-2622: Updating notebook: dart_api by @NHAL22 in #2266
• Updating pipeline: pln_horizon_to_odw by @KranthiRayipudi in #2262
• Feat/theodw 2495 rework logging and tracking in...

25.0.0

What's Changed

• Data model Enforcement by @Imran-23 in #2255
• Feat/dqf 681 move pbi tables data from odw into mi pins stage by @prathapA100 in #2260
• Theodw 2556 update harmonised and curated notebooks for pads sap id attribute (appeal s78) by @roytalari in #2252
• Feat/theodw 2557 update harmonised and curated notebooks for pads sap id attribute appeal has by @roytalari in #2249
• Feat/dqf 729 creating a pipeline for the curated layer for green specialist cases and casework local plans by @EvangelosMagklaras in #2256
• Feat/dqf 646 unmanaged data transform the harmonised dimension scd type 2 to curated layer by @prathapA100 in #2246
• Feat/dqf 707 developing a pyspark notebook for post validation data quality checks by @RohitShuklaPINS in #2247
• Feat/dqf 732 developing a pyspark notebook for post validation data quality checks local plans by @RohitShuklaPINS in #2248
• Feat/dqf 688 create unmanaged data master pipeline in odw by @RohitShuklaPINS in #2263
• Feat/dqf 727 sickness reocrds missing due to incremental logic required adjsutment by @prathapA100 in #2265
• Feat/theodw 2481 implement logging and tracking in pln service bus notebook create table from schema by @stef-solirius in #2258
• Fix/theodw 2481 by @stef-solirius in #2271
• Feat/dqf 748 bug fix for sap hr weekly monthly files load failure by @RohitShuklaPINS in #2274
• Updating notebook: create_table_from_schema by @roytalari in #2276

Full Changelog: 24.0.0...25.0.0

24.0.0

What's Changed

• Integrate horizon_advert_attributes into S78 by @Imran-23 in #2223
• Feat/theodw 2440 full adverts hrm by @Imran-23 in #2229
• Adding notebook: py_load_green_specialist_case by @prathapA100 in #2231
• Feat/theodw 2562 p1 horizon pipeline failing at harmonised notebook but pipeline run shows as successful by @KranthiRayipudi in #2228
• Feat/theodw 2568 full adverts curated by @Imran-23 in #2235
• Feat/dqf 693 unmanaged data ingest local plans csv file into the standardised layer by @RohitShuklaPINS in #2232
• Adding notebook: py_load_casework_localplan by @prathapA100 in #2233
• Adding notebook: py_vw_green_specialist_case_HighHedges by @prathapA100 in #2238
• FEAT/THEODW-2444: CAS adverts harmonised layer by @NHAL22 in #2234
• Feat/dqf 728 create a view and vw mipins specialist hedgegrows and tpo also pbi tables for hedgegrows tpo by @prathapA100 in #2239
• Feat/dqf 642 unmanaged data create azure pipeline to copy excel files from pinsdatalab to odw raw and convert them into csv files by @EvangelosMagklaras in #2236
• Feat/dqf 692 unmanaged data create azure pipeline to copy local plans excel files from pinsdatalab to odw raw and convert into csv file by @EvangelosMagklaras in #2240
• Theodw 2555 validate ingestion of pads sap id in service bus standardised harmonised layers by @anushaveerapalli68 in #2242
• Feat/dqf 726 absence sickness data missing records by @prathapA100 in #2241
• Feat/dqf 724 transform the local plans data from harmonised to curated by @prathapA100 in #2243
• Feat/theodw 2441 update mi pins data flowto include new group b attributesfor full adverts by @BandlamudiNaveen in #2244
• THEODW-2572: CAS adverts standardised layer by @NHAL22 in #2230
• Anonymisation: classification-driven masking; simplified apply_from_purview; tests updated by @stef-solirius in #2212
• FEAT/THEODW-2583: CAS adverts curated layer by @NHAL22 in #2245
• Feat/2345 2346 2b nsip invoice entity create nsip invoice table in odw harmonised by @KranthiRayipudi in #2237
• FEAT/THEODW-2445: Curated appealhas mipins by @NHAL22 in #2250
• Revert "Feat/2345 2346 2b nsip invoice entity create nsip invoice tab… by @stef-solirius in #2253
• Fix/theodw 2596 timeout issue with dart curated notebook in test by @stef-solirius in #2257
• Revert "Feat/theodw 2562 p1 horizon pipeline failing at harmonised notebook but pipeline run shows as successful" by @HarrisonBoyleThomas in #2261

Full Changelog: 23.0.0...24.0.0