Build and Gated Check-in презентация

Goals
Develop understanding of:
Build system architecture
Differences between AX6.x, and AX7
Current system state
Roadmap
Common operations scenarios

Knowledge
Process FAQ - http://fargobuild/processfaq/
AX FAQ - http://fargobuild/axfaq/
SharePoint sites:
ESS Ops
Build Infrastructure
Dev Infrastructure
ESS Documents
Happydev - http://happydev

Overview

Build Central Infrastructure

Lab Footprint

Source Depot - Overview
Server Info:
Hosted/Managed by BGIT
Connect To - vedbuitfsa06:4020
Actual Server - TK5SDSERVDAX01
What to do when you notice the depot is down

Source Depot - Overview Contd.
Proxies
Enhance local user experience
Both BGIT-managed and private
Current proxies : “sd proxies -o”
14 exist today
Administration
//depot/Admin/…
SD Admin Explorer - http://toolbox/22929

Source Depot - Overview Contd.
Help
Sd help
Sd help undoc
Protections
Controls who has permission to what
Modifying protections
Triggers
Logic to run on command execution
Main usage – preventing direct submits
Modifying triggers

Source Depot - Overview Contd.
Maintenance
BGIT-managed updates to server/SD instance
Remove unused clients
Client components
sd.exe
sdapi.dll
sdb
sdv
Packing tools: sdp, jjpack

Source Depot - Branching
Purpose:

Source Depot - Integrations
Introduction
…it is “to propagate changes happened to a namespace (source namespace) during a certain period of time into another namespace (target namespace)”. -SD Integrations Guide
Four States to Consider:
State of the source just before a given period of time
State of the source exactly at the end of a given period of time
State of the target at the moment of integration
State of the target at the end of the integration

Source Depot - Integrations Contd.
Revision
Versioning at the file level
Referenced with ‘#’
Example – sd sync //depot/rainier/rainfnd/foo.txt#2
Changelist
Collection of file revisions commited at a single point in time
Referenced with ‘@’, -c
,
Example – sd describe 123456
Label
Collection of changelists
Referenced with ‘@’
Example – sd sync …@7.0.123.0

Source Depot - Integrations Contd.
SD Integrate
Forward Integrate (FI) versus Reverse Integrate (RI)
Full branch integration
sd integrate -b
@
Point integration
Sd integrate -b
@
,
Manual mapping - renaming or baseless integrates
sd integrate //depot/b1/foo.txt#1,2 //depot/b2/foo.txt

Source Depot - Integrations Contd.
SD Resolve
Automatic resolve – sd resolve -as
Manual resolve – sd resolve
Accept Thiers (at) vs. Accept Yours (ay) vs. manual merge (e)
Conflict/Manual resolution:
>>>> ORIGINAL sources.all#2 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=4.0
==== THEIRS sources.all#3 ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
==== YOURS sources.all ++++++++++++++++++++++++++++++++++++++++++++++++
#endif
MSBUILD_VERSION=12.0
<<<< END ++++++++++++++++++++++++++++++++++++++++++++++++

Source Depot - Integrations Contd.
AutoIntegrate tool
Set of automation wrapping “sd.exe integrate”
Custom resolvers
MSBuild custom tasks/targets
//depot/../Build/AX_Tools/AutoIntegrate/...
Web Resolve
Run via command line “Integrate” command or BuildTracker jobs

Source Depot - Pack Files
Jjpack.exe
Primary packing format used in build/gates
Supports integrations
Does not support ‘undo’ – downgrades file op
SDP.exe
Does not support integrate options
Double-click diff
Used more commonly by Engineers to pass around changes

BuildTracker
Our Instance:
http://daxbuild/BuildTracker
BT product site:
http://buildtracker
BT Lab Solutions DG:
btracker

BuildTracker - Services
RR1SQLB02
Database
RR1SQLB03
BuildA – Access Management
BuildK – Manages BT encryption key
BuildL – Lab Monitoring (WMI monitoring)
BuildM – Mail and notifications
Web UI – http://daxbuild/buildtracker
API - net.tcp://RR1SQLB03:9700/BuildTrackerApi
DCSRDBTRACKER01
BuildS – Schedules job queues
Client machines
BuildQ – Worker service

BuildTracker - Lab Maintenance
Build Team Services Project
Windows update on clients
Reports
DB Backups
Auto-archive and drop cleanup
Gate Queue Processor check
BT Automation rules

BuildTracker - Lab Maintenance Contd.
Build Tracker Automation Library
Rules framework for enforcing consistent practices in our BT instance
Runs daily at 3am PST
Source in BuildServices at \build\source\MS.Dynamics.Build\...

BuildTracker - BT.exe
Wrapper for common lab maintenance and reporting functions using BuildTracker’s APIs
Available functions differ between BuildServices and product branches
Source - \build\ax_tools\buildtracker\bt\...

AX7 / Rainier

The Product
CTP Technical Concepts guide on HappyDev

The Product
CTP Technical Concepts guide on HappyDev

Lab Footprint (AX7)

Ax7 / Rainier Branches

CoreXT - Introduction
What is it?
A layer of rules, processes, features and automation
A common set of tools for developers, testers, builders and labs alike.
Generic and decoupled from the applications being built
Community effort within Microsoft
Existed since 2000 – estimate is used by 1/3 of the company
Latest release – v5.0
More information – http://corext

CoreXT - Our Environment
Overview
Current version – v3.2
Build.exe to each project node
Dependency Based Build (DBB) enabled
Tracer/Analyzer disabled
No adoption of CoreXT integration targets/properties
CoreXT V5 package manager backported

CoreXT - Environment Setup
Entry point
%inetroot%\tools\path1st\myenv.cmd
Shortcut by:
Open cmd.exe
Navigate to enlistment root
Run “setenv”
Branch-specific configurations
%inetroot%\build\branchcfg\myenv_
.cmd

CoreXT – Package Management
Branch package dependencies defined in a checked-in package manifest
On CoreXT startup, missing packages are retrieved and installed into a local cache directory.
Build processes consume package contents from the local cache
More information on HappyDev

CoreXT - Externals
Public Externals
ANY external reference or toolset
ANY internal reference or toolset submitted as a binary
Location – NuGet package repository (previously: %inetroot%\public\ext\...)
Toolsets
Often accompanied by makefiles/targets which integrate into CoreXT
Approvals
True external libraries (both Non-Dynamics Microsoft produced or 3rd party) need to receive LCA approval before check-in.
External Components Search

CoreXT - Build
Build.exe
Windows build tool
Orchestrator for building/traversing the build graph
Our version – 6.2.8427.0
‘dirs’ and ‘sources’
amd64 is default build target
Dependency Based Build (DBB)
Build node dependencies drive build graph traversal
Sources.dep

CoreXT - Build Contd.
PASS[0|1|2]
PASS0 – Generation of files, copying of static content
PASS1 – Compilation, X++, C#, C++ (both compile and link) all happen here
PASS2 – Packaging/Binplacing/Setup
Assembling the content from PASS0 and PASS1 into their final containers
PASS2_BINPLACE – binplacing of built bits (mostly from PASS1)
Project Inclusion
Integrating your project into CoreXT’s build
Dirs, sources, sources.dep, makefile, makefile.inc

CoreXT - Build Contd.
Rainier Build Topics
XPPC
Proxies/FormAdaptors
DBSync
Packaging

CoreXT - MSBuild
Current Version – v14.0
Build.exe to MSBuild Handoff
\public\ext\tools\msbuild.def inclusion in project makefile
\tools\path1st\msbuild.cmd wraps msbuild.exe
MSBUILD_PROJECT in sources

CoreXT - FxCop
Source:
All Branches: %inetroot%\public\ext\fxcop\...
Runs on:
Binplace via %inetroot%\tools\build\buildtypes\fxcop_build.cmd
MSBuild project when RunCodeAnalysis=true
Nothing currently enforcing correct ruleset here
Relevant Sources Macros:
Specify additional parameters - FXCOP_USER_FLAGS
Skip FxCop on Binplace - BINPLACE_FLAGS=--no-postbuild-fxcop

CoreXT - FxCop Contd.
Rule Categories
SWI/Stock – rules fulfilling QE requirements or shipped with FxCop
Example – Require CultureInfo be specified when calling a member that accepts it
Dynamics source - enforce Org-specific patterns
Example - Format of Copyright on binary’s properties
Dynamics Test – enforce patterns on test code
Example – Enforce inclusion of required test attributes on test cases

CoreXT - FxCop Contd.
Common Problems
Missing reference assembly
Multi-proc/race condition
Deadlock/timeout
Custom rule crashes
Logs
XML format per scanned assembly
From Build log root – FXCopLogs%BuildType%
FxCop error codes - http://msdn.microsoft.com/en-us/library/bb429400(v=vs.80).aspx

CoreXT - Customization
Stock CoreXT
Most source scripts under \tools\...
Makefiles under \public\ext\tools\...
As a rule don’t modify stock CoreXT
Customization points
\build\automation\...
\build\path1st\...
\build\buildtypes\...
\build\sources.all
\build\preenv.cmd
\build\myenv.cmd

CoreXT - Automation
CoreXT + BuildTracker
Stock automation scripts - %inetroot%\tools\build\automation\...
Customized automation scripts - %inetroot%\build\automation\...
bldTree.cmd
bldCommon.cmd
bldDone.cmd
bldStart.cmd
bldShare.cmd

CoreXT - Codereview
Wizard based on AutoProcess console that wraps CodeFlow submissions
Binaries\Configuration - \build\CodeReview\...
Source - \build\source\checkinwizard\...
Managed pack share - \\dcsrdchkinfsa\CodeReviews
Packs retained for ~2 years.
Command: “Codereview -c
”

CoreXT - DynCop
Home-grown static analysis tool
Developed and used initially for CoreXT v3.2 upgrade
Develop rules which enforce consistency and best practices across the build environment
Executes locally, in gates, and LMC
Source – VSTS DynCop Git repo
Documentation on HappyDev
Product teams adopting for enforcing practices in their teams

CoreXT – Platform-As-A-Binary
Historically, AX has been a monolithic codebase
Re-architect product into Platform component and Application component to be build/deployed/validated/released/serviced independently
Platform and Application are exchanged with one another as Last Known Good (LKG) zip packages
Further work expected to the product and ESS infrastructure supporting this
More on LKG Management here

CoreXT - Dirs Filtering Framework
Mechanism used to partition codebase for initial Platform/Application build separation
Uses build.exe’s filtering on dirs {tokens}
Tool source - \build\ax_tools\DirsFilterManager\...
Executable - \build\ax_tools\bin\DirsFilterManager.exe
DynCop rules enforcing correct tokens
HappyDev documentation

CoreXT - Dirs Filtering Framework Contd.
Examples:
Build retail –BuildOnly:fnd
Build retail –BuildConfig:Platform
Build retail –ExcludeTokens:app

CoreXT - Odds and ends
Aliases
Maps a command to an alias
Example – ‘codereview’ invokes the script to launch the code review wizard
Specified globally per branch in %inetroot%\build\aliases\aliases.pub
PrivateDev
Global private directory
Include //depot/private/privatedev/developer/
/… in your client spec
Map to %inetroot%\private\developer\
Private aliases
Add a file \build\aliases\aliases.pub under private directory

Gates

Gates - Technologies
WTT
Workflow engine
Machine management/maintenance
Powershell
Log parsing
Reporting
Queue processor
Managed code
Log parsing
Reporting
Scheduling
Orchestration
Monitoring
Wizards
Web Service
Cmd
(nearly) All of the above ☹

Gates - 50k Feet

Gates - Submitting

Gates - “Checkin” Command
Usage: checkin
Source script: \public\ext\tools\x86\checkin.cmd

Gates - Checkin Wizard
Wizard for submitting to the gates
Based on AutoProcess console framework
Configuration: \build\CheckinWizard\...
Source: \build\source\CheckinWizard\...
Command: checkin submit -c

Gates - Queue Processor
Core
Orchestrates scheduling of requested check-ins
Scheduling dimensions:
Machine availability
Blocking files
Branch lockdown
Main script is written in Powershell
Cmd wrapper manages logs and invokes core PS script
Logs retained for 24 hours
Executes every 5 minutes as a schedule task on gate server

Gates - Queue Processor Contd.
Core Contd.
BuildServices branch - %inetroot%\build\wtt_checkin\SetExternalReferences.cmd
Log location - %QueueProcessLogDirectory%
Execution server - %QueueProcessServer%
BuildServices - %inetroot%\build\wtt_checkin\QueueProcessor\...
BBCheck.exe
Core scheduling utility to the gate WTT pool
Wrapped by “checkin” command
Abort, Submit, Resubmit, Status, Diff, Apply, BypassQueueLock, Prioritize, Repack, debug
All branches - %inetroot%\build\source\BBCheckSystem\bbcheck\...

Gates - Queue Processor Contd.
Merge Resolution Check
Every 15 minutes for branches that have queued, blocked checkins
3-way merge attempted on all files involving:
Depot state of the file
State of file in checkin blocked on
State of file in checkin that’s blocked
0 conflicts on all blocked files unblocks checkin
All branches - %inetroot%\build\wtt_checkin\MergeCheck\...
QueueParse.exe
Part of ReportingParser cycle
Reflects blocking file state into DB
BuildServices - %inetroot%\build\source\QueueParse\...

Gates - Execution

Gates - Execution Contd.
Primary Machine
Maintains checkin state
Executes retail build
Schedules leg builds and SCRAM runs
Deploys
Executes unit tests (C# and X++)
Evaluates task results
Submits (or rejects) changelist
Sends alerts
Updates bugs
Hosts checkin build share
Retained for up to 3 days
\\
\CheckinBuilds\
\

Gates - Execution Contd.
Legs
Build leg
Applies change
Executes a build
Reports result back to primary machine (via file share)
Test leg (SCRAM)
Runs in SCRAM infrastructure
Consumes build hosted on primary machine
Returns result summary to primary machine (via SCRAM.exe/XML file)

Gates - Execution Contd.
Environment
%binFolder% - %inetroot%\build\wtt_checkin
%workFolder% - %inetroot%\build\wtt_checkin\work
%parm
% - Value from WTT job parameter
%workFolder%\WTTFlag.tmp – indicates running in the gates
%binFolder%\SetExternalReferences.cmd
Sets things like %StatisticsDatabaseServer%, %QueueProcessorServer%, etc.

Gates - Tasks
Task Intro
Single script under %binFolder%
Invoked from WTT job task
Format
Set up gate-specific variables (includes getting WTT job parameters)
Log task start in [Statistics].[dbo].[GateTask]
Determine if task can run (disabled, pre-req missing, aborted)
Task setup – E.g. restoring DBs, stopping/starting services
Task logic – E.g. running “build” command
Evaluate result – Write as semaphore file to %workFolder%
Finalize the task in [Statistics].[dbo].[GateTask]
Update task detail in [Statistics]
Exit

Gates - Tasks Contd.
Primary Machine
Preparation
Sync
Apply Changelist
Pre-Checkin Checks
Pre-Build Preparation
Build Retail-platform (if applicable)
Build Retail-application (if applicable)
Post-Build
Schedule Tests
Deploy
MSTest Tests (C# UTs)
Xref-selected tests (if applicable)
Unit tests (X++ CITs)
* Wait
Submit
Bug Update
Log Copy
Dev gate Leg
Preparation
Sync
Apply Changelist
Build Debug-platform
Build Debug-application
Log Copy
SCRAM
BVT
Primitive Tests
Stress Tests

Gates - Tasks Contd.
Adding a task
Create a task script
Create it under %inetroot%\build\wtt_checkin\
.cmd
Add the task to the WTT job (usually requires a new job version)
Update determine results to evaluate if the task ran, passed or failed
Create the task code in [Statistics].[dbo].[GateTaskCode]
Create the task exit codes in [Statistics].[dbo].[GateTaskExitCode]
Update GCM and DB to include configuration of the task.
Create the shell for GateTask to execute it locally (create it under %inetroot%\build\wtt_checkin\
FE.cmd)
Update GateTask.cmd to include the option and execution for the task

Gates - Fast Build
Consists of multiple processes to shorten gate execution time by using “seed” builds
Processes use common infrastructure
%inetrot%\Build\wtt_checkin\FastBuild\...

Gates - Fast Build Eligibility
Eligibility for each process evaluated at the start of the checkin
Sets flag that’s evaluated during the execution of the fast build process

Gates - Fast Build Execution
Valid seed build if:
All churned files themselves qualify for the fast build process
Sync point of seed build is greater than the defined minimum changelist

Gates - Fast Build : FastApp
Uses regular rolling builds of the platform to skip building of the platform at check-in time
Eligible check-ins also skip BVT, MSTest, Primitive and Stress test tasks
Impact: shortens checkins by up to 40%
Invalidated by:
Platform changes: build\ax_tools\bin\DirsFilterManager.exe GetBuildDirectories –FilterGroup Platform
One or more files not in whitelist: build\wtt_checkin\FastBuild\FastAppBuildWhitelist.txt
One or more files in blacklist: build\wtt_checkin\FastBuild\FastAppBuildBlacklist.txt
Retail team – can still use static platform build but forced to run BVT/MSTest tasks still
FAQ for more details, FAQ on Troubleshooting failures

Gates - Fast Build : NightlyBootstrap
Uses nightly build’s DemoData bootstrap instead of building from scratch.
Prior to retail-application, discovers and copies bootstrap local
At build time, consume the bootstrap in \axpackage\dbsync\
In the Post-Build task, skip loading of DemoData
Impact: saves ~14 minutes
Invalidated by changes to:
DP.exe
Build infrastructure
DemoData data set or post-import scripts
FAQ on disabling the process

Gates - Symbol/binary indexing
Every gate build indexed to http://symweb
Deduplication and filtering of binaries (RemoveDuplicates.exe)
Location set in branch config as %GateSynShare%
Process invoked in Post Build
Build\wtt_checkin\CheckinBuildDrop.cmd
Space-based retention handled by the file share
Additional Documentation here

Gates - Synthetic Checkins
Service scheduling regular test checkins to assess the stability of a branch/checkin path
Windows service hosted on RDVMHL1109-05
Source in BuildServices branch at \build\source\SyntheticCheckinService
Current flavors:
SyntheticCheckin – default type, executes like a typical checkin in the branch
BuildOnly – executes a checkin that just does a build in a dedicated pool
Differentiated in reporting on [Statistics].[Stats].[SyntheticType]

Gates - Reporting / Alerts
MBSUSP
MBSRPT
Email
Status
Volume
Build Breaks
Offline Machines
Queue Cycle Report
…many many more…

Gates - Statistics DB / BuildService
Statistics
Primary repository of gate execution and configuration data
Lives on dcsredbbcfs01
FAQ on common tables
Exposed via BuildService (for new development)
BuildService
WCF Service
Expose gate data from [statistics], [unittests]; build data from [dynamicsaxbuilds]
Source in BuildServices branch - \build\source\BuildService
Hosted on dcsrdgtdfs01

Gates - Development
Task Source
%inetroot%\build\wtt_checkin\...
Tools Source
%inetroot%\build\ax_tools\...
%inetroot%\build\source\...
%inetroot%\build\scripts\...
WTT Artifacts
Checkin jobs - $\GatedCheckin
Utilities/Maintenance - $\GatedCheckin\Utilities

Gates - Development Contd.
Maintenance and core tools
BuildServices branch
%inetroot%\build\wtt_checkin\MachineConfiguration\...
%inetroot%\build\wtt_checkin\
Test environment
Run locally using “gatetask” command
Redirect to private WTT pools
Create new synthetic checkin type

Gates - Monitoring / Analysis
Gated Checkin Monitor (GCM)
Gate/build monitoring dashboard
Quick access to common actions (e.g. Resubmit, diff, incident creation)
Gate configuration
Queue processor by branch
Gate tasks by branch
Source in ‘BuildServices’ branch
%inetroot%\build\source\GatedCheckinMonitor\...
Logs
Console log vs. build.exe vs. detail log

BuildTracker Managed Builds

BuildTracker Builds - Official Builds
Purpose
Produce a ship-quality build
All infrastructure in place to be ‘All On’
Provide a rollup of changes to validate against
Cadence
Runs daily with sporadic OOB runs
Kick-off staggered across branches
RainMain* – 9pm PST
RainFND* – 10pm PST

BuildTracker Builds - Buddy Builds
Purpose
Provide engineers with a private build containing their pending changes to be used in pre-checkin validation.
Cadence
On-demand/user initiated
Initiated either directly from the BuildTracker web UI or by running “buddybuild –c
” in CoreXT
Permissions controlled by groups:
redmond\bbscramu
europe\axsrcacc_ram
Resources
Jobs– SCRAM Buddy Build-Redmond – Platform, SCRAM Buddy Build-Redmond - Application
Machine pool – Redmond – Rainier Buddy Build
Drop share - \\dcsrdblddrop03\Build1\BuddyBuilds\Rainier
Retention is space-based and managed external to BuildTracker

BuildTracker Builds - Buddy Build Wizard
Binaries/Configuration - %inetroot%\build\BBCheckinWizard
Source locations
Source control grid (first screen): build\Source\CheckinWizard\src\CheckinComponents\Generic\
Buildtracker screen (second screen): build\BTCheckinWizard\ExternalContributors\Produc\
Managed pack share
Location - \\dcsrdblddrop03\BuddyBuildPackages
Retention – 30 day policy

BuildTracker Builds - Rolling Builds
Purpose
Provide faster build/validation cycle than nightly official builds
Cadence
RainFND Rolling Master
4x per day
Initiates PDL runs per build

BuildTracker Builds - Jobs

BuildTracker Builds – Master Builds
Collection of child jobs to execute as a single job instance
Typically include a build job and some post build activities to run if the build was successful
Usually what daily schedules are set to kick off

CodeSign
Authenticode versus Strong Name
Signing process
Delay signing
Authsign.exe
Certificates
Non-standard formats/processes
CSPKG
APPX
ClickOnce/VSTO
VSIX
Packages (general)
Links:
CodeSign – http://codesign.gtm.microsoft.com/
CodeSignInfo – http://codesigninfo/

Localization
LSBuild
Command line build tool to create localized product files
Uses English version of file as input
Consumes LCL files provided by MBSI
%inetroot%\source\translation\LCL
Use various parsers to determine what to do
Only one custom one – KTD files

Localization - Contd.
PseudoLoc
Simulated localized content
Intended to catch localization problems as early as possible
Pseudoloc examples from AX 2012 R3
Build Legs
Localization-initialize – copies drop\translation to release share
Localization-server – Builds localized KTD files
Retail-Platform - \source\Platform\Localization\GeneratedPASS2
Retail-Application - \source\net64resources
Build localized resource files
%inetroot%\build\localization\localization.binaries.config.xml
Translation-submit – Signs and submits localized files to \source\translation\

QE / LMC
Quality Essentials (QE)
Set of validations focused on ensuring overall release quality
Regulatory/legal
Industry standards
Customer Experience
Security
Consistency with Microsoft Engineering Policies
More information - http://qe/
Last Minute Checks (LMC)
Executes QE and Org-specific build quality checks
Example from DAX62

Post-ship activities
Local and long-term backup
Source code archive
Symbol index/archive

Visual Studio Online (VSO)
What is VSO?
Our instance: http://msdyneng.visualstudio.com
VSO-hosted
Source control via Git
Work item tracking
Build definitions
Build requests/scheduling/reporting
On-Premises
Build controller/agent – RBLD0071AC-TFS
Build drops - \\DCSRDBLDDROP04\Build1\Drop

In-Market

AX6.x Product
AX 2012 Technet site
AX 2012 Architecture
AX 2012 MSDN Developer site

AX6.x Product Contd.
AX32.exe – Win32 client
AX32Serv.exe - Server component running as a Windows Service
MorphX – development environment
Model store – database storing the P-code representing the application inside the platform
AXDBBuild_Model – build-time instance name
AXDBDev_Model – runtime instance name build/gates/deploy.cmd
AXDB – common runtime instance name on other lab machines
Business database – application database housing the data model for business logic
AXDBDev – runtime instance name

Lab Footprint (In-Market)

In-Market Branches

Servicing Concepts
Dependency toolset
Hotfix
Binary vs. Application
Rollup
Cumulative Update (CU)

CoreXT - Our Environment
Overview
Current version – v1
Build.exe to each project node
SYNCHRONIZE_BLOCK, SYNCHRONIZE_DRAIN and dirs file ordering used to drive build ordering
Custom ‘mydirs’ logic used to partition the build tree corresponding to custom build types
Binplace, Placefiles/miscfiles both supported

CoreXT - MSBuild
Current Version – v4.0
Build.exe to MSBuild Handoff
Automatically included in all project makefiles via \build\makefile.inc
\build\makefiles.msbuild creates a response file (msbuild.rsp) in the project directory and invokes msbuild.exe directly.
MSBUILD_PROJECT in sources

CoreXT - Build
Build.exe
Windows build tool
Orchestrator for building/traversing the build graph
Our version – 4.22.4093
‘dirs’ and ‘sources’
i386 is default build target
Depth-first Traversal of Dirs
SYNCHRONIZE_BLOCK/DRAIN
Filter on platform tokens {i386}, {amd64}

CoreXT - Build Contd.
PASS[0|1|2]
PASS0 – Generation of files, copying of static content
PASS1 – C++ compilation
PASS2 – C++ linking, managed code builds, setup/packaging, binplacing
PASS2_BINPLACE – binplacing of built bits (mostly from PASS2)
Project Inclusion
Integrating your project into CoreXT’s build
Dirs, sources, makefile, makefile.inc, placefile

CoreXT - mydirs
If a “mydirs” file is present in a directory, build.exe will use it instead of any “dirs” file to determine what to traverse in that directory.
CoreXT build logic to copy/rename “mydirs.
” to “mydirs” when
matches the current build command.
Example: build layer

CoreXT - Build types

CoreXT - Layer Build
Builds the application component for a specific workload on a specific layer
Near 1:1 mapping of Product workload : Layer
Exceptions – industry solutions in DAX6HF and DAX5SP1HF
Settings for layer product: \build\productConfig.cmd
Example: call %inetroot%\build\productConfig.cmd -Product RUPayrollSE
Build command: build layer .product:
Example: build layer .product:FoundationSE

CoreXT - Layer Build Contd.

CoreXT - Layer Build Contd.

CoreXT - Layer Build Contd.

CoreXT - Layer Build Contd.

CoreXT - VCSDef
MorphX version control settings defined in dynamically-generated file %inetroot%\Definition\VCSDef.xml
Defines source/label file locations for the models in the current layer
Layer template location: %inetroot%\build\ax_public\deployment\tools\vcsdef_
.xml
Template values expand from branch and product configs (e.g. label ID ranges)
Changing layers: SetStartupLayer

CoreXT - Label Files
AX Label (.ald) files edited through the label editor in MorphX.
1..* label files per layer product
Located under %inetroot%\source\application\labels
LabelUtil command – validates correctness of label files prior to submitting

CoreXT – Deploy.cmd
Installs the AX components onto a developer’s machine and initializes the developer environment
Discovery of latest official build
Initialize test environment (import of framework and tests)
Used for local developer and gated check-in deployments
Source: \build\ax_public\deployment\...

Gates

Gates - Execution

Gates - FastKernel
Uses a pre-build kernel build to seed the gate process and skip the retail-kernel and retail-amd64 builds
Seed build share: \\dcsrdgtdfs01\KernelSeedDropShare
Impact – can save over 60 minutes per checkin
Invalidated by:
One or more files not in whitelist: build\wtt_checkin\FastBuild\FastKernelBuildWhitelist.txt
One or more files in blacklist: build\wtt_checkin\FastBuild\FastKernelBuildBlacklist.txt

Gates - FastApp
Uses a pre-build application drop to replace the layer build done prior to PreSubmitSDDump
Seed builds come from the Rolling Hotfix BuildTracker jobs
Impact – can save over 3 hours per checkin
Invalidated by:
One or more files not in whitelist: build\wtt_checkin\FastBuild\FastAppBuildWhitelist.txt
One or more files in blacklist: build\wtt_checkin\FastBuild\FastAppBuildBlacklist.txt

Gates - SDDump
Process used to capture the affected elements of a change
Dependency data submitted to a central DB on DCSRDRPTSQLA
Dependency Database per servicing branch: DependencyDB_
Data later used to generate a dependency bucket for hotfix packages
SDDump Validation
Macro (Pre-Checkin & Pre-Submit SDDump)
Changelist is unique
Changelist description contains bug ID
Bug is in the expected state and contains a valid KB number
Checkin layer is enabled for servicing
Micro (Pre-Submit SDDump)
Changes don’t result in deleted elements
Changes don’t result in elements reverting to base layer logic

Gates - SDDump Contd.
Create scripts to revert/restore non-X++ changes
Prepare environment without changes applied
Restore X++ changes
Import dependency toolset
Prepare models where changed elements will go
Apply changes to AX environment
Execute checklist items for fully built/synced AOT
Disable dependency collection
Validate changed elements follow servicing rules
Extract model and dependency data

Gates - Multi-product Check-in
Only in DAX6HF
Checkin workflow to validate changes to multiple industry solutions in a single checkin
Enables SYP and dependenent overlayered chanes to IS solutions to be submitted in one change
Primary machine does submit and post-submit SDDump

BuildTracker Managed Builds

BuildTracker Builds - Rollup
Purpose
Produce a full kernel + application build to the tip of the current depot state
Checkpoint build to execute automations against
BVT, BAT, Full X++, Full Best Practice, LMC
Refresh localization binaries that subsequent hotfix builds will use
Build type for Cumulative Update releases
Produces update package for all changes since RTM
Cadence
Full Rollup master with validation occurs ~3 times per week
Runs daily when stabilizing a CU release

BuildTracker Builds - Application Hotfix
Purpose
Produce a granular hotfix containing the X++ change fixing a specific product issue + any dependent change since RTM
Builds Application against RTM kernel to assert no dependency taken on binary hotfixes
Cadence
On-demand.
Auto-initiated in R3 branches on successful check-in. Otherwise manually initiated by issue owner.
Also produced as part of Rolling Hotfix schedule

BuildTracker Builds - Binary Hotfix
Purpose
Produce a cumulative build of the kernel and frameworks components
Create patch installers (.msp) to apply fix to existing deployments
Cadence
On-demand.
Also produced as part of Rolling Hotfix schedule

BuildTracker Builds - Jobs

UA Build

UABuild – Overview (Core Concepts)
UABuild – UABuild is our documentation build process
Content is generated for word doc format, AX Help Server format, MSDN, and Technet format.
DXStudio – Content editor used by writers
Content is store in DB: Axapta01 on DBServer: DCSRDUASQL01
CoreXT/SD – Used for source control and to build content branch: MainUA
Get For Build – Process to pull content from DXStudio and check it into Source Depot.

UABuild – Core Build Types
A GFB (Get For Build) pulls content from a DxStudio project into Source Depot
HXS Build – Builds content into HxS format, and optionally publishes the compiled content to MTPS (MSDN or TechNet) using DxPublisher
HelpServer -- Builds content into a format suitable for the Dynamics AX Help Server
DocBuild – Builds content into Doc, Chm, and/or PDF formats.
ManagedReference -- Builds MRef content into HxS format.

UABuild - Core Build Flow

Daily Activities

Assessing Official Build State
Nightly builds scheduled, ran as expected, kicked off test runs
Expected state - everything runs as scheduled and finishes in a timely manner
Troubleshooting - For scheduling problems, check BuildTracker. For failures and extended execution times, analyze the build logs.
Evaluate build drop share resource check report
Expected state - all drop shares report a healthy amount of free space
Troubleshooting - if any drop shares show errors about low space (like the one linked above)
Run the bt.exe tool to get failed retention jobs and retry them. If they continue failing, manually delete the drops
If all retention jobs are passing, investigate if per-build size has increased or retention policies have changed to result in more or larger builds on the drop. Log bugs accordingly

Assessing BuddyBuild State
Check on buddy build queue
Expected state - majority of the time the queue is shrinking or 0 and jobs are completing (queue can be growing during peak times)
Troubleshooting - Check on machine availability (e.g. are any offline). Check if running jobs are experiencing hangs or other systemic issues causing them to take longer

Assessing Gated Check-in State
Check queue length on MBSRPT
Expected state - Queue as small as possible
Troubleshooting
Queue processor not running
If queue is closed due to a break, work with owner of the issue to drive to resolution then reopen the queue.
Check if checkins are taking longer to complete or if there are fewer resources available to schedule against
Check running checkins
Expected state - no checkins experiencing systemic issues resulting affecting the execution of the gate workflow
Troubleshooting – check WTT for machine states, check recent changes to build\wtt_checkin