Skip to content

Build evaluation report in CI and when enabled #5903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shyamnamboodiripad merged 12 commits into from
Feb 14, 2025
Merged

Build evaluation report in CI and when enabled #5903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
66e9043
Move targets into a Directory.Build.targets file
peterwald Feb 13, 2025
27909ec
Update the output path for the VSIX
peterwald Feb 13, 2025
1bd3987
Be sure to pull in parent target files
peterwald Feb 13, 2025
922ddfe
Disable report build in inner loop, enable in CI
peterwald Feb 13, 2025
352ed65
Perform npm install incrementally.
peterwald Feb 13, 2025
23869ca
Remove msbuild property quotes
peterwald Feb 13, 2025
065ccdb
Use env variable to enable local report builds.
peterwald Feb 13, 2025
2a1b9b9
Merge branch 'main' into ext-build
peterwald Feb 13, 2025
ed12a67
Merge branch 'main' into ext-build
shyamnamboodiripad Feb 13, 2025
21ead60
Also use Directory.Build.targets for Console.csproj
shyamnamboodiripad Feb 13, 2025
270a3ca
Merge branch 'main' into ext-build
shyamnamboodiripad Feb 13, 2025
7b69d92
Unify enviroment variable and property
shyamnamboodiripad Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion eng/pipelines/templates/BuildAndTest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,13 @@ steps:
-configuration ${{ parameters.buildConfig }}
-warnAsError ${{ parameters.warnAsError }}
/bl:${{ parameters.repoLogPath }}/build.binlog
/p:EnableEvaluationReportBuild=true
$(_OfficialBuildIdArgs)
displayName: Build

- ${{ if eq(parameters.isWindows, 'true') }}:
- pwsh: |
$(Build.SourcesDirectory)/src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/TypeScript/azure-devops-report/build.ps1 -OutputPath $(Build.ArtifactStagingDirectory)\VSIX
$(Build.SourcesDirectory)/src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/TypeScript/azure-devops-report/build.ps1 -OutputPath $(Build.Arcade.ArtifactsPath)/packages
displayName: Build Azure DevOps plugin

- ${{ if ne(parameters.skipTests, 'true') }}:
Expand Down
33 changes: 33 additions & 0 deletions src/Libraries/Microsoft.Extensions.AI.Evaluation.Console/Directory.Build.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<Project>
<Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.targets', '$(MSBuildThisFileDirectory)../'))" />

<!-- Generate a version constant that can be accessed from the C# code. -->
<Target Name="GenerateVersionConstant" BeforeTargets="CoreCompile">
<PropertyGroup>
<_ConstantsFilePath>$(IntermediateOutputPath)Constants.g.cs</_ConstantsFilePath>
<_Lines>
// <auto-generated>
// This file is auto-generated by MSBuild.
// </auto-generated>

namespace $(RootNamespace)%3B

internal static class Constants
{
public const string Version = "$(Version)"%3B
}
</_Lines>
</PropertyGroup>

<Message Text="Generating $(_ConstantsFilePath)." Importance="high" />

<WriteLinesToFile File="$(_ConstantsFilePath)"
Lines="$(_Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />

<ItemGroup>
<Compile Include="$(_ConstantsFilePath)" />
</ItemGroup>
</Target>
</Project>
30 changes: 0 additions & 30 deletions ...rosoft.Extensions.AI.Evaluation.Console/Microsoft.Extensions.AI.Evaluation.Console.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,34 +33,4 @@
<ItemGroup>
<InternalsVisibleToTest Include="Microsoft.Extensions.AI.Evaluation.Console.Tests" />
</ItemGroup>

<!-- Generate a version constant that can be accessed from the C# code. -->
<Target Name="GenerateVersionConstant" BeforeTargets="CoreCompile">
<PropertyGroup>
<ConstantsFilePath>$(IntermediateOutputPath)Constants.g.cs</ConstantsFilePath>
<Lines>
// <auto-generated>
// This file is auto-generated by MSBuild.
// </auto-generated>

namespace Microsoft.Extensions.AI.Evaluation.Console%3B

internal static class Constants
{
public const string Version = "$(Version)"%3B
}
</Lines>
</PropertyGroup>

<Message Text="Generating $(ConstantsFilePath)." Importance="high" />

<WriteLinesToFile File="$(ConstantsFilePath)"
Lines="$(Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />

<ItemGroup>
<Compile Include="$(ConstantsFilePath)" />
</ItemGroup>
</Target>
</Project>
65 changes: 9 additions & 56 deletions ...nsions.AI.Evaluation.Reporting/CSharp/Microsoft.Extensions.AI.Evaluation.Reporting.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">

<!--
NOTE: To build and test the TypeScript code that this project depends on, set the 'EnableEvaluationReportBuild'
environment variable to 'true'. Note that the TypeScript build depends on Node.js tools. So you will need to
install the latest version of Node.js from https://nodejs.org/ as a prerequisite before setting the
'EnableEvaluationReportBuild' environment variable.
-->
Comment thread
shyamnamboodiripad marked this conversation as resolved.

<PropertyGroup>
<Description>A library for aggregating and reporting evaluation data. This library also includes support for caching LLM responses.</Description>
<TargetFrameworks>$(TargetFrameworks);netstandard2.0</TargetFrameworks>
Expand All @@ -23,7 +30,8 @@
</ItemGroup>

<ItemGroup>
<EmbeddedResource Include="..\TypeScript\html-report\dist\index.html" Visible="false">
<EmbeddedResource Condition="'$(EnableEvaluationReportBuild)'=='true'"
Include="..\TypeScript\html-report\dist\index.html" Visible="false">
Comment thread
peterwald marked this conversation as resolved.
<LogicalName>Reporting.HTML.index.html</LogicalName>
</EmbeddedResource>
</ItemGroup>
Expand All @@ -33,59 +41,4 @@
<InternalsVisibleToTest Include="Microsoft.Extensions.AI.Evaluation.Reporting.Tests" />
</ItemGroup>

<!-- Build the report code before this, so it can be embedded.
Comment thread
peterwald marked this conversation as resolved.
Setting the BeforeTargets=DispatchToInnerBuilds allows it to build only once before
the multiple parallel builds are invoked for each target framework. -->
<Target Name="BuildTypescriptBundle"
BeforeTargets="DispatchToInnerBuilds"
Outputs="..\TypeScript\html-report\dist\index.html">
<Message Text="Build the Typescript bundle" Importance="high" />
<Exec Command="npm install" WorkingDirectory="..\TypeScript" />
<Exec Command="node build-if-out-of-date.js" WorkingDirectory="..\TypeScript" />
</Target>

<!-- Generate a version constant that can be accessed from the C# code. -->
<Target Name="GenerateVersionConstant" BeforeTargets="CoreCompile">
<PropertyGroup>
<ConstantsFilePath>$(IntermediateOutputPath)Constants.g.cs</ConstantsFilePath>
<Lines>
// <auto-generated>
// This file is auto-generated by MSBuild.
// </auto-generated>

namespace Microsoft.Extensions.AI.Evaluation.Reporting%3B

internal static class Constants
{
public const string Version = "$(Version)"%3B
}
</Lines>
</PropertyGroup>

<Message Text="Generating $(ConstantsFilePath)." Importance="high" />

<WriteLinesToFile File="$(ConstantsFilePath)"
Lines="$(Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />

<ItemGroup>
<Compile Include="$(ConstantsFilePath)" />
</ItemGroup>
</Target>

<!-- Generate a version file that can be accessed from the script that builds the Azure DevOps Extension pacakge. -->
<Target Name="StampVSIXPackageVersion" BeforeTargets="CoreCompile">
<PropertyGroup>
<VSIXPackageVersionFile>..\TypeScript\azure-devops-report\VSIXPackageVersion.json</VSIXPackageVersionFile>
<VSIXVersion>$(VersionPrefix).42424242</VSIXVersion>
<VSIXVersion Condition=" '$(VersionSuffixDateStamp)'!='' ">$(VersionPrefix).$(VersionSuffixDateStamp)$(VersionSuffixBuildOfTheDayPadded)</VSIXVersion>
<Lines>{"PackageVersion":"$(VSIXVersion)"}</Lines>
</PropertyGroup>
<WriteLinesToFile File="$(VSIXPackageVersionFile)"
Lines="$(Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />
</Target>

</Project>
76 changes: 76 additions & 0 deletions src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/Directory.Build.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<Project>
<Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.targets', '$(MSBuildThisFileDirectory)../'))" />

<PropertyGroup>
<_TypescriptRootPath>$(MSBuildThisFileDirectory)/TypeScript</_TypescriptRootPath>
<_NpmInstallStampFile>$(_TypescriptRootPath)/node_modules/.installed</_NpmInstallStampFile>
</PropertyGroup>

<Target Name="NpmInstall"
Condition="'$(EnableEvaluationReportBuild)'=='true'"
Inputs="$(_TypescriptRootPath)/package.json"
Outputs="$(_NpmInstallStampFile)">
<Exec Command="npm install" WorkingDirectory="$(_TypescriptRootPath)" />
<Touch Files="$(_NpmInstallStampFile)" AlwaysCreate="true" />
</Target>

<!-- Build the report code before this, so it can be embedded.
Setting the BeforeTargets=DispatchToInnerBuilds allows it to build only once before
the multiple parallel builds are invoked for each target framework. -->
<Target Name="BuildTypescriptBundle"
DependsOnTargets="NpmInstall"
BeforeTargets="DispatchToInnerBuilds"
Condition="'$(EnableEvaluationReportBuild)'=='true'"
Inputs="$(_TypescriptRootPath)/**/*.*"
Outputs="$(_TypescriptRootPath)/html-report/dist/index.html">
<Message Text="Build the Typescript bundle" Importance="high" />
<Exec Command="node build-if-out-of-date.js" WorkingDirectory="$(_TypescriptRootPath)" />
</Target>

<!-- Generate a version constant that can be accessed from the C# code. -->
<Target Name="GenerateVersionConstant" BeforeTargets="CoreCompile">
<PropertyGroup>
<_ConstantsFilePath>$(IntermediateOutputPath)Constants.g.cs</_ConstantsFilePath>
<_Lines>
// <auto-generated>
// This file is auto-generated by MSBuild.
// </auto-generated>

namespace $(RootNamespace)%3B

internal static class Constants
{
public const string Version = "$(Version)"%3B
}
</_Lines>
</PropertyGroup>

<Message Text="Generating $(_ConstantsFilePath)." Importance="high" />

<WriteLinesToFile File="$(_ConstantsFilePath)"
Lines="$(_Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />

<ItemGroup>
<Compile Include="$(_ConstantsFilePath)" />
</ItemGroup>
</Target>

<!-- Generate a version file that can be accessed from the script that builds the Azure DevOps Extension pacakge. -->
<Target Name="StampVSIXPackageVersion" BeforeTargets="CoreCompile">
<PropertyGroup>
<_VSIXPackageVersionFile>$(MSBuildThisFileDirectory)\TypeScript\azure-devops-report\VSIXPackageVersion.json</_VSIXPackageVersionFile>

<_VSIXVersion>$(VersionPrefix).42424242</_VSIXVersion>
<_VSIXVersion Condition=" '$(VersionSuffixDateStamp)'!='' ">$(VersionPrefix).$(VersionSuffixDateStamp)$(VersionSuffixBuildOfTheDayPadded)</_VSIXVersion>

<_Lines>{"PackageVersion":"$(_VSIXVersion)"}</_Lines>
</PropertyGroup>

<WriteLinesToFile File="$(_VSIXPackageVersionFile)"
Lines="$(_Lines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />
</Target>
</Project>