Various changes to enable FIO and PostgreSQL tests in (Docker/ACI) container

[micromaomao]

Added a DummyDiskManager that just returns one root disk. However this is a TODO item - we want to make it use df to determine any additional volumes mounted to the container, and report correct filesystem size (makes more sense in ACI where it's the sandbox VHD's size), as well as any mounted volumes (need to experiment to see how volumes actually turn up in containers / ACI)

Test command:

./VirtualClient --profile=PERF-IO-FIO.json --profile=MONITORS-NONE.json --iterations=1 --ltf --log-level=Trace --pm FileSize=1G --scenarios -DiskFill --packages=https://virtualclient.blob.core.windows.net/packages

./VirtualClient --profile=PERF-POSTGRESQL-HAMMERDB-TPCC.json --profile=MONITORS-NONE.json --iterations=1 --ltf --log-level=Trace --pm DiskFilter=osdisk:true --packages=https://virtualclient.blob.core.windows.net/packages --pm WarehouseCount=40 --pm SharedMemoryBuffer=2000

For some reason this currently still fails in Docker (testing TODO in ACI):

[02/11/2025 21:35:55] Executing: python3 /VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py --dbName hammerdb_tpcc --serverIp 127.0.0.1 --password ... --port 5432 --inMemory 2000
[02/11/2025 21:35:55] PostgreSQLServerConfiguration.PostgreSQLServerConfiguration.ProcessDetails
[02/11/2025 21:35:55] could not change directory to "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64": Permission denied
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
Traceback (most recent call last):
  File "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py", line 49, in <module>
    subprocess.run(f'sudo -u postgres psql -c "ALTER USER postgres PASSWORD \'{password}\';"', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY=';"' returned non-zero exit status 2 (error/exit code=1, command=python3 /VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py --dbName hammerdb_tpcc --serverIp 127.0.0.1 --password p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY= --port 5432 --inMemory 2000).

StandardOutput: Copying pg_hba.conf file to change authentication to peer to allow user authentication
Restarting the PostgreSQL services on the system.


StandardError: could not change directory to "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64": Permission denied
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
Traceback (most recent call last):
  File "/VirtualClient/packages/postgresql.14.0.0.rev1/linux-x64/configure-server.py", line 49, in <module>
    subprocess.run(f'sudo -u postgres psql -c "ALTER USER postgres PASSWORD \'{password}\';"', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'p5P4ospdxIF4LU7P+fWIWFvrnUeoJ8paCAXlZ7O1TgY=';"' returned non-zero exit status 2.


   at VirtualClient.ProcessExtensions.ThrowErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 385
   at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, IEnumerable`1 successCodes, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 228
   at VirtualClient.ProcessExtensions.ThrowIfDependencyInstallationFailed(IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 180
   at VirtualClient.Dependencies.PostgreSQLServerConfiguration.ConfigurePostgreSQLServerAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Dependencies/PostgreSQLServer/PostgreSQLServerConfiguration.cs:line 196
   at VirtualClient.Dependencies.PostgreSQLServerConfiguration.ExecuteAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Dependencies/PostgreSQLServer/PostgreSQLServerConfiguration.cs:line 160
   at VirtualClient.Contracts.VirtualClientComponent.<>c__DisplayClass130_1.<<ExecuteAsync>b__4>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientComponent.cs:line 659
--- End of stack trace from previous location ---
   at VirtualClient.Contracts.VirtualClientLoggingExtensions.<>c__DisplayClass17_0.<<LogMessageAsync>b__0>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 603
--- End of stack trace from previous location ---
   at VirtualClient.Contracts.VirtualClientLoggingExtensions.LogMessageAsync[TResult](ILogger logger, String message, LogLevel level, EventContext eventContext, Func`1 body) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 641

But works on WSL using DOTNET_RUNNING_IN_CONTAINER=true

Not written any unit tests at this point, nor have I tested if the profile still works in a VM (I think it should)

[yangpanMS]

Great to see this POC, blocking in case of accidental check in. We will transform this into LinuxContainerDiskManager before checking in

[micromaomao]

Note that postgresql still doesn't work "out of the box", and we need a mock systemctl in the container image. However I decided to not bake that into VirtualClient - maybe the configure scripts in the package need to be updated.

#!/usr/bin/bash

if [ "$#" -ne 2 ]; then
  echo "Usage: $0 {start|restart|stop} postgresql"
  exit 1
fi

action="$1"
service="$2"

if [ "$service" != "postgresql" ]; then
  echo "Error: Unsupported service '$service'. Only 'postgresql' is supported."
  exit 1
fi

case "$action" in
  start)
    /usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main start
    exit $?
    ;;
  restart)
    /usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main stop || true
    /usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main start
    exit $?
    ;;
  stop)
    /usr/bin/pg_ctlcluster --skip-systemctl-redirect 14-main stop
    exit $?
    ;;
  *)
    echo "Error: Unsupported action '$action'. Use start, restart, or stop."
    exit 1
    ;;
esac

[micromaomao]

Actually there is one more thing that need to be fixed - calculate({SystemMemoryMegabytes}) doesn't work because it uses dmidecode. Maybe it can use something like free?

[micromaomao]

Note: the SuperuserPassword parameter change is not actually required, and since it causes CodeQL alert I can drop it

[micromaomao]

Looks like unfortunately the sysbench postgresql tests still doesn't quite work:

[02/13/2025 17:39:21] Profile: Action = SysbenchConfiguration (scenario=PopulatePostgreSQLDatabase)
[02/13/2025 17:39:21] SysbenchConfiguration.ExecuteStart
[02/13/2025 17:39:21] PackageManager.GetPackageStart
[02/13/2025 17:39:21] PackageManager.GetPackageStop
[02/13/2025 17:39:21] PackageManager.GetPackageStart
[02/13/2025 17:39:21] PackageManager.GetPackageStop
[02/13/2025 17:39:21] Executing: python3 /opt/virtualclient/packages/sysbench-1.0.20.rev1/configure-workload-generator.py --distro Ubuntu --databaseSystem PostgreSQL --packagePath /opt/virtualclient/packages/sysbench-1.0.20.rev1
[02/13/2025 17:39:21] SysbenchConfiguration.SysbenchExecutor.ProcessDetails
[02/13/2025 17:39:21] usage: configure-workload-generator.py [-h] -d DISTRO [-y SERVERTYPE] -p
                                       PACKAGEPATH
configure-workload-generator.py: error: unrecognized arguments: --databaseSystem PostgreSQL (error/exit code=2, command=python3 /opt/virtualclient/packages/sysbench-1.0.20.rev1/configure-workload-generator.py --distro Ubuntu --databaseSystem PostgreSQL --packagePath /opt/virtualclient/packages/sysbench-1.0.20.rev1).

StandardError: usage: configure-workload-generator.py [-h] -d DISTRO [-y SERVERTYPE] -p
                                       PACKAGEPATH
configure-workload-generator.py: error: unrecognized arguments: --databaseSystem PostgreSQL


   at VirtualClient.ProcessExtensions.ThrowErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 385
   at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, IEnumerable`1 successCodes, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 228
   at VirtualClient.ProcessExtensions.ThrowIfErrored[TError](IProcessProxy process, String errorMessage, ErrorReason errorReason) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Core/ProcessExtensions.cs:line 209
   at VirtualClient.Actions.SysbenchExecutor.InitializeExecutablesAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Actions/Sysbench/SysbenchExecutor.cs:line 388
   at VirtualClient.Actions.SysbenchExecutor.InitializeAsync(EventContext telemetryContext, CancellationToken cancellationToken) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Actions/Sysbench/SysbenchExecutor.cs:line 319
   at VirtualClient.Contracts.VirtualClientComponent.<>c__DisplayClass130_1.<<ExecuteAsync>b__4>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientComponent.cs:line 656
--- End of stack trace from previous location ---
   at VirtualClient.Contracts.VirtualClientLoggingExtensions.<>c__DisplayClass17_0.<<LogMessageAsync>b__0>d.MoveNext() in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 603
--- End of stack trace from previous location ---
   at VirtualClient.Contracts.VirtualClientLoggingExtensions.LogMessageAsync[TResult](ILogger logger, String message, LogLevel level, EventContext eventContext, Func`1 body) in /home/mao/src/VirtualClient/src/VirtualClient/VirtualClient.Contracts/VirtualClientLoggingExtensions.cs:line 641

[02/13/2025 17:39:21] SysbenchConfiguration.ExecuteStop