AutoML 2.0: how to set sampling key

Is your feature request related to a problem? Please describe.
I am trying to run crossvalidation with a sampling key.

However, .SetDataset(data, fold: 10); only accepts the number of folds. I can create cross-validation set with sampling key using ctx.Data.CrossValidationSplit, but the .SetDataset does not have overload to accept a set of TrainTestData.

How can I set my own IReadOnlyList<TrainTestData> cvdata using the new AutoML API ?

Examples:

AutoMLExperiment experiment = ctx.Auto().CreateExperiment();

IReadOnlyList<TrainTestData> cvdata = ctx.Data.CrossValidationSplit(data, numberOfFolds: 10, samplingKeyColumnName: SamplingKeyColumn); // how to use this in the experiment?

            // Configure experiment
experiment
                .SetPipeline(pipeline)
                .SetBinaryClassificationMetric(BinaryClassificationMetric.Accuracy, labelColumn: columnInference.ColumnInformation.LabelColumnName)
                .SetTrainingTimeInSeconds(60)
                .SetDataset(data, fold: 10); //  how to set sampling key?

Is this enough, and how could I confirm sampling key is being used?
How about if I read it from a binary IDV file, then columnInference is not even used when reading, unless sampling column name is saved to the binary file when serializing.

            columnInference.ColumnInformation.SamplingKeyColumnName = SamplingKeyColumn;
            // Create text loader
            TextLoader loader = ctx.Data.CreateTextLoader(columnInference.TextLoaderOptions);
            // Load data into IDataView
            data = loader.Load(trainingDataFiles.ToArray());

(edit: I am now running a test, but so far it would appear it is not using the sampling key due to a accuracy that seems unrealistically high, I may try again with anohter dataset I have results to compare to)

I believe these 3 small updates will fix my issue, at least for now:

https://github.com/torronen/machinelearning/commit/34a0ca1eb9b2e13cf2742be352182072fdf27252

@torronen

@torronen could you submit a pr with those fixes?

@LittleLittleCloud can you take a look?

Navigation Menu

Repository navigation

AutoML 2.0: how to set sampling key #6643

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AutoML 2.0: how to set sampling key #6643

Description

Activity

torronen commented on May 3, 2023

torronen commented on May 3, 2023

michaelgsharp commented on May 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions