CLI tool: Add a script to generate the config files used in testing

[Aniruddh25]

Today, we use the checked in files for our integration testing. Instead, write a script to use the CLI tool and generate these config files to be useful in testing.

[Aniruddh25]

Moving to M2 since this is more of a fundamentals that we will continue to fix beyond M1.5