Been testing out the Production Validator toolkit, just to see what we can/not do with it. Seems really interesting and there seem to be some use cases for it that can really streamline some upgrades (or at least parts of upgrades) but I was running into so many hurdles with the documentation. I am curious if anyone else has used it.

Did you experience any issues getting it working? Any clarification that you would have liked in the documentation? any use cases that you worked through that made it particularly valuable? etc?

The hurdles I experienced included:

1. The documentation stated that the toolkit comes pre-installed in 2024.3+ but I got to a point where it kept throwing an <METHOD DOES NOT EXIST> error. Upon investigation the toolkit either wasn't installed or only part of it was installed so I had to download and install the package from WRC.
2. The documentation was frequently a little vague about things like some of the terminal command inputs. For example, it isn't clear what the arguments were for in the following command (later I figured out that it creates a new temporary namespace wherein the copied production is loaded and run):

   HSLIB>set sc = ##class(HS.InteropTools.HL7.Compare.ProductionExtract).ConfigureAndImport(<Testing_Namespace>,<Full_Path>)

3. Since I was simulating an upgrade between 2024.1 and 2024.3 I wasn't able to get an output that resulted in any differences. The documentation doesn't have any kind of test or demo baked in, just screenshots so I'm still not 100% sure what it can handle and what it can't (see Outstanding Questions below)

Outstanding Questions

1. I don't have any demo prods that use BPLs so there is a question regarding how the PV manages when those are present
2. Doesn't it seem that a good amount of this could be scripted since these are commands that mostly don't take user input? (and if they do wouldn't it make sense to get as much scripted as possible?)
3. I had not worked with the syntax that produces the JSON and initially thought that it was invalid JSON (what with the doubled double quotes, e.g. ""example""). Would a link to that syntactical documentation be unnecessary? or would an example of the JSON output be out of place?
4. It isn't totally clear, until you bang your head into every wall, what the workflow is, is there a way to clarify that with a diagram/wireframe of the workflow? or is that inappropriate?
5. Does this allow for rerunning a test once it has been run once? I tried doing so but kept bumping into errors like "DB already exists", "namespace already exists", etc. so I had to create a new NS and delete the COMPARE file each time. Is that just how it runs or is there a way to rerun that isn't easily identifiable?
6. Initially I had been trying to get this all running in a container (Docker) and seemed to be running into too many challenges to justify the time spent. I ended up running 2 parallel instances (e.g. 2024.1 and 2024.3) locally and that worked a treat. But is there a way anyone has been able to test this using a docker container?

Interested in what you all have experienced and if you have made any customizations of this or if you have been able to figure out any workarounds, etc.

Thanks