HFSS15: Large Scale DSO Job Monitoring

Large Scale DSO avoids detailed intra-variation monitoring is avoided as it increases network traffic for large-scale jobs. Large Scale DSO jobs are monitored as below:

• Cluster monitoring tools: The resource usage (CPU, Memory, Network) of Large Scale DSO jobs is monitored using standard cluster monitoring tools. Such job-neutral resource monitoring is ideal as it uses negligible network bandwidth, CPU/Memory.

• Detailed monitoring of analysis of a variation: For any detailed monitoring you must examine the information provided in the job's log files. Specifically, the large-scale-dso job writes detailed logs conveying information regarding the machines where engines are running and the local storage location of per-engine distributed database. With such information, you can login to individual machines for deeper probing of each distributed engine. Following logs are available:

• Per-node logs:

There is one 'desktopjob.log' file per node assigned to the job. This log contains information regarding the node such as name, local storage folder, number of engines started on this node, etc. It is located in <workdir>/<jobid>/r<nodeIndex>. E.g. "<workdir>/<jobid>/r0" has desktopjob.log corresponding to the engines running on the first node of job, while "<workdir>/<jobid>/r2" has logs corresponding to engines running on third node

• Per-engine logs:

There is one desktopjob.log file per distributed engine. It is located in <workdir>/<jobid>/r<nodeIndex>/r<coreIndex>. E.g. "<workdir>/<jobid>/r0/r0" has logs corresponding to first engine running on first node, while "<workdir>/<jobid>/r1/r2" has logs corresponding to third engine running on second node. Engine unique information (such as local storage of this engine) is logged here

• Parametric analysis log:

This log file is located in '<workdir>/<jobid>/r<nodeIndex>/r<coreIndex>' folder and corresponds to Desktop's local-machine parametric 'batchsolve'. It is available only at the end of analysis and contains information regarding the variations solved by this engine and any info/warning/error messages.

• Root desktopjob.log:

This is the top-level log that logs job distribution information such as hierarchical activation and the list of nodes assigned to this job

• GM specific extraction

In R14, GM extraction is OFF, by default. It can be turned on through setting environment variable “ANSOFT_ENABLE_GM_EXTRACTION” to a value of 1