Are you using the “spooling” option when you submit runs? It might save you some run time.
This can be a useful way of speeding up your optimization runs. “What does it do?” you ask. It makes a zip of all the files needed for a run in a designated spooling folder, unzips it into a run-time working directory, then gathers the results back afterwards into the named output folder for that case. Why does it make things faster? The biggest benefit comes from being able to run more than one optimization at a time, if you have a computer with multiple CPUs and the multi-core extension to GRTMPS.
There are 26 cases in the small model we use for the course. I can run them
 |
 |
| without spooling, one by one, in 11 minutes or | with spooling, 6 at a time in 4 minutes |
However, you can have some benefit from spooling even if you are running one-at-a-time – so read on, even if you don’t have multi-core (yet).
Spooling is configured via the File Settings menu in the Queue Manager.
Spooling requires a Temporary Directory, where you have read / write / delete access. This defaults to blank so you must select something here before you start.
- Use a short name. This maximizes the number of characters left for the run-id.
- Use a local drive, if possible, to avoid adding network traffic delays to your run times. This can give you a noticeable improvement in speed if your output folder is on a server.
- Exempt it from any mirroring or back-up software. These files are temporary, so there is no need to keep copies.
- Exempt them from your anti-virus scans and real-time protection. That will make your runs faster as security software finds it hard to keep up with the creating and deleting.
If you ever have “Access Denied” errors when running (spooled or not), those are usually due to clashes with software doing background scanning. You are more likely to be allowed to exempt one specific dedicated temporary directory than a whole drive.
Special Spooling Directories is a list of locations with files that should not be included in the zip. When a job runs, it will use the file from its normal location. I have exempted my GRTMPS system folder, as anything from there, primarily the GRTMPS PGB, is used read-only. There is no need to copy this rather large file multiple times.
If you need any additional files in the run-time directory, that are not part of the model data, you can use the Extra Data panel to add them to the list.
You can see that “output_21” is older than the other files. It is a left-over from a run that did not finish successfully. There might have been an error in the run, or maybe you forced a shut down because you were about to board or plane or maybe Windows 10 decided to restart to finish installing an update in the middle of the night, because it doesn’t treat the machine as active if you are not typing or clicking even if you have every CPU occupied. If the run crashed and there is apparently no output, these left-over input and output files are sometimes useful for debugging. Sometimes they are just empty. In any case, it is a good house-keeping practice to clean these out periodically. Sometime when you are not doing any runs, just open the spooling folder and delete everything to reclaim the number and the disk space.
From Kathy's Desk 13th April 2017.
Comments and suggestions gratefully received via the usual e-mail addresses or here.
You may also use this form to ask to be added to the distribution list to be notified via e-mail when new articles are posted.