MultiProcessAgentParallelRunner

class pams.runners.MultiProcessAgentParallelRunner(settings, prng=None, logger=None, simulator_class=<class 'pams.simulator.Simulator'>)

Multi Process Agent Parallel runner class. This is experimental.

In this runner, only pams.agents.Agent.submit_orders() of normal (non high-frequency) agents is parallelized in each step using concurrent.futures.ProcessPoolExecutor. See pams.runners.MultiThreadAgentParallelRunner for the details of the parallelization.

The start method of multiprocessing for the worker processes can be set by simulation.startMethod in the config, e.g., "spawn". It must be one of multiprocessing.get_all_start_methods() on the platform. If it is not set, default_start_method is used; if that is None, the default start method of the platform is used.

In addition to the internal methods listed in pams.runners.MultiThreadAgentParallelRunner, subclasses can customize the worker processes by overriding the following members:

• default_start_method: the start method used when simulation.startMethod is not set. For example, "spawn" is required for libraries that are not fork-safe, such as PyTorch with CUDA, TensorFlow, and JAX.

• _get_mp_context: the multiprocessing context of the worker processes.

• _receive_synced_attributes_from_worker: sets the attributes listed in pams.agents.Agent.synced_attributes on the agent on the main process.

Note

The agents and the markets are pickled and copied to a worker process for each task. Therefore, only the returned orders, the state of the agent’s pseudo random number generator, and the attributes listed in pams.agents.Agent.synced_attributes are reflected to the agent on the main process. Other attributes modified in pams.agents.Agent.submit_orders() are discarded, although in-place changes to memory that a library shares between processes, e.g., PyTorch tensors on the CPU, can remain. A warning is issued when an agent assigns or deletes an attribute in its __dict__ that is not listed, at most once for each agent class and attribute name per runner. Assigning a str, a number, or a tuple of them equal to the old value is not reported, and changes made in place, e.g., by appending to a list, and changes to attributes in __slots__ are not detected. The listed attributes are sent back after every call, even if the agent submits no orders, and are set on the agent before the orders are handled. The listed values and the orders of all the agents of a task are pickled together, so references among them are kept as far as pickling keeps them, e.g., an order that an agent returns and also keeps in a listed attribute is one object on the main process. The objects of the simulation among them, i.e., the simulator, the logger, the fundamentals, the markets, their order books and the orders in the books, the agents, the sessions, the events, the event hooks, the pseudo random number generators of the simulator, the markets, the agents, the sessions, and the events, and the lists, dicts, and sets that these objects except the agents hold as attributes, are sent back as references to the objects on the main process, so the listed values can refer to them. Other objects are copies, including orders that are in no order book when the task is sent, e.g., executed, canceled, or expired orders, which are new copies after every call; such a copy is equal to the order but is not the object that other agents or unlisted attributes refer to. An unlisted attribute that refers to a listed object keeps referring to the old object, so such attributes must be listed too. Other objects shared with other agents, e.g., a model shared by agents, are copied, and which agents share a copy afterwards depends on how the agents are split into tasks. "simulator", "logger", and "prng", which the runner manages, and "__dict__", which holds all the attributes of the agent, cannot be listed. The names are read on the main process whenever the agent is asked, and a ValueError is raised if they are invalid, both then and in _setup. If the listed values or the orders of an agent cannot be pickled, a RuntimeError that names the agent is raised. See Agents that change their own attributes for an example with PyTorch and other caveats. User-defined agents for this runner can also keep their states through the callbacks such as pams.agents.Agent.submitted_order() and pams.agents.Agent.executed_order(), which are called on the main process, or derive their states from markets.

Note

User-defined classes (agents, markets, loggers, etc.) must be picklable. Especially when the start method of multiprocessing is spawn or forkserver (e.g., macOS and Windows), the definition of user-defined classes must be importable from worker processes, i.e., they should be saved to a module file rather than defined in __main__ or in an interactive session.

Warning

This class is much slower than pams.runners.MultiThreadAgentParallelRunner because of the cost of pickling. Agents and markets refer to the simulator, so every task pickles and unpickles the whole simulation: all the agents, all the markets with their histories, the events, and the logger. For example, with 1000 pams.agents.FCNAgent agents, the pickled data is about 4 MB, mostly the states of the agents’ pseudo random number generators, and it takes tens of milliseconds to pickle and as long to unpickle. To reduce the cost, the agents asked in parallel are split into at most numParallel chunks of consecutive agents, and each chunk is one task. Thus, the simulation is pickled at most numParallel times per batch instead of once per agent. Large objects held by agents, such as neural network models, are pickled in every task, even in the tasks of other agents, unless they are excluded from pickling, e.g., by __getstate__. Read-only objects of this kind can be loaded once per worker process by the worker initializer (see _get_worker_initializer) instead. The agents should use such an object where the initializer keeps it, e.g., in a module variable, or put it back on themselves in __setstate__; assigning it in pams.agents.Agent.submit_orders() is warned about. These two ways are only for objects that the agents do not change: an object that an agent changes in pams.agents.Agent.submit_orders(), such as a model that it trains, must be listed in pams.agents.Agent.synced_attributes. A listed attribute must be pickled with the agent; if __getstate__ drops it, it is deleted from the agent on the main process. The listed attributes are also pickled back after every call of the agent’s pams.agents.Agent.submit_orders(). If you want to use parallelization, it is recommended to use pams.runners.MultiThreadAgentParallelRunner.

default_start_method: Optional[str] = None

start method of the worker processes used when simulation.startMethod is not set in the config. None means the default start method of the platform. Subclasses can override it, e.g., with "spawn" for libraries that are not fork-safe.

Type:

Optional[str]