Design Discussion: Incorporating multi-input networks

[vpratz]

For more complex settings, dealing with multi-modal inputs is an important requirement. This requires specifying multiple inputs, which is usually handled in a named (dictionaries) or unnamed (lists) way of passing the inputs. We have already included a few changes to allow passing dictionaries to summary networks (Adapter.group, Adapter.ungroup, handling of dictionary inputs in the new Standardization layer), but @LarsKue voiced concerns that the current approach leads to a cluttered/inconsistent user interface. Therefore, we want to open the discussion again to gather the requirements and needs, as well as ideas and directions for the implementation.

I see mainly the following discussion points, feel free to add more if you see other aspects:

1. Importance: Is this an edge case that only experienced users need, or do we want it to be a first-class feature just as single-input networks?
2. Which parts of the pipeline should handle the increased complexity?

Areas with models with multi-modal data include psychology (see e.g. [1]), and physical models, where the same simulator can produce data for different experimental settings, which can then be combined for more exact inference.

Some related discussion has already taken place in #503, but I think it would be good to restart the discussion here.

I will provide my current thoughts below, also tagging @LarsKue and @stefanradev93 for input.

[1] https://link.springer.com/article/10.1007/s42113-023-00167-4

[vpratz]

My current thoughts on the questions:

Re 1: I think it is a common enough setting that it would be good to ensure the normal workflows work, and you get a meaningful warning/error when you try to call functionalities with a dictionary that do not support it. Full support would be great if we find a way to integrate it consistently and without too large a cost regarding maintainability.

Re 2: For now I see two main options, both have their advantages and disadvantages.

a) We allow handling dictionaries in the adapter (and have to decide how far we take it).

Pro: The approximator does not have to care about the contents of summary_variables, and can expect to get the correct output from the summary network without worrying how the networks and its input look like, as long as they are compatible with each other.
Con: The adapter, which is currently quite clean as it (more or less) only operates on arrays, gets substantially more complex, as it has to handle nested inputs as well. We would have to see how much we can smooth this out with flatten-restructure wrappers around transforms.

b) We try to keep the adapter "pure" (no handling of nested inputs), and assemble the required structures further downstream in the pipeline. This could be either in the approximator, or in an additional dedicated class between adapter and approximator.

Pro: The adapter is simpler, adding/maintaining transforms is easier.
Con: Depends on the exact implementation.

• in the approximator: the approximator has to assemble inputs as well, which might not be the intuitive place for it, as there is already the adapter which does this
• @LarsKue proposed a special approximator for multi-modal inputs. I believe this is not viable as we can encounter them in any of the approximators we currently have, so we would have to maintain two variants of each
• an additional input handler class would add another element for the user to grasp, which is not desirable as we have quite a few moving parts already.

[paul-buerkner]

Can you provide a code example for how we would currently deal with it (to the degree that we already can)?

[stefanradev93]

I would not try to do it elsewhere except in the Adapter.

[vpratz]

@paul-buerkner You can find an example in this notebook.

For context: This works when the adapter is specified explicitly, but preparing the required summary_variables manually directly in the simulator and then trying to use the BasicWorkflow with the default adapter fails, as this would require handling the dictionary input in the default transforms of the adapter. This was the motivation for #503, but changing the behavior there leads to the question where in the adapter we then want/have to support dictionary inputs as well, to avoid inconsistencies in the user interface, and what the cost of this would be.

[paul-buerkner]

Thanks! I agree that we should handle it in the adapter. I am not super concerned about not having direct support in BasicWorkflow (at least immediately) since working with adapter and approximator directly does not create too much code overhead.