The DataPlaneManager (DPM) enqueues DataFlowRequest, which may originate from an ingress such as an HTTP endpoint. The queue implementation will be a bounded, in-memory data structure that provides support for backpressure when the system is overloaded. However, since the queue is in-memory, the client control plane submitting the request will be responsible for re-submitting a request in the event of failure.
The DPM implementation has a configurable number of workers that dequeue requests for processing (data transfer). This processing operation will complete asynchronously. When a result is returned, an entry for the process indicating success or failure is made using extensible storage. This allows the Control Plane to query the DPF for the status ( COMPLETED or ERROR) of a process. No other state is maintained by the DPF.
The DPM also contains transfer methods that take a DataSource and DataSink, respectively, These can be used to provide an ad-hoc transfer source or destination. For example, user code may dequeue a message to memory and send it to a destination using an InputStreamDataSource that wraps the in-memory message. Similarly, user code may wish to transfer data from a source to a provided OutputStreamDataSink.