Runner object that runs spark session
Reader object that reads data from data source
Writer object that writes pipelined data from operators
Message object that has one of messages object containing attributes related to controlling process pipelining
a list of BaseGenericOperator objects that processes data one by one.
Reader object that reads data from data source
Reader object that reads data from data source
Writer object that writes pipelined data from operators
Writer object that writes pipelined data from operators
a list of BaseGenericOperator objects that processes data one by one.
a list of BaseGenericOperator objects that processes data one by one.
Message object that has one of messages object containing attributes related to controlling process pipelining
Message object that has one of messages object containing attributes related to controlling process pipelining
Pipelines data from reader to writer via a series of operators.
Pipelines data from reader to writer via a series of operators.
Runner object that runs spark session
Runner object that runs spark session
:: ApplicationDeveloperApi ::
Controller that simply pipelines DataFrame from reader to writer via a series of operators. This controller can be run both on streaming engine and batch-style engine, and work as batch and streaming processing controller.
Data type class to be passes through data pipeline from reader to operators
Data type for the final data format
Message type class
Session type class for reader object
Session type class for writer object