Class

ksb.csle.component.pipe.stream.reader

FilePipeReader

Related Doc: package reader

Permalink

class FilePipeReader extends BasePipeReader[DataFrame, StreamPipeReaderInfo, SparkSession]

:: ApplicationDeveloperApi ::

Operator that reads dataframe from file source.

Linear Supertypes
BasePipeReader[DataFrame, StreamPipeReaderInfo, SparkSession], BaseGenericPipeOperator[Int, Int, DataFrame, StreamPipeReaderInfo, SparkSession], BaseGenericMutantOperator[StreamPipeReaderInfo, Int, (Int) ⇒ DataFrame], BaseDoer, Logging, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FilePipeReader
  2. BasePipeReader
  3. BaseGenericPipeOperator
  4. BaseGenericMutantOperator
  5. BaseDoer
  6. Logging
  7. Serializable
  8. Serializable
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FilePipeReader(o: StreamPipeReaderInfo, session: SparkSession)

    Permalink

    o

    Object that contains message FilePipeReaderInfo FilePipeReaderInfo contains attributes as follows:

    • filePath: List of data file path (required)
    • fileType: File type (optional)
    • maxFilesPerTrigger: The maximum number of new files to be considered in every trigger.
    • timeColName: Column name containing time information (required)
    • watermark: Time slot in seconds or minutest The event time column and the threshold on how late the data is expected to be in terms of event time.
    • field: Field information (repeated)
    • delimiter: delimiter, default value is ',' (optional)
    • header: existance of header (optional)

    FilePipeReaderInfo

    message FilePipeReaderInfo {
     required string filePath = 1;
     required FileType fileType = 2 [default = JSON];
     optional int32 maxFilesPerTrigger = 3;
     optional string timeColName = 5 [default = "timestamp"];
     optional string watermark = 6 [default = "1 seconds"];
     repeated FieldInfo field = 7;
     optional string delimiter = 8 [default = ","];
     optional bool header = 9 [default = false];
     enum FileType {
      CSV = 0;
      JSON = 1;
      PARQUET = 2;
      TEXT = 3;
     }
    }

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. def close: Unit

    Permalink
    Definition Classes
    FilePipeReader → BasePipeReader
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. val fileType: String

    Permalink
  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  14. val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  15. val maxFiles: Int

    Permalink
  16. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. val o: StreamPipeReaderInfo

    Permalink

    Object that contains message FilePipeReaderInfo FilePipeReaderInfo contains attributes as follows:

    Object that contains message FilePipeReaderInfo FilePipeReaderInfo contains attributes as follows:

    • filePath: List of data file path (required)
    • fileType: File type (optional)
    • maxFilesPerTrigger: The maximum number of new files to be considered in every trigger.
    • timeColName: Column name containing time information (required)
    • watermark: Time slot in seconds or minutest The event time column and the threshold on how late the data is expected to be in terms of event time.
    • field: Field information (repeated)
    • delimiter: delimiter, default value is ',' (optional)
    • header: existance of header (optional)

    FilePipeReaderInfo

    message FilePipeReaderInfo {
     required string filePath = 1;
     required FileType fileType = 2 [default = JSON];
     optional int32 maxFilesPerTrigger = 3;
     optional string timeColName = 5 [default = "timestamp"];
     optional string watermark = 6 [default = "1 seconds"];
     repeated FieldInfo field = 7;
     optional string delimiter = 8 [default = ","];
     optional bool header = 9 [default = false];
     enum FileType {
      CSV = 0;
      JSON = 1;
      PARQUET = 2;
      TEXT = 3;
     }
    }
  20. final def operate(in: Int): (Int) ⇒ DataFrame

    Permalink
    Definition Classes
    BasePipeReader → BaseGenericPipeOperator → BaseGenericMutantOperator
  21. val p: FilePipeReaderInfo

    Permalink
  22. def read(): DataFrame

    Permalink
    Definition Classes
    FilePipeReader → BasePipeReader
  23. val session: SparkSession

    Permalink
  24. def stop: Unit

    Permalink
    Definition Classes
    BasePipeReader → BaseGenericPipeOperator → BaseGenericMutantOperator
  25. val streamReader: DataStreamReader

    Permalink
  26. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  27. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  28. val userSchema: StructType

    Permalink
  29. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from BasePipeReader[DataFrame, StreamPipeReaderInfo, SparkSession]

Inherited from BaseGenericPipeOperator[Int, Int, DataFrame, StreamPipeReaderInfo, SparkSession]

Inherited from BaseGenericMutantOperator[StreamPipeReaderInfo, Int, (Int) ⇒ DataFrame]

Inherited from BaseDoer

Inherited from Logging

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped