dataloader

package
v0.50.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 2, 2025 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewCsvReader added in v0.16.0 

func NewCsvReader(r io.ReadCloser) (*csvReader, error)

NewCsvReader creates a csvReader from a given ReadCloser.

The interpretation of the bytes of the supplied reader is a little murky. If there is a UTF8, UTF16LE or UTF16BE BOM as the first bytes read, then the BOM is stripped and the remaining contents of the reader are treated as that encoding. If we are not in any of those marked encodings, then some of the bytes go uninterpreted until we get to the SQL layer. It is currently the case that newlines must be encoded as a '0xa' byte.

func NewStringPrefixReader added in v0.16.0 

func NewStringPrefixReader(prefix string, reader io.Reader) *stringPrefixReader

NewStringPrefixReader creates a new stringPrefixReader that first returns the data in |prefix| and then returns data from |reader|.

Types

type CsvDataLoader 

type CsvDataLoader struct {
	// contains filtered or unexported fields
}

CsvDataLoader is an implementation of DataLoader that reads data from chunks of CSV files and inserts them into a table.

func NewCsvDataLoader 

func NewCsvDataLoader(colNames []string, sch sql.Schema, delimiter string, header bool) (*CsvDataLoader, error)

NewCsvDataLoader creates a new DataLoader instance that will produce rows for the schema provided. |header| is true, the first line of the data will be treated as a header and ignored. If |delimiter| is not the empty string, it will be used as the delimiter separating value.

func (*CsvDataLoader) Children added in v0.17.0 

func (cdl *CsvDataLoader) Children() []sql.Node

func (*CsvDataLoader) Finish 

func (cdl *CsvDataLoader) Finish(ctx *sql.Context) (*LoadDataResults, error)

Finish implements the DataLoader interface

func (*CsvDataLoader) IsReadOnly added in v0.17.0 

func (cdl *CsvDataLoader) IsReadOnly() bool

func (*CsvDataLoader) Resolved added in v0.17.0 

func (cdl *CsvDataLoader) Resolved() bool

func (*CsvDataLoader) RowIter added in v0.17.0 

func (cdl *CsvDataLoader) RowIter(ctx *sql.Context, r sql.Row) (sql.RowIter, error)

func (*CsvDataLoader) Schema added in v0.17.0 

func (cdl *CsvDataLoader) Schema() sql.Schema

func (*CsvDataLoader) SetNextDataChunk added in v0.17.0 

func (cdl *CsvDataLoader) SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

func (*CsvDataLoader) String added in v0.17.0 

func (cdl *CsvDataLoader) String() string

func (*CsvDataLoader) WithChildren added in v0.17.0 

func (cdl *CsvDataLoader) WithChildren(children ...sql.Node) (sql.Node, error)

type DataLoader 

type DataLoader interface {
	sql.ExecSourceRel

	// SetNextDataChunk sets the next data chunk to be processed by the DataLoader.  Data records
	// are not guaranteed to start and end cleanly on chunk boundaries, so implementations must recognize incomplete
	// records and save them to prepend on the next processed chunk.
	SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

	// Finish finalizes the current load operation and cleans up any resources used. Implementations should check that
	// the last call to LoadChunk did not end with an incomplete record and return an error to the caller if so. The
	// returned LoadDataResults describe the load operation, including how many rows were inserted.
	Finish(ctx *sql.Context) (*LoadDataResults, error)
}

DataLoader allows callers to insert rows from multiple chunks into a table. Rows encoded in each chunk will not necessarily end cleanly on a chunk boundary, so DataLoader implementations must handle recognizing partial, or incomplete records, and saving that partial record until the next call to LoadChunk, so that it may be prefixed with the incomplete record.

type LoadDataResults 

type LoadDataResults struct {
	// RowsLoaded contains the total number of rows inserted during a load data operation.
	RowsLoaded int32
}

LoadDataResults contains the results of a load data operation, including the number of rows loaded.

type TabularDataLoader 

type TabularDataLoader struct {
	// contains filtered or unexported fields
}

TabularDataLoader tracks the state of a load data operation from a tabular data source.

func NewTabularDataLoader 

func NewTabularDataLoader(colNames []string, tableSch sql.Schema, delimiterChar, nullChar string, header bool) (*TabularDataLoader, error)

NewTabularDataLoader creates a new TabularDataLoader to insert into the specified |table| using the specified |delimiterChar| and |nullChar|. If |header| is true, the first line of the data will be treated as a header and ignored.

func (*TabularDataLoader) Children added in v0.17.0 

func (tdl *TabularDataLoader) Children() []sql.Node

func (*TabularDataLoader) Finish 

func (tdl *TabularDataLoader) Finish(ctx *sql.Context) (*LoadDataResults, error)

Finish completes the current load data operation and finalizes the data that has been inserted.

func (*TabularDataLoader) IsReadOnly added in v0.17.0 

func (tdl *TabularDataLoader) IsReadOnly() bool

func (*TabularDataLoader) Resolved added in v0.17.0 

func (tdl *TabularDataLoader) Resolved() bool

func (*TabularDataLoader) RowIter added in v0.17.0 

func (tdl *TabularDataLoader) RowIter(ctx *sql.Context, r sql.Row) (sql.RowIter, error)

func (*TabularDataLoader) Schema added in v0.17.0 

func (tdl *TabularDataLoader) Schema() sql.Schema

func (*TabularDataLoader) SetNextDataChunk added in v0.17.0 

func (tdl *TabularDataLoader) SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

func (*TabularDataLoader) String added in v0.17.0 

func (tdl *TabularDataLoader) String() string

func (*TabularDataLoader) WithChildren added in v0.17.0 

func (tdl *TabularDataLoader) WithChildren(children ...sql.Node) (sql.Node, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.