Package org.pentaho.di.trans.steps.file

Class BaseFileInputStep<M extends BaseFileInputMeta<?,?,?>,D extends BaseFileInputStepData>

java.lang.Object

org.pentaho.di.trans.step.BaseStep

org.pentaho.di.trans.steps.file.BaseFileInputStep<M,D>

All Implemented Interfaces:

org.pentaho.di.core.ExtensionDataInterface, HasLogChannelInterface, org.pentaho.di.core.logging.LoggingObjectInterface, org.pentaho.di.core.logging.LoggingObjectLifecycleInterface, org.pentaho.di.core.variables.VariableSpace, StepInterface, IBaseFileInputStepControl

Direct Known Subclasses:

TextFileInput

public abstract class BaseFileInputStep<M extends BaseFileInputMeta<?,?,?>,D extends BaseFileInputStepData>
extends BaseStep
implements IBaseFileInputStepControl

This class contains base functionality for file-based input steps.

Author:

Alexander Buloichik

Field Summary

Fields

Modifier and Type	Field	Description
protected D	data
protected M	meta

Fields inherited from class org.pentaho.di.trans.step.BaseStep

deadLockCounter, extensionDataMap, first, linesInput, linesOutput, linesRead, linesRejected, linesSkipped, linesUpdated, linesWritten, log, loggingObjectInUse, metaStore, repository, rowListeners, safeStopped, terminator, terminator_rows, variables

Constructor Summary

Constructors

Constructor	Description
BaseFileInputStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int copyNr, TransMeta transMeta, Trans trans)

Method Summary

Modifier and Type	Method	Description
boolean	checkFeedback(long lines)	Check feedback.
protected void	closeLastFile()	Close last opened file/
protected abstract IBaseFileInputReader	createReader(M meta, D data, org.apache.commons.vfs2.FileObject file)	Create reader for specific file.
void	dispose(StepMetaInterface smi, StepDataInterface sdi)	Dispose step.
boolean	failAfterBadFile(String errorMsg)
protected void	fillFileAdditionalFields(D data, org.apache.commons.vfs2.FileObject file)	Prepare file-dependent data for fill additional fields.
protected boolean	handleOpenFileException(Exception e)
protected abstract boolean	init()	Content-dependent initialization.
boolean	init(StepMetaInterface smi, StepDataInterface sdi)	Initialize step before execute.
protected boolean	openNextFile()	Open next VFS file for processing.
protected void	prepareToRowProcessing()	Prepare to process.
boolean	processRow(StepMetaInterface smi, StepDataInterface sdi)	Process next row.

Methods inherited from class org.pentaho.di.trans.step.BaseStep

addResultFile, addRowListener, addRowSetToInputRowSets, addRowSetToOutputRowSets, addStepListener, batchComplete, beforeStartProcessing, buildLog, canProcessOneRow, cleanup, clearInputRowSets, clearOutputRowSets, closeQuietly, copyVariablesFrom, decrementLinesRead, decrementLinesWritten, dispatch, environmentSubstitute, environmentSubstitute, environmentSubstitute, fieldSubstitute, findInputRowSet, findInputRowSet, findOutputRowSet, findOutputRowSet, getBooleanValueOfVariable, getClusterSize, getContainerObjectId, getCopy, getCurrentInputRowSetNr, getCurrentOutputRowSetNr, getDispatcher, getErrorRowMeta, getErrors, getExtensionDataMap, getFilename, getFirstInputRowSet, getInputRowMeta, getInputRowSets, getLinesInput, getLinesOutput, getLinesRead, getLinesRejected, getLinesSkipped, getLinesUpdated, getLinesWritten, getLogChannel, getLogChannelId, getLogFields, getLogLevel, getMetaStore, getNextClassNr, getObjectCopy, getObjectId, getObjectName, getObjectRevision, getObjectType, getOutputRowSets, getParent, getParentVariableSpace, getPartitionID, getPartitionTargets, getPreviewRowMeta, getProcessed, getRegistrationDate, getRemoteInputSteps, getRemoteOutputSteps, getRepartitioning, getRepository, getRepositoryDirectory, getResultFiles, getRow, getRowFrom, getRowHandler, getRowListeners, getRuntime, getServerSockets, getSlaveNr, getSocketRepository, getStatus, getStatusDescription, getStepDataInterface, getStepID, getStepListeners, getStepMeta, getStepMetaInterface, getStepname, getTrans, getTransMeta, getTypeId, getUniqueStepCountAcrossSlaves, getUniqueStepNrAcrossSlaves, getVariable, getVariable, handleGetRowFrom, handlePutRowTo, identifyErrorOutput, incrementLinesInput, incrementLinesOutput, incrementLinesRead, incrementLinesRejected, incrementLinesSkipped, incrementLinesUpdated, incrementLinesWritten, initBeforeStart, initializeVariablesFrom, injectVariables, isBasic, isDebug, isDetailed, isDistributed, isForcingSeparateLogging, isGatheringMetrics, isInitialising, isLoggingObjectInUse, isMapping, isPartitioned, isPaused, isRowLevel, isRunning, isSafeStopped, isStopped, isUsingThreadPriorityManagment, listVariables, logBasic, logBasic, logDebug, logDebug, logDetailed, logDetailed, logError, logError, logError, logMinimal, logMinimal, logRowlevel, logRowlevel, logSummary, markStart, markStop, openRemoteInputStepSocketsOnce, openRemoteOutputStepSocketsOnce, outputIsDone, pauseRunning, putError, putRow, putRowTo, removeRowListener, resumeRunning, rowsetInputSize, rowsetOutputSize, safeModeChecking, safeModeChecking, setCarteObjectId, setCopy, setCurrentInputRowSetNr, setCurrentOutputRowSetNr, setDistributed, setErrorRowMeta, setErrors, setForcingSeparateLogging, setGatheringMetrics, setInputRowMeta, setInputRowSets, setInternalVariables, setLinesInput, setLinesOutput, setLinesRead, setLinesRejected, setLinesSkipped, setLinesUpdated, setLinesWritten, setLoggingObjectInUse, setLogLevel, setMetaStore, setOutputDone, setOutputRowSets, setParentVariableSpace, setPartitioned, setPartitionID, setPartitionTargets, setPaused, setPaused, setPreviewRowMeta, setRepartitioning, setRepository, setRowHandler, setRunning, setSafeStopped, setServerSockets, setSocketRepository, setStepDataInterface, setStepListeners, setStepMeta, setStepMetaInterface, setStepname, setStopped, setTransMeta, setUsingThreadPriorityManagment, setVariable, shareVariablesWith, stopAll, stopRunning, stopRunning, swapFirstInputRowSetIfExists, toString, verifyInputDeadLock, waitUntilTransformationIsStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.pentaho.di.trans.steps.file.IBaseFileInputStepControl

getErrors, getLinesInput, getLinesWritten, incrementLinesInput, incrementLinesUpdated, putRow, setErrors, stopAll

Methods inherited from interface org.pentaho.di.core.logging.LoggingObjectLifecycleInterface

callAfterLog, callBeforeLog

Methods inherited from interface org.pentaho.di.trans.step.StepInterface

afterFinishProcessing, subStatuses

Field Detail

meta

protected M extends BaseFileInputMeta<?,?,?> meta

data

protected D extends BaseFileInputStepData data

Constructor Detail

BaseFileInputStep

public BaseFileInputStep(StepMeta stepMeta,
                         StepDataInterface stepDataInterface,
                         int copyNr,
                         TransMeta transMeta,
                         Trans trans)

Method Detail

init

protected abstract boolean init()

Content-dependent initialization.

createReader

protected abstract IBaseFileInputReader createReader(M meta,
                                                     D data,
                                                     org.apache.commons.vfs2.FileObject file)
                                              throws Exception

Create reader for specific file.

Throws:

Exception

init

public boolean init(StepMetaInterface smi,
                    StepDataInterface sdi)

Initialize step before execute.

Specified by:

init in interface StepInterface

Overrides:

init in class BaseStep

Parameters:

smi - The metadata to work with

sdi - The data to initialize

openNextFile

protected boolean openNextFile()

Open next VFS file for processing. This method will support different parallelization methods later.

handleOpenFileException

protected boolean handleOpenFileException(Exception e)

processRow

public boolean processRow(StepMetaInterface smi,
                          StepDataInterface sdi)
                   throws org.pentaho.di.core.exception.KettleException

Process next row. This methods opens next file automatically.

Specified by:

processRow in interface StepInterface

Overrides:

processRow in class BaseStep

Parameters:

smi - The steps metadata to work with

sdi - The steps temporary working data to work with (database connections, result sets, caches, temporary variables, etc.)

Returns:

false if no more rows can be processed or an error occurred.

Throws:

org.pentaho.di.core.exception.KettleException

prepareToRowProcessing

protected void prepareToRowProcessing()
                               throws org.pentaho.di.core.exception.KettleException

Prepare to process. Executed only first time row processing. It can't be possible to prepare to process in the init() phrase, because files can be in fields from previous step.

Throws:

org.pentaho.di.core.exception.KettleException

checkFeedback

public boolean checkFeedback(long lines)

Description copied from class: BaseStep

Check feedback.

Specified by:

checkFeedback in interface IBaseFileInputStepControl

Overrides:

checkFeedback in class BaseStep

Parameters:

lines - the lines

Returns:

true, if successful

closeLastFile

protected void closeLastFile()

Close last opened file/

dispose

public void dispose(StepMetaInterface smi,
                    StepDataInterface sdi)

Dispose step.

Specified by:

dispose in interface StepInterface

Overrides:

dispose in class BaseStep

Parameters:

smi - The metadata to work with

sdi - The data to dispose of

failAfterBadFile

public boolean failAfterBadFile(String errorMsg)

Specified by:

failAfterBadFile in interface IBaseFileInputStepControl

Parameters:

errorMsg - Message to send to rejected row if enabled

Returns:

If should stop processing after having problems with a file

fillFileAdditionalFields

protected void fillFileAdditionalFields(D data,
                                        org.apache.commons.vfs2.FileObject file)
                                 throws org.apache.commons.vfs2.FileSystemException

Prepare file-dependent data for fill additional fields.

Throws:

org.apache.commons.vfs2.FileSystemException