org.pentaho.di.trans.step
Class BaseStep

java.lang.Object
  extended by org.pentaho.di.trans.step.BaseStep
All Implemented Interfaces:
HasLogChannelInterface, LoggingObjectInterface, VariableSpace, StepInterface
Direct Known Subclasses:
Abort, AbstractStep, AccessInput, AccessOutput, AddSequence, AddXML, AggregateRows, AnalyticQuery, Append, AutoDoc, BlockingStep, BlockUntilStepsFinish, Calculator, ChangeFileEncoding, CheckSum, CloneRow, ClosureGenerator, ColumnExists, CombinationLookup, Constant, CreditCardValidator, CsvInput, CubeInput, CubeOutput, DatabaseJoin, DatabaseLookup, DataGrid, DBProc, Delay, Delete, Denormaliser, DetectEmptyStream, DetectLastRow, DimensionLookup, DummyTrans, DynamicSQLRow, Edi2Xml, ElasticSearchBulk, ExcelInput, ExcelOutput, ExcelWriterStep, ExecProcess, ExecSQL, ExecSQLRow, FieldsChangeSequence, FieldSplitter, FileExists, FileLocked, FilesFromResult, FilesToResult, FilterRows, FixedInput, Flattener, Formula, FuzzyMatch, GaInputStep, GetFileNames, GetFilesRowsCount, GetPreviousRowField, GetRepositoryNames, GetSlaveSequence, GetSubFolders, GetTableNames, GetVariable, GetXMLData, GPBulkLoader, GroupBy, HTTP, HTTPPOST, IfNull, InfobrightLoader, IngresVectorwiseLoader, Injector, InsertUpdate, Janino, JavaFilter, JoinRows, JsonInput, JsonOutput, LDAPInput, LDAPOutput, LDIFInput, LoadFileInput, LucidDBBulkLoader, LucidDBStreamingLoader, Mail, MailInput, MailValidator, Mapping, MappingInput, MappingOutput, MemoryGroupBy, MergeJoin, MergeRows, MetaInject, MondrianInput, MonetDBBulkLoader, MultiMergeJoin, MySQLBulkLoader, Normaliser, NullIf, NumberRange, OlapInput, OraBulkLoader, ParGzipCsvInput, PentahoReportingOutput, PGBulkLoader, PrioritizeStreams, ProcessFiles, PropertyInput, PropertyOutput, RandomCCNumberGenerator, RandomValue, RegexEval, ReplaceString, ReservoirSampling, Rest, RowGenerator, RowsFromResult, RowsToResult, RssInput, RssOutput, Rules, SalesforceDelete, SalesforceInput, SalesforceInsert, SalesforceUpdate, SalesforceUpsert, SampleRows, SapInput, SasInput, Script, ScriptValuesMod, SecretKeyGenerator, SelectValues, SetValueConstant, SetValueField, SetVariable, SingleThreader, SocketReader, SocketWriter, SortedMerge, SortRows, SplitFieldToRows, SQLFileOutput, SSH, StepMetastructure, StepsMetrics, StreamLookup, StringCut, StringOperations, SwitchCase, SymmetricCryptoTrans, SynchronizeAfterMerge, SyslogMessage, SystemData, TableExists, TableInput, TableOutput, TextFileInput, TextFileOutput, UniqueRows, UniqueRowsByHashSet, UnivariateStats, Update, UserDefinedJavaClass, Validator, ValueMapper, WebService, WebServiceAvailable, WriteToLog, XBaseInput, XMLInput, XMLInputSax, XMLInputStream, XMLJoin, XMLOutput, XsdValidator, Xslt, YamlInput

public class BaseStep
extends Object
implements VariableSpace, StepInterface, LoggingObjectInterface

This class can be extended for the actual row processing of the implemented step.

The implementing class can rely mostly on the base class, and has only three important methods it implements itself. The three methods implement the step lifecycle during transformation execution: initialization, row processing, and clean-up.


Field Summary
 boolean first
          if true then the row being processed is the first row
 long linesInput
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesOutput
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesRead
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesRejected
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesSkipped
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesUpdated
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 long linesWritten
          Deprecated. please use the supplied getters, setters and increment/decrement methods
 boolean terminator
           
 List<Object[]> terminator_rows
           
 
Constructor Summary
BaseStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int copyNr, TransMeta transMeta, Trans trans)
          This is the base step that forms that basis for all steps.
 
Method Summary
 void addResultFile(ResultFile resultFile)
          Adds the result file.
 void addRowListener(RowListener rowListener)
          Add a rowlistener to the step allowing you to inspect (or manipulate, be careful) the rows coming in or exiting the step.
 void addStepListener(StepListener stepListener)
          Attach a step listener to be notified when a step arrives in a certain state.
 void batchComplete()
          Calling this method will alert the step that we finished passing a batch of records to the step.
 RowMetaAndData buildLog(String sname, int copynr, long lines_read, long lines_written, long lines_updated, long lines_skipped, long errors, Date start_date, Date end_date)
          Builds the log.
 boolean canProcessOneRow()
          This method checks if the step is capable of processing at least one row.
 void cleanup()
          Call this method typically, after ALL the slave transformations in a clustered run have finished.
static void closeQuietly(Closeable cl)
          Close quietly.
 void copyVariablesFrom(VariableSpace space)
          Copy the variables from another space, without initializing with the defaults.
 long decrementLinesRead()
          Decrements the number of lines read from previous steps by one
 long decrementLinesWritten()
          Decrements the number of lines written to next steps by one
 void dispatch()
          This method finds the surrounding steps and rowsets for this base step.
 void dispose(StepMetaInterface smi, StepDataInterface sdi)
          Dispose of this step: close files, empty logs, etc.
 String environmentSubstitute(String aString)
          Substitute the string using the current variable space.
 String[] environmentSubstitute(String[] aString)
          Replaces environment variables in an array of strings.
 RowSet findInputRowSet(String sourceStep)
          Find input row set.
 RowSet findInputRowSet(String from, int fromcopy, String to, int tocopy)
          Find input row set.
 RowSet findOutputRowSet(String targetStep)
          Find output row set.
 RowSet findOutputRowSet(String from, int fromcopy, String to, int tocopy)
          Find an output rowset in a running transformation.
 boolean getBooleanValueOfVariable(String variableName, boolean defaultValue)
          This method returns a boolean for the new variable check boxes.
 int getClusterSize()
          Returns the cluster size.
 String getContainerObjectId()
          Returns the container object ID.
 int getCopy()
           
 Trans getDispatcher()
          Gets the dispatcher.
 RowMetaInterface getErrorRowMeta()
           
 long getErrors()
          Get the number of errors
 String getFilename()
          Gets the filename.
 RowMetaInterface getInputRowMeta()
           
 List<RowSet> getInputRowSets()
           
 long getLinesInput()
           
 long getLinesOutput()
           
 long getLinesRead()
           
 long getLinesRejected()
           
 long getLinesSkipped()
           
 long getLinesUpdated()
           
 long getLinesWritten()
           
 LogChannelInterface getLogChannel()
           
 String getLogChannelId()
          Gets the log channel id.
static RowMetaInterface getLogFields(String comm)
          Gets the log fields.
 LogLevel getLogLevel()
          Gets the logging level of the log channel of this logging object.
 int getNextClassNr()
          Gets the next class nr.
 String getObjectCopy()
          Gets a string identifying a copy in a series of steps.
 ObjectId getObjectId()
          Gets the object id in a repository.
 String getObjectName()
          Gets the object name.
 ObjectRevision getObjectRevision()
          Gets the objects revision in a repository.
 LoggingObjectType getObjectType()
          Gets the object type.
 List<RowSet> getOutputRowSets()
           
 LoggingObjectInterface getParent()
          Gets the parent.
 VariableSpace getParentVariableSpace()
          Get the parent of the variable space.
 String getPartitionID()
           
 Map<String,BlockingRowSet> getPartitionTargets()
           
 RowMetaInterface getPreviewRowMeta()
           
 long getProcessed()
           
 Date getRegistrationDate()
          Returns the registration date
 List<RemoteStep> getRemoteInputSteps()
          Gets the remote input steps.
 List<RemoteStep> getRemoteOutputSteps()
          Gets the remote output steps.
 int getRepartitioning()
           
 RepositoryDirectory getRepositoryDirectory()
          Gets the repository directory.
 Map<String,ResultFile> getResultFiles()
           
 Object[] getRow()
          In case of getRow, we receive data from previous steps through the input rowset.
 Object[] getRowFrom(RowSet rowSet)
          Gets the row from.
 List<RowListener> getRowListeners()
           
 long getRuntime()
           
 List<ServerSocket> getServerSockets()
          Returns the serverSockets.
 int getSlaveNr()
          Returns the unique slave number in the cluster.
 SocketRepository getSocketRepository()
          Retutns the socket repository.
 BaseStepData.StepExecutionStatus getStatus()
           
 String getStatusDescription()
          Gets the status description.
 StepDataInterface getStepDataInterface()
           
 String getStepID()
           
 List<StepListener> getStepListeners()
          Returns the step listeners.
 StepMeta getStepMeta()
           
 StepMetaInterface getStepMetaInterface()
           
 String getStepname()
          Get the name of the step.
 Trans getTrans()
           
 TransMeta getTransMeta()
           
 String getTypeId()
          Returns the step ID via the getStepID() method call.
 int getUniqueStepCountAcrossSlaves()
          Returns the number of unique steps across all slave servers.
 int getUniqueStepNrAcrossSlaves()
          Returns a unique step number across all slave servers: slaveNr * nrCopies + copyNr.
 String getVariable(String variableName)
          Get the value of a variable.
 String getVariable(String variableName, String defaultValue)
          Get the value of a variable with a default in case the variable is not found.
 void identifyErrorOutput()
          To be used to flag an error output channel of a step prior to execution for performance reasons.
 long incrementLinesInput()
          Increments the number of lines read from an input source: database, file, socket, etc.
 long incrementLinesOutput()
          Increments the number of lines written to an output target: database, file, socket, etc.
 long incrementLinesRead()
          Increments the number of lines read from previous steps by one
 long incrementLinesRejected()
          Increments the number of lines rejected to an error handling step
 long incrementLinesSkipped()
          Increments the number of lines skipped
 long incrementLinesUpdated()
          Increments the number of lines updated in an output target: database, file, socket, etc.
 long incrementLinesWritten()
          Increments the number of lines written to next steps by one
 boolean init(StepMetaInterface smi, StepDataInterface sdi)
          Initialize and do work where other steps need to wait for...
 void initBeforeStart()
          This method is executed by Trans right before the threads start and right after initialization.
 void initializeVariablesFrom(VariableSpace parent)
          Initialize variable space using the defaults, copy over the variables from the parent (using copyVariablesFrom()), after this the "injected" variables should be inserted (injectVariables()).
 void injectVariables(Map<String,String> prop)
          Inject variables.
 boolean isBasic()
          Checks if is basic.
 boolean isDebug()
          Checks if is debug.
 boolean isDetailed()
          Checks if is detailed.
 boolean isDistributed()
           
 boolean isInitialising()
          Checks if is initialising.
 boolean isMapping()
           
 boolean isPartitioned()
           
 boolean isPaused()
           
 boolean isRowLevel()
          Checks if is row level.
 boolean isRunning()
           
 boolean isStopped()
           
 boolean isUsingThreadPriorityManagment()
          Retusn true if we are actively managing priorities of step threads.
 String[] listVariables()
          List the variables (not the values) that are currently in the variable space.
 void logBasic(String message)
          Log basic.
 void logBasic(String message, Object... arguments)
          Log basic.
 void logDebug(String message)
          Log debug.
 void logDebug(String message, Object... arguments)
          Log debug.
 void logDetailed(String message)
          Log detailed.
 void logDetailed(String message, Object... arguments)
          Log detailed.
 void logError(String message)
          Log error.
 void logError(String message, Object... arguments)
          Log error.
 void logError(String message, Throwable e)
          Log error.
 void logMinimal(String message)
          Log minimal.
 void logMinimal(String message, Object... arguments)
          Log minimal.
 void logRowlevel(String message)
          Log rowlevel.
 void logRowlevel(String message, Object... arguments)
          Log rowlevel.
 void logSummary()
          Log summary.
 void markStart()
          Mark the start time of the step.
 void markStop()
          Mark the end time of the step.
 boolean outputIsDone()
          Output is done.
 void pauseRunning()
          Pause a running step
 boolean processRow(StepMetaInterface smi, StepDataInterface sdi)
          Perform the equivalent of processing one row.
 void putError(RowMetaInterface rowMeta, Object[] row, long nrErrors, String errorDescriptions, String fieldNames, String errorCodes)
          Put error.
 void putRow(RowMetaInterface rowMeta, Object[] row)
          putRow is used to copy a row, to the alternate rowset(s) This should get priority over everything else! (synchronized) If distribute is true, a row is copied only once to the output rowsets, otherwise copies are sent to each rowset!
 void putRowTo(RowMetaInterface rowMeta, Object[] row, RowSet rowSet)
          putRowTo is used to put a row in a certain specific RowSet.
 void removeRowListener(RowListener rowListener)
          Remove a rowlistener from this step.
 void resumeRunning()
          Resume a running step
 int rowsetInputSize()
           
 int rowsetOutputSize()
           
static void safeModeChecking(RowMetaInterface referenceRowMeta, RowMetaInterface rowMeta)
          Safe mode checking.
 void setCarteObjectId(String containerObjectId)
          Sets the container object ID.
 void setCopy(int cop)
          Sets the copy.
 void setDistributed(boolean distributed)
           
 void setErrorRowMeta(RowMetaInterface errorRowMeta)
           
 void setErrors(long e)
          Sets the number of errors
 void setInputRowMeta(RowMetaInterface rowMeta)
           
 void setInputRowSets(List<RowSet> inputRowSets)
           
 void setInternalVariables()
          Sets the internal variables.
 void setLinesInput(long newLinesInputValue)
           
 void setLinesOutput(long newLinesOutputValue)
           
 void setLinesRead(long newLinesReadValue)
           
 void setLinesRejected(long newLinesRejectedValue)
           
 void setLinesSkipped(long newLinesSkippedValue)
           
 void setLinesUpdated(long newLinesUpdatedValue)
           
 void setLinesWritten(long newLinesWrittenValue)
           
 void setLogLevel(LogLevel logLevel)
          Sets the log level.
 void setOutputDone()
          Signal output done to destination steps
 void setOutputRowSets(List<RowSet> outputRowSets)
           
 void setParentVariableSpace(VariableSpace parent)
          Set the parent variable space
 void setPartitioned(boolean partitioned)
           
 void setPartitionID(String partitionID)
           
 void setPartitionTargets(Map<String,BlockingRowSet> partitionTargets)
           
 void setPaused(AtomicBoolean paused)
          Sets the paused.
 void setPaused(boolean paused)
          Sets the paused.
 void setPreviewRowMeta(RowMetaInterface previewRowMeta)
           
 void setRepartitioning(int repartitioning)
           
 void setRunning(boolean running)
          Flag the step as running or not
 void setServerSockets(List<ServerSocket> serverSockets)
           
 void setSocketRepository(SocketRepository socketRepository)
          Sets the socket repository.
 void setStepDataInterface(StepDataInterface stepDataInterface)
           
 void setStepListeners(List<StepListener> stepListeners)
          Sets the step listeners.
 void setStepMeta(StepMeta stepMeta)
           
 void setStepMetaInterface(StepMetaInterface stepMetaInterface)
           
 void setStepname(String stepname)
          Sets the stepname.
 void setStopped(boolean stopped)
           
 void setTransMeta(TransMeta transMeta)
           
 void setUsingThreadPriorityManagment(boolean usingThreadPriorityManagment)
          Set to true to actively manage priorities of step threads.
 void setVariable(String variableName, String variableValue)
          Sets a variable in the Kettle Variables list.
 void shareVariablesWith(VariableSpace space)
          Share a variable space from another variable space.
 void stopAll()
          Flags all rowsets as stopped/completed/finished.
 void stopRunning()
          Deprecated.  
 void stopRunning(StepMetaInterface stepMetaInterface, StepDataInterface stepDataInterface)
          Perform actions to stop a running step.
 String toString()
           
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

linesRead

public long linesRead
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of lines read from previous step(s)


linesWritten

public long linesWritten
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of lines written to next step(s)


linesInput

public long linesInput
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of lines read from file or database


linesOutput

public long linesOutput
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of lines written to file or database


linesUpdated

public long linesUpdated
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of updates in a database table or file


linesSkipped

public long linesSkipped
Deprecated. please use the supplied getters, setters and increment/decrement methods
nr of lines skipped


linesRejected

public long linesRejected
Deprecated. please use the supplied getters, setters and increment/decrement methods
total sleep time in ns caused by an empty input buffer (previous step is slow)


first

public boolean first
if true then the row being processed is the first row


terminator

public boolean terminator

terminator_rows

public List<Object[]> terminator_rows
Constructor Detail

BaseStep

public BaseStep(StepMeta stepMeta,
                StepDataInterface stepDataInterface,
                int copyNr,
                TransMeta transMeta,
                Trans trans)
This is the base step that forms that basis for all steps. You can derive from this class to implement your own steps.

Parameters:
stepMeta - The StepMeta object to run.
stepDataInterface - the data object to store temporary data, database connections, caches, result sets, hashtables etc.
copyNr - The copynumber for this step.
transMeta - The TransInfo of which the step stepMeta is part of.
trans - The (running) transformation to obtain information shared among the steps.
Method Detail

init

public boolean init(StepMetaInterface smi,
                    StepDataInterface sdi)
Description copied from interface: StepInterface
Initialize and do work where other steps need to wait for...

Specified by:
init in interface StepInterface
Parameters:
smi - The metadata to work with
sdi - The data to initialize

dispose

public void dispose(StepMetaInterface smi,
                    StepDataInterface sdi)
Description copied from interface: StepInterface
Dispose of this step: close files, empty logs, etc.

Specified by:
dispose in interface StepInterface
Parameters:
smi - The metadata to work with
sdi - The data to dispose of

cleanup

public void cleanup()
Description copied from interface: StepInterface
Call this method typically, after ALL the slave transformations in a clustered run have finished.

Specified by:
cleanup in interface StepInterface

getProcessed

public long getProcessed()
Specified by:
getProcessed in interface StepInterface
Returns:
The number of "processed" lines of a step. Well, a representable metric for that anyway.

setCopy

public void setCopy(int cop)
Sets the copy.

Parameters:
cop - the new copy

getCopy

public int getCopy()
Specified by:
getCopy in interface StepInterface
Returns:
The steps copy number (default 0)

getErrors

public long getErrors()
Description copied from interface: StepInterface
Get the number of errors

Specified by:
getErrors in interface StepInterface
Returns:
the number of errors

setErrors

public void setErrors(long e)
Description copied from interface: StepInterface
Sets the number of errors

Specified by:
setErrors in interface StepInterface
Parameters:
e - the number of errors to set

getLinesRead

public long getLinesRead()
Specified by:
getLinesRead in interface StepInterface
Returns:
Returns the number of lines read from previous steps

incrementLinesRead

public long incrementLinesRead()
Increments the number of lines read from previous steps by one

Returns:
Returns the new value

decrementLinesRead

public long decrementLinesRead()
Decrements the number of lines read from previous steps by one

Returns:
Returns the new value

setLinesRead

public void setLinesRead(long newLinesReadValue)
Parameters:
newLinesReadValue - the new number of lines read from previous steps

getLinesInput

public long getLinesInput()
Specified by:
getLinesInput in interface StepInterface
Returns:
Returns the number of lines read from an input source: database, file, socket, etc.

incrementLinesInput

public long incrementLinesInput()
Increments the number of lines read from an input source: database, file, socket, etc.

Returns:
the new incremented value

setLinesInput

public void setLinesInput(long newLinesInputValue)
Parameters:
newLinesInputValue - the new number of lines read from an input source: database, file, socket, etc.

getLinesOutput

public long getLinesOutput()
Specified by:
getLinesOutput in interface StepInterface
Returns:
Returns the number of lines written to an output target: database, file, socket, etc.

incrementLinesOutput

public long incrementLinesOutput()
Increments the number of lines written to an output target: database, file, socket, etc.

Returns:
the new incremented value

setLinesOutput

public void setLinesOutput(long newLinesOutputValue)
Parameters:
newLinesOutputValue - the new number of lines written to an output target: database, file, socket, etc.

getLinesWritten

public long getLinesWritten()
Specified by:
getLinesWritten in interface StepInterface
Returns:
Returns the linesWritten.

incrementLinesWritten

public long incrementLinesWritten()
Increments the number of lines written to next steps by one

Returns:
Returns the new value

decrementLinesWritten

public long decrementLinesWritten()
Decrements the number of lines written to next steps by one

Returns:
Returns the new value

setLinesWritten

public void setLinesWritten(long newLinesWrittenValue)
Parameters:
newLinesWrittenValue - the new number of lines written to next steps

getLinesUpdated

public long getLinesUpdated()
Specified by:
getLinesUpdated in interface StepInterface
Returns:
Returns the number of lines updated in an output target: database, file, socket, etc.

incrementLinesUpdated

public long incrementLinesUpdated()
Increments the number of lines updated in an output target: database, file, socket, etc.

Returns:
the new incremented value

setLinesUpdated

public void setLinesUpdated(long newLinesUpdatedValue)
Parameters:
newLinesOutputValue - the new number of lines updated in an output target: database, file, socket, etc.

getLinesRejected

public long getLinesRejected()
Specified by:
getLinesRejected in interface StepInterface
Returns:
the number of lines rejected to an error handling step

incrementLinesRejected

public long incrementLinesRejected()
Increments the number of lines rejected to an error handling step

Returns:
the new incremented value

setLinesRejected

public void setLinesRejected(long newLinesRejectedValue)
Specified by:
setLinesRejected in interface StepInterface
Parameters:
newLinesRejectedValue - lines number of lines rejected to an error handling step

getLinesSkipped

public long getLinesSkipped()
Returns:
the number of lines skipped

incrementLinesSkipped

public long incrementLinesSkipped()
Increments the number of lines skipped

Returns:
the new incremented value

setLinesSkipped

public void setLinesSkipped(long newLinesSkippedValue)
Parameters:
newLinesSkippedValue - lines number of lines skipped

getStepname

public String getStepname()
Description copied from interface: StepInterface
Get the name of the step.

Specified by:
getStepname in interface StepInterface
Returns:
the name of the step

setStepname

public void setStepname(String stepname)
Sets the stepname.

Parameters:
stepname - the new stepname

getDispatcher

public Trans getDispatcher()
Gets the dispatcher.

Returns:
the dispatcher

getStatusDescription

public String getStatusDescription()
Gets the status description.

Returns:
the status description

getStepMetaInterface

public StepMetaInterface getStepMetaInterface()
Returns:
Returns the stepMetaInterface.

setStepMetaInterface

public void setStepMetaInterface(StepMetaInterface stepMetaInterface)
Parameters:
stepMetaInterface - The stepMetaInterface to set.

getStepDataInterface

public StepDataInterface getStepDataInterface()
Returns:
Returns the stepDataInterface.

setStepDataInterface

public void setStepDataInterface(StepDataInterface stepDataInterface)
Parameters:
stepDataInterface - The stepDataInterface to set.

getStepMeta

public StepMeta getStepMeta()
Specified by:
getStepMeta in interface StepInterface
Returns:
Returns the stepMeta.

setStepMeta

public void setStepMeta(StepMeta stepMeta)
Parameters:
stepMeta - The stepMeta to set.

getTransMeta

public TransMeta getTransMeta()
Returns:
Returns the transMeta.

setTransMeta

public void setTransMeta(TransMeta transMeta)
Parameters:
transMeta - The transMeta to set.

getTrans

public Trans getTrans()
Specified by:
getTrans in interface StepInterface
Returns:
Returns the trans.

putRow

public void putRow(RowMetaInterface rowMeta,
                   Object[] row)
            throws KettleStepException
putRow is used to copy a row, to the alternate rowset(s) This should get priority over everything else! (synchronized) If distribute is true, a row is copied only once to the output rowsets, otherwise copies are sent to each rowset!

Specified by:
putRow in interface StepInterface
Parameters:
row - The row to put to the destination rowset(s).
rowMeta - The row to send to the destinations steps
Throws:
KettleStepException

putRowTo

public void putRowTo(RowMetaInterface rowMeta,
                     Object[] row,
                     RowSet rowSet)
              throws KettleStepException
putRowTo is used to put a row in a certain specific RowSet.

Parameters:
rowMeta - The row meta-data to put to the destination RowSet.
row - the data to put in the RowSet
rowSet - the RoWset to put the row into.
Throws:
KettleStepException - In case something unexpected goes wrong

putError

public void putError(RowMetaInterface rowMeta,
                     Object[] row,
                     long nrErrors,
                     String errorDescriptions,
                     String fieldNames,
                     String errorCodes)
              throws KettleStepException
Put error.

Parameters:
rowMeta - the row meta
row - the row
nrErrors - the nr errors
errorDescriptions - the error descriptions
fieldNames - the field names
errorCodes - the error codes
Throws:
KettleStepException - the kettle step exception

getRow

public Object[] getRow()
                throws KettleException
In case of getRow, we receive data from previous steps through the input rowset. In case we split the stream, we have to copy the data to the alternate splits: rowsets 1 through n.

Specified by:
getRow in interface StepInterface
Returns:
a row from the source step(s).
Throws:
KettleException

identifyErrorOutput

public void identifyErrorOutput()
Description copied from interface: StepInterface
To be used to flag an error output channel of a step prior to execution for performance reasons.

Specified by:
identifyErrorOutput in interface StepInterface

safeModeChecking

public static void safeModeChecking(RowMetaInterface referenceRowMeta,
                                    RowMetaInterface rowMeta)
                             throws KettleRowException
Safe mode checking.

Parameters:
referenceRowMeta - the reference row meta
rowMeta - the row meta
Throws:
KettleRowException - the kettle row exception

getRowFrom

public Object[] getRowFrom(RowSet rowSet)
                    throws KettleStepException
Gets the row from.

Parameters:
rowSet - the row set
Returns:
the row from
Throws:
KettleStepException - the kettle step exception

findInputRowSet

public RowSet findInputRowSet(String sourceStep)
                       throws KettleStepException
Find input row set.

Parameters:
sourceStep - the source step
Returns:
the row set
Throws:
KettleStepException - the kettle step exception

findInputRowSet

public RowSet findInputRowSet(String from,
                              int fromcopy,
                              String to,
                              int tocopy)
Find input row set.

Parameters:
from - the from
fromcopy - the fromcopy
to - the to
tocopy - the tocopy
Returns:
the row set

findOutputRowSet

public RowSet findOutputRowSet(String targetStep)
                        throws KettleStepException
Find output row set.

Parameters:
targetStep - the target step
Returns:
the row set
Throws:
KettleStepException - the kettle step exception

findOutputRowSet

public RowSet findOutputRowSet(String from,
                               int fromcopy,
                               String to,
                               int tocopy)
Find an output rowset in a running transformation. It will also look at the "to" step to see if this is a mapping. If it is, it will find the appropriate rowset in that transformation.

Parameters:
from -
fromcopy -
to -
tocopy -
Returns:
The rowset or null if none is found.

setOutputDone

public void setOutputDone()
Description copied from interface: StepInterface
Signal output done to destination steps

Specified by:
setOutputDone in interface StepInterface

dispatch

public void dispatch()
This method finds the surrounding steps and rowsets for this base step. This steps keeps it's own list of rowsets (etc.) to prevent it from having to search every time.


isBasic

public boolean isBasic()
Checks if is basic.

Returns:
true, if is basic

isDetailed

public boolean isDetailed()
Checks if is detailed.

Returns:
true, if is detailed

isDebug

public boolean isDebug()
Checks if is debug.

Returns:
true, if is debug

isRowLevel

public boolean isRowLevel()
Checks if is row level.

Returns:
true, if is row level

logMinimal

public void logMinimal(String message)
Log minimal.

Parameters:
message - the message

logMinimal

public void logMinimal(String message,
                       Object... arguments)
Log minimal.

Parameters:
message - the message
arguments - the arguments

logBasic

public void logBasic(String message)
Log basic.

Parameters:
message - the message

logBasic

public void logBasic(String message,
                     Object... arguments)
Log basic.

Parameters:
message - the message
arguments - the arguments

logDetailed

public void logDetailed(String message)
Log detailed.

Parameters:
message - the message

logDetailed

public void logDetailed(String message,
                        Object... arguments)
Log detailed.

Parameters:
message - the message
arguments - the arguments

logDebug

public void logDebug(String message)
Log debug.

Parameters:
message - the message

logDebug

public void logDebug(String message,
                     Object... arguments)
Log debug.

Parameters:
message - the message
arguments - the arguments

logRowlevel

public void logRowlevel(String message)
Log rowlevel.

Parameters:
message - the message

logRowlevel

public void logRowlevel(String message,
                        Object... arguments)
Log rowlevel.

Parameters:
message - the message
arguments - the arguments

logError

public void logError(String message)
Log error.

Parameters:
message - the message

logError

public void logError(String message,
                     Throwable e)
Log error.

Parameters:
message - the message
e - the e

logError

public void logError(String message,
                     Object... arguments)
Log error.

Parameters:
message - the message
arguments - the arguments

getNextClassNr

public int getNextClassNr()
Gets the next class nr.

Returns:
the next class nr

outputIsDone

public boolean outputIsDone()
Output is done.

Returns:
true, if successful

stopAll

public void stopAll()
Description copied from interface: StepInterface
Flags all rowsets as stopped/completed/finished.

Specified by:
stopAll in interface StepInterface

isStopped

public boolean isStopped()
Specified by:
isStopped in interface StepInterface
Returns:
True if the step is marked as stopped. Execution should stop immediate.

isRunning

public boolean isRunning()
Specified by:
isRunning in interface StepInterface
Returns:
true if the step is running after having been initialized

isPaused

public boolean isPaused()
Specified by:
isPaused in interface StepInterface
Returns:
True if the step is paused

setStopped

public void setStopped(boolean stopped)
Specified by:
setStopped in interface StepInterface
Parameters:
stopped - true if the step needs to be stopped

setRunning

public void setRunning(boolean running)
Description copied from interface: StepInterface
Flag the step as running or not

Specified by:
setRunning in interface StepInterface
Parameters:
running - the running flag to set

pauseRunning

public void pauseRunning()
Description copied from interface: StepInterface
Pause a running step

Specified by:
pauseRunning in interface StepInterface

resumeRunning

public void resumeRunning()
Description copied from interface: StepInterface
Resume a running step

Specified by:
resumeRunning in interface StepInterface

setPaused

public void setPaused(boolean paused)
Sets the paused.

Parameters:
paused - the new paused

setPaused

public void setPaused(AtomicBoolean paused)
Sets the paused.

Parameters:
paused - the new paused

isInitialising

public boolean isInitialising()
Checks if is initialising.

Returns:
true, if is initialising

markStart

public void markStart()
Description copied from interface: StepInterface
Mark the start time of the step.

Specified by:
markStart in interface StepInterface

setInternalVariables

public void setInternalVariables()
Sets the internal variables.


markStop

public void markStop()
Description copied from interface: StepInterface
Mark the end time of the step.

Specified by:
markStop in interface StepInterface

getRuntime

public long getRuntime()
Specified by:
getRuntime in interface StepInterface
Returns:
The number of ms that this step has been running

buildLog

public RowMetaAndData buildLog(String sname,
                               int copynr,
                               long lines_read,
                               long lines_written,
                               long lines_updated,
                               long lines_skipped,
                               long errors,
                               Date start_date,
                               Date end_date)
Builds the log.

Parameters:
sname - the sname
copynr - the copynr
lines_read - the lines_read
lines_written - the lines_written
lines_updated - the lines_updated
lines_skipped - the lines_skipped
errors - the errors
start_date - the start_date
end_date - the end_date
Returns:
the row meta and data

getLogFields

public static final RowMetaInterface getLogFields(String comm)
Gets the log fields.

Parameters:
comm - the comm
Returns:
the log fields

toString

public String toString()
Overrides:
toString in class Object

rowsetOutputSize

public int rowsetOutputSize()
Specified by:
rowsetOutputSize in interface StepInterface
Returns:
The total amount of rows in the output buffers

rowsetInputSize

public int rowsetInputSize()
Specified by:
rowsetInputSize in interface StepInterface
Returns:
The total amount of rows in the input buffers

stopRunning

public void stopRunning(StepMetaInterface stepMetaInterface,
                        StepDataInterface stepDataInterface)
                 throws KettleException
Perform actions to stop a running step. This can be stopping running SQL queries (cancel), etc. Default it doesn't do anything.

Specified by:
stopRunning in interface StepInterface
Parameters:
stepDataInterface - The interface to the step data containing the connections, resultsets, open files, etc.
stepMetaInterface - The metadata that might be needed by the step to stop running.
Throws:
KettleException - in case something goes wrong

stopRunning

public void stopRunning()
Deprecated. 

Stops running operations This method is deprecated, please use the method specifying the metadata and data interfaces.


logSummary

public void logSummary()
Log summary.


getStepID

public String getStepID()
Specified by:
getStepID in interface StepInterface
Returns:
the type ID of the step...

getInputRowSets

public List<RowSet> getInputRowSets()
Specified by:
getInputRowSets in interface StepInterface
Returns:
Returns the inputRowSets.

setInputRowSets

public void setInputRowSets(List<RowSet> inputRowSets)
Parameters:
inputRowSets - The inputRowSets to set.

getOutputRowSets

public List<RowSet> getOutputRowSets()
Specified by:
getOutputRowSets in interface StepInterface
Returns:
Returns the outputRowSets.

setOutputRowSets

public void setOutputRowSets(List<RowSet> outputRowSets)
Parameters:
outputRowSets - The outputRowSets to set.

isDistributed

public boolean isDistributed()
Returns:
Returns the distributed.

setDistributed

public void setDistributed(boolean distributed)
Parameters:
distributed - The distributed to set.

addRowListener

public void addRowListener(RowListener rowListener)
Description copied from interface: StepInterface
Add a rowlistener to the step allowing you to inspect (or manipulate, be careful) the rows coming in or exiting the step.

Specified by:
addRowListener in interface StepInterface
Parameters:
rowListener - the rowlistener to add

removeRowListener

public void removeRowListener(RowListener rowListener)
Description copied from interface: StepInterface
Remove a rowlistener from this step.

Specified by:
removeRowListener in interface StepInterface
Parameters:
rowListener - the rowlistener to remove

getRowListeners

public List<RowListener> getRowListeners()
Specified by:
getRowListeners in interface StepInterface
Returns:
a list of the installed RowListeners

addResultFile

public void addResultFile(ResultFile resultFile)
Adds the result file.

Parameters:
resultFile - the result file

getResultFiles

public Map<String,ResultFile> getResultFiles()
Specified by:
getResultFiles in interface StepInterface
Returns:
The result files for this step

getStatus

public BaseStepData.StepExecutionStatus getStatus()
Specified by:
getStatus in interface StepInterface
Returns:
the description as in StepDataInterface

getPartitionID

public String getPartitionID()
Specified by:
getPartitionID in interface StepInterface
Returns:
the partitionID

setPartitionID

public void setPartitionID(String partitionID)
Specified by:
setPartitionID in interface StepInterface
Parameters:
partitionID - the partitionID to set

getPartitionTargets

public Map<String,BlockingRowSet> getPartitionTargets()
Returns:
the partitionTargets

setPartitionTargets

public void setPartitionTargets(Map<String,BlockingRowSet> partitionTargets)
Parameters:
partitionTargets - the partitionTargets to set

getRepartitioning

public int getRepartitioning()
Returns:
the repartitioning type

setRepartitioning

public void setRepartitioning(int repartitioning)
Specified by:
setRepartitioning in interface StepInterface
Parameters:
repartitioning - the repartitioning type to set

isPartitioned

public boolean isPartitioned()
Specified by:
isPartitioned in interface StepInterface
Returns:
the partitioned

setPartitioned

public void setPartitioned(boolean partitioned)
Specified by:
setPartitioned in interface StepInterface
Parameters:
partitioned - the partitioned to set

getInputRowMeta

public RowMetaInterface getInputRowMeta()
Returns:
the rowMeta

setInputRowMeta

public void setInputRowMeta(RowMetaInterface rowMeta)
Parameters:
rowMeta - the rowMeta to set

getErrorRowMeta

public RowMetaInterface getErrorRowMeta()
Returns:
the errorRowMeta

setErrorRowMeta

public void setErrorRowMeta(RowMetaInterface errorRowMeta)
Parameters:
errorRowMeta - the errorRowMeta to set

getPreviewRowMeta

public RowMetaInterface getPreviewRowMeta()
Returns:
the previewRowMeta

setPreviewRowMeta

public void setPreviewRowMeta(RowMetaInterface previewRowMeta)
Parameters:
previewRowMeta - the previewRowMeta to set

copyVariablesFrom

public void copyVariablesFrom(VariableSpace space)
Description copied from interface: VariableSpace
Copy the variables from another space, without initializing with the defaults. This does not affect any parent relationship.

Specified by:
copyVariablesFrom in interface VariableSpace
Parameters:
space - the space to copy the variables from.

environmentSubstitute

public String environmentSubstitute(String aString)
Description copied from interface: VariableSpace
Substitute the string using the current variable space.

Specified by:
environmentSubstitute in interface VariableSpace
Parameters:
aString - The string to substitute.
Returns:
The substituted string.

environmentSubstitute

public String[] environmentSubstitute(String[] aString)
Description copied from interface: VariableSpace
Replaces environment variables in an array of strings. See also: environmentSubstitute(String string)

Specified by:
environmentSubstitute in interface VariableSpace
Parameters:
aString - The array of strings that wants its variables to be replaced.
Returns:
the array with the environment variables replaced.

getParentVariableSpace

public VariableSpace getParentVariableSpace()
Description copied from interface: VariableSpace
Get the parent of the variable space.

Specified by:
getParentVariableSpace in interface VariableSpace
Returns:
the parent.

setParentVariableSpace

public void setParentVariableSpace(VariableSpace parent)
Description copied from interface: VariableSpace
Set the parent variable space

Specified by:
setParentVariableSpace in interface VariableSpace
Parameters:
parent - The parent variable space to set

getVariable

public String getVariable(String variableName,
                          String defaultValue)
Description copied from interface: VariableSpace
Get the value of a variable with a default in case the variable is not found.

Specified by:
getVariable in interface VariableSpace
Parameters:
variableName - The name of the variable
defaultValue - The default value in case the variable could not be found
Returns:
the String value of a variable

getVariable

public String getVariable(String variableName)
Description copied from interface: VariableSpace
Get the value of a variable.

Specified by:
getVariable in interface VariableSpace
Parameters:
variableName - The name of the variable
Returns:
the String value of a variable or null in case the variable could not be found.

getBooleanValueOfVariable

public boolean getBooleanValueOfVariable(String variableName,
                                         boolean defaultValue)
Description copied from interface: VariableSpace
This method returns a boolean for the new variable check boxes. If the variable name is not set or the variable name is not specified, this method simply returns the default value. If not, it convert the variable value to a boolean. "Y", "YES" and "TRUE" all convert to true. (case insensitive)

Specified by:
getBooleanValueOfVariable in interface VariableSpace
Parameters:
variableName - The variable to look up.
defaultValue - The default value to return.
Returns:
See Also:
static method ValueMeta.convertStringToBoolean()

initializeVariablesFrom

public void initializeVariablesFrom(VariableSpace parent)
Description copied from interface: VariableSpace
Initialize variable space using the defaults, copy over the variables from the parent (using copyVariablesFrom()), after this the "injected" variables should be inserted (injectVariables()). The parent is set as parent variable space.

Specified by:
initializeVariablesFrom in interface VariableSpace
Parameters:
parent - the parent to start from, or null if root.

listVariables

public String[] listVariables()
Description copied from interface: VariableSpace
List the variables (not the values) that are currently in the variable space.

Specified by:
listVariables in interface VariableSpace
Returns:
Array of String variable names.

setVariable

public void setVariable(String variableName,
                        String variableValue)
Description copied from interface: VariableSpace
Sets a variable in the Kettle Variables list.

Specified by:
setVariable in interface VariableSpace
Parameters:
variableName - The name of the variable to set
variableValue - The value of the variable to set. If the variableValue is null, the variable is cleared from the list.

shareVariablesWith

public void shareVariablesWith(VariableSpace space)
Description copied from interface: VariableSpace
Share a variable space from another variable space. This means that the object should take over the space used as argument.

Specified by:
shareVariablesWith in interface VariableSpace
Parameters:
space - Variable space to be shared.

injectVariables

public void injectVariables(Map<String,String> prop)
Description copied from interface: VariableSpace
Inject variables. The behaviour should be that the properties object will be stored and at the time the VariableSpace is initialized (or upon calling this method if the space is already initialized). After injecting the link of the properties object should be removed.

Specified by:
injectVariables in interface VariableSpace
Parameters:
prop - Properties object containing key-value pairs.

getTypeId

public String getTypeId()
Returns the step ID via the getStepID() method call. Support for CheckResultSourceInterface.

Returns:
getStepID()

getSlaveNr

public int getSlaveNr()
Returns the unique slave number in the cluster.

Returns:
the unique slave number in the cluster

getClusterSize

public int getClusterSize()
Returns the cluster size.

Returns:
the cluster size

getUniqueStepNrAcrossSlaves

public int getUniqueStepNrAcrossSlaves()
Returns a unique step number across all slave servers: slaveNr * nrCopies + copyNr.

Returns:
a unique step number across all slave servers: slaveNr * nrCopies + copyNr

getUniqueStepCountAcrossSlaves

public int getUniqueStepCountAcrossSlaves()
Returns the number of unique steps across all slave servers.

Returns:
the number of unique steps across all slave servers

getServerSockets

public List<ServerSocket> getServerSockets()
Returns the serverSockets.

Returns:
the serverSockets

setServerSockets

public void setServerSockets(List<ServerSocket> serverSockets)
Parameters:
serverSockets - the serverSockets to set

setUsingThreadPriorityManagment

public void setUsingThreadPriorityManagment(boolean usingThreadPriorityManagment)
Set to true to actively manage priorities of step threads.

Specified by:
setUsingThreadPriorityManagment in interface StepInterface
Parameters:
usingThreadPriorityManagment - set to true to actively manage priorities of step threads

isUsingThreadPriorityManagment

public boolean isUsingThreadPriorityManagment()
Retusn true if we are actively managing priorities of step threads.

Specified by:
isUsingThreadPriorityManagment in interface StepInterface
Returns:
true if we are actively managing priorities of step threads

initBeforeStart

public void initBeforeStart()
                     throws KettleStepException
This method is executed by Trans right before the threads start and right after initialization. More to the point: here we open remote output step sockets.

Specified by:
initBeforeStart in interface StepInterface
Throws:
KettleStepException - In case there is an error

getStepListeners

public List<StepListener> getStepListeners()
Returns the step listeners.

Returns:
the stepListeners

setStepListeners

public void setStepListeners(List<StepListener> stepListeners)
Sets the step listeners.

Parameters:
stepListeners - the stepListeners to set

processRow

public boolean processRow(StepMetaInterface smi,
                          StepDataInterface sdi)
                   throws KettleException
Description copied from interface: StepInterface
Perform the equivalent of processing one row. Typically this means reading a row from input (getRow()) and passing a row to output (putRow)).

Specified by:
processRow in interface StepInterface
Parameters:
smi - The steps metadata to work with
sdi - The steps temporary working data to work with (database connections, result sets, caches, temporary variables, etc.)
Returns:
false if no more rows can be processed or an error occurred.
Throws:
KettleException

canProcessOneRow

public boolean canProcessOneRow()
Description copied from interface: StepInterface
This method checks if the step is capable of processing at least one row.

For example, if a step has no input records but needs at least one to function, it will return false.

Specified by:
canProcessOneRow in interface StepInterface
Returns:
true if the step can process a row.

addStepListener

public void addStepListener(StepListener stepListener)
Description copied from interface: StepInterface
Attach a step listener to be notified when a step arrives in a certain state. (finished)

Specified by:
addStepListener in interface StepInterface
Parameters:
stepListener - The listener to add to the step

isMapping

public boolean isMapping()
Specified by:
isMapping in interface StepInterface
Returns:
true if the thread is a special mapping step

getSocketRepository

public SocketRepository getSocketRepository()
Retutns the socket repository.

Returns:
the socketRepository

setSocketRepository

public void setSocketRepository(SocketRepository socketRepository)
Sets the socket repository.

Parameters:
socketRepository - the socketRepository to set

getObjectName

public String getObjectName()
Description copied from interface: LoggingObjectInterface
Gets the object name.

Specified by:
getObjectName in interface LoggingObjectInterface
Returns:
the name

getLogChannel

public LogChannelInterface getLogChannel()
Specified by:
getLogChannel in interface HasLogChannelInterface
Specified by:
getLogChannel in interface StepInterface
Returns:
the logging channel for this step

getFilename

public String getFilename()
Description copied from interface: LoggingObjectInterface
Gets the filename.

Specified by:
getFilename in interface LoggingObjectInterface
Returns:
the filename

getLogChannelId

public String getLogChannelId()
Description copied from interface: LoggingObjectInterface
Gets the log channel id.

Specified by:
getLogChannelId in interface LoggingObjectInterface
Returns:
the log channel id

getObjectId

public ObjectId getObjectId()
Description copied from interface: LoggingObjectInterface
Gets the object id in a repository.

Specified by:
getObjectId in interface LoggingObjectInterface
Returns:
the objectId in a repository

getObjectRevision

public ObjectRevision getObjectRevision()
Description copied from interface: LoggingObjectInterface
Gets the objects revision in a repository.

Specified by:
getObjectRevision in interface LoggingObjectInterface
Returns:
the object revision in a repository

getObjectType

public LoggingObjectType getObjectType()
Description copied from interface: LoggingObjectInterface
Gets the object type.

Specified by:
getObjectType in interface LoggingObjectInterface
Returns:
the objectType

getParent

public LoggingObjectInterface getParent()
Description copied from interface: LoggingObjectInterface
Gets the parent.

Specified by:
getParent in interface LoggingObjectInterface
Returns:
the parent

getRepositoryDirectory

public RepositoryDirectory getRepositoryDirectory()
Description copied from interface: LoggingObjectInterface
Gets the repository directory.

Specified by:
getRepositoryDirectory in interface LoggingObjectInterface
Returns:
the repositoryDirectory

getObjectCopy

public String getObjectCopy()
Description copied from interface: LoggingObjectInterface
Gets a string identifying a copy in a series of steps.

Specified by:
getObjectCopy in interface LoggingObjectInterface
Returns:
A string identifying a copy in a series of steps.

getLogLevel

public LogLevel getLogLevel()
Description copied from interface: LoggingObjectInterface
Gets the logging level of the log channel of this logging object.

Specified by:
getLogLevel in interface LoggingObjectInterface
Returns:
The logging level of the log channel of this logging object.

setLogLevel

public void setLogLevel(LogLevel logLevel)
Sets the log level.

Parameters:
logLevel - the new log level

closeQuietly

public static void closeQuietly(Closeable cl)
Close quietly.

Parameters:
cl - the object that can be closed.

getContainerObjectId

public String getContainerObjectId()
Returns the container object ID.

Specified by:
getContainerObjectId in interface LoggingObjectInterface
Returns:
the containerObjectId

setCarteObjectId

public void setCarteObjectId(String containerObjectId)
Sets the container object ID.

Parameters:
containerObjectId - the containerObjectId to set

batchComplete

public void batchComplete()
                   throws KettleException
Description copied from interface: StepInterface
Calling this method will alert the step that we finished passing a batch of records to the step. Specifically for steps like "Sort Rows" it means that the buffered rows can be sorted and passed on.

Specified by:
batchComplete in interface StepInterface
Throws:
KettleException - In case an error occurs during the processing of the batch of rows.

getRemoteInputSteps

public List<RemoteStep> getRemoteInputSteps()
Gets the remote input steps.

Returns:
the remote input steps

getRemoteOutputSteps

public List<RemoteStep> getRemoteOutputSteps()
Gets the remote output steps.

Returns:
the remote output steps

getRegistrationDate

public Date getRegistrationDate()
Returns the registration date

Specified by:
getRegistrationDate in interface LoggingObjectInterface
Returns:
The registration date of this logging object.