org.pentaho.di.trans.step
Interface StepMetaInterface

All Known Subinterfaces:

InputFileMetaInterface

All Known Implementing Classes:

AbortMeta, AbstractStepMeta, AccessInputMeta, AccessOutputMeta, AddSequenceMeta, AddXMLMeta, AggregateRowsMeta, AnalyticQueryMeta, AppendMeta, AutoDocMeta, BlockingStepMeta, BlockUntilStepsFinishMeta, CalculatorMeta, ChangeFileEncodingMeta, CheckSumMeta, CloneRowMeta, ClosureGeneratorMeta, ColumnExistsMeta, CombinationLookupMeta, ConcatFieldsMeta, ConstantMeta, CreditCardValidatorMeta, CsvInputMeta, CubeInputMeta, CubeOutputMeta, DatabaseJoinMeta, DatabaseLookupMeta, DataGridMeta, DBProcMeta, DelayMeta, DeleteMeta, DenormaliserMeta, DetectEmptyStreamMeta, DetectLastRowMeta, DimensionLookupMeta, DummyTransMeta, DynamicSQLRowMeta, Edi2XmlMeta, ElasticSearchBulkMeta, ExcelInputMeta, ExcelOutputMeta, ExcelWriterStepMeta, ExecProcessMeta, ExecSQLMeta, ExecSQLRowMeta, FieldsChangeSequenceMeta, FieldSplitterMeta, FileExistsMeta, FileLockedMeta, FilesFromResultMeta, FilesToResultMeta, FilterRowsMeta, FixedInputMeta, FlattenerMeta, FormulaMeta, FuzzyMatchMeta, GaInputStepMeta, GetFileNamesMeta, GetFilesRowsCountMeta, GetPreviousRowFieldMeta, GetRepositoryNamesMeta, GetSlaveSequenceMeta, GetSubFoldersMeta, GetTableNamesMeta, GetVariableMeta, GetXMLDataMeta, GPBulkLoaderMeta, GroupByMeta, HTTPMeta, HTTPPOSTMeta, IfNullMeta, InfobrightLoaderMeta, IngresVectorwiseLoaderMeta, InjectorMeta, InsertUpdateMeta, JaninoMeta, JavaFilterMeta, JoinRowsMeta, JsonInputMeta, JsonOutputMeta, LDAPInputMeta, LDAPOutputMeta, LDIFInputMeta, LoadFileInputMeta, LucidDBBulkLoaderMeta, LucidDBStreamingLoaderMeta, MailInputMeta, MailMeta, MailValidatorMeta, MappingInputMeta, MappingMeta, MappingOutputMeta, MemoryGroupByMeta, MergeJoinMeta, MergeRowsMeta, MetaInjectMeta, MondrianInputMeta, MonetDBAgileMartMeta, MonetDBBulkLoaderMeta, MultiMergeJoinMeta, MySQLBulkLoaderMeta, NormaliserMeta, NullIfMeta, NumberRangeMeta, OlapInputMeta, OraBulkLoaderMeta, ParGzipCsvInputMeta, PentahoReportingOutputMeta, PGBulkLoaderMeta, PrioritizeStreamsMeta, ProcessFilesMeta, PropertyInputMeta, PropertyOutputMeta, RandomCCNumberGeneratorMeta, RandomValueMeta, RegexEvalMeta, ReplaceStringMeta, ReservoirSamplingMeta, RestMeta, RowGeneratorMeta, RowsFromResultMeta, RowsToResultMeta, RssInputMeta, RssOutputMeta, RulesMeta, SalesforceDeleteMeta, SalesforceInputMeta, SalesforceInsertMeta, SalesforceUpdateMeta, SalesforceUpsertMeta, SampleRowsMeta, SapInputMeta, SasInputMeta, ScriptMeta, ScriptValuesMetaMod, SecretKeyGeneratorMeta, SelectValuesMeta, SetValueConstantMeta, SetValueFieldMeta, SetVariableMeta, SingleThreaderMeta, SocketReaderMeta, SocketWriterMeta, SortedMergeMeta, SortRowsMeta, SplitFieldToRowsMeta, SQLFileOutputMeta, SSHMeta, StepMetastructureMeta, StepsMetricsMeta, StreamLookupMeta, StringCutMeta, StringOperationsMeta, SwitchCaseMeta, SymmetricCryptoTransMeta, SynchronizeAfterMergeMeta, SyslogMessageMeta, SystemDataMeta, TableAgileMartMeta, TableExistsMeta, TableInputMeta, TableOutputMeta, TeraFastMeta, TextFileInputMeta, TextFileOutputMeta, UniqueRowsByHashSetMeta, UniqueRowsMeta, UnivariateStatsMeta, UpdateMeta, UserDefinedJavaClassMeta, ValidatorMeta, ValueMapperMeta, WebServiceAvailableMeta, WebServiceMeta, WriteToLogMeta, XBaseInputMeta, XMLInputMeta, XMLInputSaxMeta, XMLInputStreamMeta, XMLJoinMeta, XMLOutputMeta, XsdValidatorMeta, XsltMeta, YamlInputMeta

public interface StepMetaInterface

This interface allows custom steps to talk to Kettle. The StepMetaInterface is the main Java interface that a plugin implements. The responsibilities of the implementing class are listed below:

• Keep track of the step settings
  The implementing class typically keeps track of step settings using private fields with corresponding getters and setters. The dialog class implementing StepDialogInterface is using the getters and setters to copy the user supplied configuration in and out of the dialog.

  The following interface methods also fall into the area of maintaining settings:

  void setDefault()

  This method is called every time a new step is created and should allocate or set the step configuration to sensible defaults. The values set here will be used by Spoon when a new step is created. This is often a good place to ensure that the step’s settings are initialized to non-null values. Null values can be cumbersome to deal with in serialization and dialog population, so most PDI step implementations stick to non-null values for all step settings.

  public Object clone()

  This method is called when a step is duplicated in Spoon. It needs to return a deep copy of this step meta object. It is essential that the implementing class creates proper deep copies if the step configuration is stored in modifiable objects, such as lists or custom helper objects. See org.pentaho.di.trans.steps.rowgenerator.RowGeneratorMeta.clone() for an example on creating a deep copy.

• Serialize step settings
  The plugin needs to be able to serialize its settings to both XML and a PDI repository. The interface methods are as follows.

  public String getXML()

  This method is called by PDI whenever a step needs to serialize its settings to XML. It is called when saving a transformation in Spoon. The method returns an XML string, containing the serialized step settings. The string contains a series of XML tags, typically one tag per setting. The helper class org.pentaho.di.core.xml.XMLHandler is typically used to construct the XML string.

  public void loadXML(...)

  This method is called by PDI whenever a step needs to read its settings from XML. The XML node containing the step’s settings is passed in as an argument. Again, the helper class org.pentaho.di.core.xml.XMLHandler is typically used to conveniently read the step settings from the XML node.

  public void saveRep(...)

  This method is called by PDI whenever a step needs to save its settings to a PDI repository. The repository object passed in as the first argument provides a convenient set of methods for serializing step settings. The transformation id and step id passed in should be used by the step as identifiers when calling the repository serialization methods.

  public void readRep(...)

  This method is called by PDI whenever a step needs to read its configuration from a PDI repository. The step id given in the arguments should be used as the identifier when using the repositories serialization methods.

• Provide instances of other plugin classes
  The StepMetaInterface plugin class is the main class tying in with the rest of PDI architecture. It is responsible for supplying instances of the other plugin classes implementing StepDialogInterface, StepInterface and StepDataInterface. The following methods cover these responsibilities. Each of the method’s implementation is typically constructing a new instance of the corresponding class forwarding the passed in arguments to the constructor. The methods are as follows.

  public StepDialogInterface getDialog(...)
  public StepInterface getStep(...)
  public StepDataInterface getStepData()
  

  Each of the above methods returns a new instance of the plugin class implementing StepDialogInterface, StepInterface and StepDataInterface.

• Report the step’s changes to the row stream PDI needs to know how a step affects the row structure. A step may be adding or removing fields, as well as modifying the metadata of a field. The method implementing this aspect of a step plugin is getFields().

  public void getFields(...)

  Given a description of the input rows, the plugin needs to modify it to match the structure for its output fields. The implementation modifies the passed in RowMetaInterface object to reflect any changes to the row stream. Typically a step adds fields to the row structure, which is done by creating ValueMeta objects (PDI’s default implementation of ValueMetaInterface), and appending them to the RowMetaInterface object. The section Working with Fields goes into deeper detail on ValueMetaInterface.

• Validate step settings
  Spoon supports a “validate transformation” feature, which triggers a self-check of all steps. PDI invokes the check() method of each step on the canvas allowing each step to validate its settings.

  public void check()

  Each step has the opportunity to validate its settings and verify that the configuration given by the user is reasonable. In addition to that a step typically checks if it is connected to preceding or following steps, if the nature of the step requires that kind of connection. An input step may expect to not have a preceding step for example. The check method passes in a list of check remarks that the method should append its validation results to. Spoon then displays the list of remarks collected from the steps, allowing the user to take corrective action in case of validation warnings or errors.

Given a description of the input rows, the plugin needs to modify it to match the structure for its output fields. The implementation modifies the passed in RowMetaInterface object to reflect any changes to the row stream. Typically a step adds fields to the row structure, which is done by creating ValueMeta objects (PDI’s default implementation of ValueMetaInterface), and appending them to the RowMetaInterface object. The section Working with Fields goes into deeper detail on ValueMetaInterface.

Since:

4-aug-2004

Author:

Matt

Method Summary

void	analyseImpact(List<DatabaseImpact> impact, TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev, String[] input, String[] output, RowMetaInterface info) Each step must be able to report on the impact it has on a database, table field, etc.
void	cancelQueries() Call this to cancel trailing database queries (too long running, etc)
void	check(List<CheckResultInterface> remarks, TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev, String[] input, String[] output, RowMetaInterface info) Checks the settings of this step and puts the findings in a remarks List.
Object	clone() Make an exact copy of this step, make sure to explicitly copy Collections etc.
boolean	excludeFromCopyDistributeVerification() This method is added to exclude certain steps from copy/distribute checking.
boolean	excludeFromRowLayoutVerification() This method is added to exclude certain steps from layout checking.
String	exportResources(VariableSpace space, Map<String,ResourceDefinition> definitions, ResourceNamingInterface resourceNamingInterface, Repository repository)
String	getDialogClassName() Get the name of the class that implements the dialog for this job entry JobEntryBase provides a default
void	getFields(RowMetaInterface inputRowMeta, String name, RowMetaInterface[] info, StepMeta nextStep, VariableSpace space) Get the fields that are emitted by this step
List<StreamInterface>	getOptionalStreams()
StepMeta	getParentStepMeta()
RowMetaInterface	getRequiredFields(VariableSpace space) The natural way of data flow in a transformation is source-to-target.
List<ResourceReference>	getResourceDependencies(TransMeta transMeta, StepMeta stepMeta) Get a list of all the resource dependencies that the step is depending on.
SQLStatement	getSQLStatements(TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev) Standard method to return an SQLStatement object with SQL statements that the step needs in order to work correctly.
StepInterface	getStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int copyNr, TransMeta transMeta, Trans trans) Get the executing step, needed by Trans to launch a step.
StepDataInterface	getStepData() Get a new instance of the appropriate data class.
StepIOMetaInterface	getStepIOMeta() Returns the Input/Output metadata for this step.
StepMetaInjectionInterface	getStepMetaInjectionInterface()
TransMeta.TransformationType[]	getSupportedTransformationTypes()
RowMetaInterface	getTableFields()
Map<String,String>	getUsedArguments() Default a step doesn't use any arguments.
DatabaseMeta[]	getUsedDatabaseConnections() This method returns all the database connections that are used by the step.
String[]	getUsedLibraries()
String	getXML() Get the XML that represents the values in this step
void	handleStreamSelection(StreamInterface stream) When an optional stream is selected, this method is called to handled the ETL metadata implications of that.
boolean	hasRepositoryReferences()
void	loadXML(Node stepnode, List<DatabaseMeta> databases, Map<String,Counter> counters) Load the values for this step from an XML Node
void	lookupRepositoryReferences(Repository repository) Look up the references after import
void	readRep(Repository rep, ObjectId id_step, List<DatabaseMeta> databases, Map<String,Counter> counters) Read the steps information from a Kettle repository
void	resetStepIoMeta() For steps that handle dynamic input (info) or output (target) streams, it is useful to be able to force the recreate the StepIoMeta definition.
void	saveRep(Repository rep, ObjectId id_transformation, ObjectId id_step) Save the steps data into a Kettle repository
void	searchInfoAndTargetSteps(List<StepMeta> steps) Change step names into step objects to allow them to be name-changed etc.
void	setDefault() Set default values
void	setParentStepMeta(StepMeta parentStepMeta) Provide original lineage for this metadata object
boolean	supportsErrorHandling()

Method Detail

setDefault

void setDefault()

Set default values

getFields

void getFields(RowMetaInterface inputRowMeta,
               String name,
               RowMetaInterface[] info,
               StepMeta nextStep,
               VariableSpace space)
               throws KettleStepException

Get the fields that are emitted by this step

Parameters:

inputRowMeta - The fields that are entering the step. These are changed to reflect the output metadata.

name - The name of the step to be used as origin

info - The input rows metadata that enters the step through the specified channels in the same order as in method getInfoSteps(). The step metadata can then choose what to do with it: ignore it or not. Interesting is also that in case of database lookups, the layout of the target database table is put in info[0]

nextStep - if this is a non-null value, it's the next step in the transformation. The one who's asking, the step where the data is targetted towards.

space - TODO

Throws:

KettleStepException - when an error occurred searching for the fields.

getXML

String getXML()
              throws KettleException

Get the XML that represents the values in this step

Returns:

the XML that represents the metadata in this step

Throws:

KettleException - in case there is a conversion or XML encoding error

loadXML

void loadXML(Node stepnode,
             List<DatabaseMeta> databases,
             Map<String,Counter> counters)
             throws KettleXMLException

Load the values for this step from an XML Node

Parameters:

stepnode - the Node to get the info from

databases - The available list of databases to reference to

counters - Counters to reference.

Throws:

KettleXMLException - When an unexpected XML error occurred. (malformed etc.)

saveRep

void saveRep(Repository rep,
             ObjectId id_transformation,
             ObjectId id_step)
             throws KettleException

Save the steps data into a Kettle repository

Parameters:

rep - The Kettle repository to save to

id_transformation - The transformation ID

id_step - The step ID

Throws:

KettleException - When an unexpected error occurred (database, network, etc)

readRep

void readRep(Repository rep,
             ObjectId id_step,
             List<DatabaseMeta> databases,
             Map<String,Counter> counters)
             throws KettleException

Read the steps information from a Kettle repository

Parameters:

rep - The repository to read from

id_step - The step ID

databases - The databases to reference

counters - The counters to reference

Throws:

KettleException - When an unexpected error occurred (database, network, etc)

check

void check(List<CheckResultInterface> remarks,
           TransMeta transMeta,
           StepMeta stepMeta,
           RowMetaInterface prev,
           String[] input,
           String[] output,
           RowMetaInterface info)

Checks the settings of this step and puts the findings in a remarks List.

Parameters:

remarks - The list to put the remarks in @see org.pentaho.di.core.CheckResult

stepMeta - The stepMeta to help checking

prev - The fields coming from the previous step

input - The input step names

output - The output step names

info - The fields that are used as information by the step

clone

Object clone()

Make an exact copy of this step, make sure to explicitly copy Collections etc.

Returns:

an exact copy of this step

getTableFields

RowMetaInterface getTableFields()

Returns:

The fields used by this step, this is being used for the Impact analyses.

excludeFromRowLayoutVerification

boolean excludeFromRowLayoutVerification()

This method is added to exclude certain steps from layout checking.

Since:

2.5.0

excludeFromCopyDistributeVerification

boolean excludeFromCopyDistributeVerification()

This method is added to exclude certain steps from copy/distribute checking.

Since:

4.0.0

getDialogClassName

String getDialogClassName()

Get the name of the class that implements the dialog for this job entry JobEntryBase provides a default

getStep

StepInterface getStep(StepMeta stepMeta,
                      StepDataInterface stepDataInterface,
                      int copyNr,
                      TransMeta transMeta,
                      Trans trans)

Get the executing step, needed by Trans to launch a step.

Parameters:

stepMeta - The step info

stepDataInterface - the step data interface linked to this step. Here the step can store temporary data, database connections, etc.

copyNr - The copy nr to get

transMeta - The transformation info

trans - The launching transformation

getStepData

StepDataInterface getStepData()

Get a new instance of the appropriate data class. This data class implements the StepDataInterface. It basically contains the persisting data that needs to live on, even if a worker thread is terminated.

Returns:

The appropriate StepDataInterface class.

analyseImpact

void analyseImpact(List<DatabaseImpact> impact,
                   TransMeta transMeta,
                   StepMeta stepMeta,
                   RowMetaInterface prev,
                   String[] input,
                   String[] output,
                   RowMetaInterface info)
                   throws KettleStepException

Each step must be able to report on the impact it has on a database, table field, etc.

Parameters:

impact - The list of impacts @see org.pentaho.di.transMeta.DatabaseImpact

transMeta - The transformation information

stepMeta - The step information

prev - The fields entering this step

input - The previous step names

output - The output step names

info - The fields used as information by this step

Throws:

KettleStepException

getSQLStatements

SQLStatement getSQLStatements(TransMeta transMeta,
                              StepMeta stepMeta,
                              RowMetaInterface prev)
                              throws KettleStepException

Standard method to return an SQLStatement object with SQL statements that the step needs in order to work correctly. This can mean "create table", "create index" statements but also "alter table ... add/drop/modify" statements.

Parameters:

transMeta - TransInfo object containing the complete transformation

stepMeta - StepMeta object containing the complete step

prev - Row containing meta-data for the input fields (no data)

Returns:

The SQL Statements for this step. If nothing has to be done, the SQLStatement.getSQL() == null. @see SQLStatement

Throws:

KettleStepException

cancelQueries

void cancelQueries()
                   throws KettleDatabaseException

Call this to cancel trailing database queries (too long running, etc)

Throws:

KettleDatabaseException

getUsedArguments

Map<String,String> getUsedArguments()

Default a step doesn't use any arguments. Implement this to notify the GUI that a window has to be displayed BEFORE launching a transformation. You can also use this to specify certain Environment variable values.

Returns:

A Map of argument values. (name and optionally a default value) Put 10 values in the map for the possible 10 arguments.

getRequiredFields

RowMetaInterface getRequiredFields(VariableSpace space)
                                   throws KettleException

The natural way of data flow in a transformation is source-to-target. However, this makes mapping to target tables difficult to do. To help out here, we supply information to the transformation meta-data model about which fields are required for a step. This allows us to automate certain tasks like the mapping to pre-defined tables. The Table Output step in this case will output the fields in the target table using this method.

Parameters:

space - the variable space to reference

Returns:

the required fields for this steps metadata.

Throws:

KettleException - in case the required fields can't be determined.

getUsedDatabaseConnections

DatabaseMeta[] getUsedDatabaseConnections()

This method returns all the database connections that are used by the step.

Returns:

an array of database connections meta-data. Return an empty array if no connections are used.

getUsedLibraries

String[] getUsedLibraries()

Returns:

the libraries that this step or plugin uses.

supportsErrorHandling

boolean supportsErrorHandling()

Returns:

true if this step supports error "reporting" on rows: the ability to send rows to a certain target step.

getResourceDependencies

List<ResourceReference> getResourceDependencies(TransMeta transMeta,
                                                StepMeta stepMeta)

Get a list of all the resource dependencies that the step is depending on.

Parameters:

transMeta -

stepMeta -

Returns:

a list of all the resource dependencies that the step is depending on

exportResources

String exportResources(VariableSpace space,
                       Map<String,ResourceDefinition> definitions,
                       ResourceNamingInterface resourceNamingInterface,
                       Repository repository)
                       throws KettleException

Parameters:

space - the variable space to use

definitions -

resourceNamingInterface -

repository - The repository to optionally load other resources from (to be converted to XML)

Returns:

the filename of the exported resource

Throws:

KettleException

getParentStepMeta

StepMeta getParentStepMeta()

Returns:

The StepMeta object to which this metadata class belongs. With this, we can see to which transformation metadata (etc) this metadata pertains to. (hierarchy)

setParentStepMeta

void setParentStepMeta(StepMeta parentStepMeta)

Provide original lineage for this metadata object

Parameters:

parentStepMeta - the parent step metadata container object

getStepIOMeta

StepIOMetaInterface getStepIOMeta()

Returns the Input/Output metadata for this step.

getOptionalStreams

List<StreamInterface> getOptionalStreams()

Returns:

The list of optional input streams. It allows the user to select from a list of possible actions like "New target step"

handleStreamSelection

void handleStreamSelection(StreamInterface stream)

When an optional stream is selected, this method is called to handled the ETL metadata implications of that.

Parameters:

stream - The optional stream to handle.

resetStepIoMeta

void resetStepIoMeta()

For steps that handle dynamic input (info) or output (target) streams, it is useful to be able to force the recreate the StepIoMeta definition. By default this definition is cached.

searchInfoAndTargetSteps

void searchInfoAndTargetSteps(List<StepMeta> steps)

Change step names into step objects to allow them to be name-changed etc.

Parameters:

steps - the steps to reference

getStepMetaInjectionInterface

StepMetaInjectionInterface getStepMetaInjectionInterface()

Returns:

Optional interface that allows an external program to inject step metadata in a standardized fasion. This method will return null if the interface is not available for this step.

getSupportedTransformationTypes

TransMeta.TransformationType[] getSupportedTransformationTypes()

Returns:

The supported transformation types that this step supports.

hasRepositoryReferences

boolean hasRepositoryReferences()

Returns:

True if the job entry defines one or more references to a repository object.

lookupRepositoryReferences

void lookupRepositoryReferences(Repository repository)
                                throws KettleException

Look up the references after import

Parameters:

repository - the repository to reference.

Throws:

KettleException