|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.pentaho.di.trans.step.BaseStepMeta org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoaderMeta
public class MySQLBulkLoaderMeta
Here are the steps that we need to take to make streaming loading possible for MySQL:
The following steps are carried out by the step at runtime:
- create a unique FIFO file (using mkfifo, LINUX ONLY FOLKS!)
- Create a target table using standard Kettle SQL generation
- Execute the LOAD DATA SQL Command to bulk load in a separate SQL thread in the background:
- Write to the FIFO file
- At the end, close the output stream to the FIFO file
* At the end, remove the FIFO file
Created on 24-oct-2007
Field Summary | |
---|---|
static int |
FIELD_FORMAT_TYPE_DATE
|
static int |
FIELD_FORMAT_TYPE_NUMBER
|
static int |
FIELD_FORMAT_TYPE_OK
|
static int |
FIELD_FORMAT_TYPE_STRING_ESCAPE
|
static int |
FIELD_FORMAT_TYPE_TIMESTAMP
|
Fields inherited from class org.pentaho.di.trans.step.BaseStepMeta |
---|
loggingObject, STEP_ATTRIBUTES_FILE |
Constructor Summary | |
---|---|
MySQLBulkLoaderMeta()
|
Method Summary | |
---|---|
void |
allocate(int nrvalues)
|
void |
analyseImpact(List<DatabaseImpact> impact,
TransMeta transMeta,
StepMeta stepMeta,
RowMetaInterface prev,
String[] input,
String[] output,
RowMetaInterface info)
Each step must be able to report on the impact it has on a database, table field, etc. |
void |
check(List<CheckResultInterface> remarks,
TransMeta transMeta,
StepMeta stepMeta,
RowMetaInterface prev,
String[] input,
String[] output,
RowMetaInterface info)
Checks the settings of this step and puts the findings in a remarks List. |
Object |
clone()
Make an exact copy of this step, make sure to explicitly copy Collections etc. |
String |
getBulkSize()
|
DatabaseMeta |
getDatabaseMeta()
Returns the database meta for this step |
String |
getDelimiter()
|
String |
getEnclosure()
|
String |
getEncoding()
|
String |
getEscapeChar()
|
int[] |
getFieldFormatType()
|
static int |
getFieldFormatType(String codeOrDescription)
|
static String |
getFieldFormatTypeCode(int type)
|
static String[] |
getFieldFormatTypeCodes()
|
static String |
getFieldFormatTypeDescription(int type)
|
static String[] |
getFieldFormatTypeDescriptions()
|
void |
getFields(RowMetaInterface rowMeta,
String origin,
RowMetaInterface[] info,
StepMeta nextStep,
VariableSpace space)
Gets the fields. |
String[] |
getFieldStream()
|
String[] |
getFieldTable()
|
String |
getFifoFileName()
|
String |
getMissingDatabaseConnectionInformationMessage()
Provides a way for this object to return a custom message when database connection information is incomplete or missing. |
RowMetaInterface |
getRequiredFields(VariableSpace space)
The natural way of data flow in a transformation is source-to-target. |
String |
getSchemaName()
Returns the schema name for this step. |
SQLStatement |
getSQLStatements(TransMeta transMeta,
StepMeta stepMeta,
RowMetaInterface prev)
Standard method to return one or more SQLStatement objects that the step needs in order to work correctly. |
StepInterface |
getStep(StepMeta stepMeta,
StepDataInterface stepDataInterface,
int cnr,
TransMeta transMeta,
Trans trans)
Get the executing step, needed by Trans to launch a step. |
StepDataInterface |
getStepData()
Get a new instance of the appropriate data class. |
String |
getTableName()
Returns the table name for this step |
DatabaseMeta[] |
getUsedDatabaseConnections()
This method returns all the database connections that are used by the step. |
String |
getXML()
Produces the XML string that describes this step's information. |
boolean |
isIgnoringErrors()
|
boolean |
isLocalFile()
|
boolean |
isReplacingData()
|
void |
loadXML(Node stepnode,
List<DatabaseMeta> databases,
Map<String,Counter> counters)
Load the values for this step from an XML Node |
void |
readRep(Repository rep,
ObjectId id_step,
List<DatabaseMeta> databases,
Map<String,Counter> counters)
Read the steps information from a Kettle repository |
void |
saveRep(Repository rep,
ObjectId id_transformation,
ObjectId id_step)
Save the steps data into a Kettle repository |
void |
setBulkSize(String bulkSize)
|
void |
setDatabaseMeta(DatabaseMeta database)
|
void |
setDefault()
Set default values |
void |
setDelimiter(String delimiter)
|
void |
setEnclosure(String enclosure)
|
void |
setEncoding(String encoding)
|
void |
setEscapeChar(String escapeChar)
|
void |
setFieldFormatType(int[] fieldFormatType)
|
void |
setFieldStream(String[] fieldStream)
|
void |
setFieldTable(String[] fieldTable)
|
void |
setFifoFileName(String fifoFileName)
|
void |
setIgnoringErrors(boolean ignoringErrors)
|
void |
setLocalFile(boolean localFile)
|
void |
setReplacingData(boolean replacingData)
|
void |
setSchemaName(String schemaName)
|
void |
setTableName(String tableName)
|
Methods inherited from class java.lang.Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final int FIELD_FORMAT_TYPE_OK
public static final int FIELD_FORMAT_TYPE_DATE
public static final int FIELD_FORMAT_TYPE_TIMESTAMP
public static final int FIELD_FORMAT_TYPE_NUMBER
public static final int FIELD_FORMAT_TYPE_STRING_ESCAPE
Constructor Detail |
---|
public MySQLBulkLoaderMeta()
Method Detail |
---|
public DatabaseMeta getDatabaseMeta()
ProvidesDatabaseConnectionInformation
getDatabaseMeta
in interface ProvidesDatabaseConnectionInformation
public void setDatabaseMeta(DatabaseMeta database)
database
- The database to set.public String getTableName()
ProvidesDatabaseConnectionInformation
getTableName
in interface ProvidesDatabaseConnectionInformation
public void setTableName(String tableName)
tableName
- The tableName to set.public String[] getFieldTable()
public void setFieldTable(String[] fieldTable)
fieldTable
- The fieldTable to set.public String[] getFieldStream()
public void setFieldStream(String[] fieldStream)
fieldStream
- The fieldStream to set.public void loadXML(Node stepnode, List<DatabaseMeta> databases, Map<String,Counter> counters) throws KettleXMLException
StepMetaInterface
loadXML
in interface StepMetaInterface
stepnode
- the Node to get the info fromdatabases
- The available list of databases to reference tocounters
- Counters to reference.
KettleXMLException
- When an unexpected XML error occurred. (malformed etc.)public void allocate(int nrvalues)
public Object clone()
StepMetaInterface
clone
in interface StepMetaInterface
clone
in class BaseStepMeta
public void setDefault()
StepMetaInterface
setDefault
in interface StepMetaInterface
public String getXML()
BaseStepMeta
getXML
in interface StepMetaInterface
getXML
in class BaseStepMeta
public void readRep(Repository rep, ObjectId id_step, List<DatabaseMeta> databases, Map<String,Counter> counters) throws KettleException
StepMetaInterface
readRep
in interface StepMetaInterface
rep
- The repository to read fromid_step
- The step IDdatabases
- The databases to referencecounters
- The counters to reference
KettleException
- When an unexpected error occurred (database, network, etc)public void saveRep(Repository rep, ObjectId id_transformation, ObjectId id_step) throws KettleException
StepMetaInterface
saveRep
in interface StepMetaInterface
rep
- The Kettle repository to save toid_transformation
- The transformation IDid_step
- The step ID
KettleException
- When an unexpected error occurred (database, network, etc)public void getFields(RowMetaInterface rowMeta, String origin, RowMetaInterface[] info, StepMeta nextStep, VariableSpace space) throws KettleStepException
BaseStepMeta
getFields
in interface StepMetaInterface
getFields
in class BaseStepMeta
rowMeta
- the input row metaorigin
- the nameinfo
- the infonextStep
- the next stepspace
- the space
KettleStepException
- the kettle step exceptionpublic void check(List<CheckResultInterface> remarks, TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev, String[] input, String[] output, RowMetaInterface info)
StepMetaInterface
check
in interface StepMetaInterface
remarks
- The list to put the remarks in @see org.pentaho.di.core.CheckResultstepMeta
- The stepMeta to help checkingprev
- The fields coming from the previous stepinput
- The input step namesoutput
- The output step namesinfo
- The fields that are used as information by the steppublic SQLStatement getSQLStatements(TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev) throws KettleStepException
BaseStepMeta
getSQLStatements
in interface StepMetaInterface
getSQLStatements
in class BaseStepMeta
transMeta
- TransInfo object containing the complete transformationstepMeta
- StepMeta object containing the complete stepprev
- Row containing meta-data for the input fields (no data)
KettleStepException
public void analyseImpact(List<DatabaseImpact> impact, TransMeta transMeta, StepMeta stepMeta, RowMetaInterface prev, String[] input, String[] output, RowMetaInterface info) throws KettleStepException
BaseStepMeta
analyseImpact
in interface StepMetaInterface
analyseImpact
in class BaseStepMeta
impact
- The list of impacts @see org.pentaho.di.transMeta.DatabaseImpacttransMeta
- The transformation informationstepMeta
- The step informationprev
- The fields entering this stepinput
- The previous step namesoutput
- The output step namesinfo
- The fields used as information by this step
KettleStepException
public StepInterface getStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int cnr, TransMeta transMeta, Trans trans)
StepMetaInterface
getStep
in interface StepMetaInterface
stepMeta
- The step infostepDataInterface
- the step data interface linked to this step. Here the step can store temporary data, database connections, etc.cnr
- The copy nr to gettransMeta
- The transformation infotrans
- The launching transformationpublic StepDataInterface getStepData()
StepMetaInterface
getStepData
in interface StepMetaInterface
public DatabaseMeta[] getUsedDatabaseConnections()
BaseStepMeta
getUsedDatabaseConnections
in interface StepMetaInterface
getUsedDatabaseConnections
in class BaseStepMeta
public RowMetaInterface getRequiredFields(VariableSpace space) throws KettleException
BaseStepMeta
getRequiredFields
in interface StepMetaInterface
getRequiredFields
in class BaseStepMeta
space
- the variable space to use to do variable substitution.
KettleException
- in case the required fields can't be determinedpublic String getSchemaName()
ProvidesDatabaseConnectionInformation
getSchemaName
in interface ProvidesDatabaseConnectionInformation
public void setSchemaName(String schemaName)
schemaName
- the schemaName to setpublic String getEncoding()
public void setEncoding(String encoding)
public String getDelimiter()
public void setDelimiter(String delimiter)
public String getEnclosure()
public void setEnclosure(String enclosure)
public String getFifoFileName()
public void setFifoFileName(String fifoFileName)
fifoFileName
- the fifoFileName to setpublic boolean isReplacingData()
public void setReplacingData(boolean replacingData)
replacingData
- the replacingData to setpublic int[] getFieldFormatType()
public void setFieldFormatType(int[] fieldFormatType)
public static String[] getFieldFormatTypeCodes()
public static String[] getFieldFormatTypeDescriptions()
public static String getFieldFormatTypeCode(int type)
public static String getFieldFormatTypeDescription(int type)
public static int getFieldFormatType(String codeOrDescription)
public String getEscapeChar()
public void setEscapeChar(String escapeChar)
escapeChar
- the escapeChar to setpublic boolean isIgnoringErrors()
public void setIgnoringErrors(boolean ignoringErrors)
ignoringErrors
- the ignoringErrors to setpublic String getBulkSize()
public void setBulkSize(String bulkSize)
bulkSize
- the bulkSize to setpublic boolean isLocalFile()
public void setLocalFile(boolean localFile)
localFile
- the localFile to setpublic String getMissingDatabaseConnectionInformationMessage()
ProvidesDatabaseConnectionInformation
null
a default message will be displayed for missing information.
getMissingDatabaseConnectionInformationMessage
in interface ProvidesDatabaseConnectionInformation
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |