Pentaho data integration performance tuning

#Pentaho data integration performance tuning software#
#Pentaho data integration performance tuning code#
#Pentaho data integration performance tuning windows#

Experience in performing Data Masking/Protection using Pentaho Data Integration (Kettle).

Good Experience in use of other ETL and reporting tools like DTS, SSIS, SSRS, SSAS and Crystal Reports.

Experience in using Pentaho report Designer to create pixel-perfect reports of data in PDF, Excel, PL/SQL, SQL, Confidential /SQL, JavaScript, HTML, Text, Rich-Text-File, XML and CSV.

Worked on the business requirements to change the behavior of the cube by using MDX scripting.

Created ETL jobs to load Twitter JSON data and server data into MongoDB and transported MongoDB into the Data Warehouse.

Loaded unstructured data into Hadoop File System (HDFS).

This is a Kettle plugin and provides connectors to HDFS, MapReduce, HBase, Cassandra, MongoDB, CouchDB that work across Pentaho Data Integration.

Integrating Kettle (ETL) with Hadoop and other various NoSQL data stores can be found in the Pentaho Big Data Plugin.

Used bunch of steps in Pentaho transformations including Row Normalizer, Row Demoralizer, Database Lookup, Database Join, Calculator, Add Sequence, Add Constants and various types of inputs and outputs for various data sources including Tables, Access, Text File, Excel and CSV file.

Experience in creating ETL transformations and jobs using Pentaho Kettle Spoon designer and Pentaho Data Integration Designer and scheduling them on Pentaho BI Server.

Hands on experience on the whole ETL (Extract Transformation & Load) process.

#Pentaho data integration performance tuning windows#

Installed and configured Pentaho BI Server on different operating systems like Red Hat, Linux and Windows Server.

Hands-on experience on Data warehouse Star Schema Modeling, Snow-Flake Modeling, FACT & Dimension Tables, Physical and Logical Data Modeling.

Experience in Architecting and building Data Warehouse systems and Business Intelligence systems including ETL using Pentaho BI Suite (Pentaho Data Integration Designer / Kettle, Pentaho Report Designer, Pentaho Schema Workbench, Pentaho Design Studio, Pentaho Enterprise Console, Pentaho BI Server, Pentaho Meta Data, Pentaho Analysis View, Pentaho Analyzer & Mondrian OLAP).

#Pentaho data integration performance tuning software#

Knowledge about Software Development Lifecycle ( SDLC), Agile, Application Maintenance Change Process ( AMCP).Experienced in Design, Development and Implementation of large - scale projects in Financial, Shipping and Retail industries using Data Warehousing ETL tools (Pentaho) and Business Intelligence tool.Over 8+ years of experience in the field of Information Technology with proficiency in ETL design/development and Data Warehouse Implementation/development.

#Pentaho data integration performance tuning code#

Worked on emergency code fixes to Production. Troubleshoot the Productions failure and provide root cause analysis.Preparing the documents for test data loading Performed Unit testing and System integration testing.Wrote documentation to describe program development, logic, coding, testing, changes and corrections.

Designed the dimension model of the OLAP data marts in Erwin.

Understanding the Functional Requirements.

Extracted the data from Oracle, DB2, CSV and Flat files, Confidential -SQL, PL/SQL.

Responsibilities include creating the sessions and scheduling the sessions.

Created sessions and batches to move data Confidential specific intervals on demand using Server Manager.

Created different type of reports including Crosstab, Tabular, Charts, graphs, Sub reports, Drill-down reports with Cascading Parameter prompts.

Generated reports using SSRS that could be used to send information to different managers of Different Branch.

Used transformation like Merge Join, Pivot, lookup, Fuzzy lookup and slowly changing Dimension.

Wrote Perl, UNIX scripts for preliminary file check and extracting data from vendors.

Implemented Logic with lookup table to maintain Parent- Child relationship and maintain hierarchy.

Applied Configuration, Logging, Error reporting to all packages to make package deployment easy and troubleshoot package on run time.

Created Profiling, Cleansing and auditing plans with pdi jobs and transformation.

Created triggers on master data tables in OLTP to keep track of insertion and updates.

Wrote cursors in the stored procedures to perform row-wise operations.

Wrote complex stored procedures in Confidential -SQL using SQL Server Management Studio (SSMS) to summarize the data in staging environment and then developed PDIвЂ™s to load the summarized data in Data warehouse Data Mart Dimension and Fact Tables.

Involved in performance tuning of SQL queries and stored procedures using SQL Profiler and Index Tuning Wizard.

Created several Pentaho Data Integration Transformations Jobs (aka Kettle with UI as Spoon) to extract data from OLTP systems and load into staging databases.

Responsible for developing, support and maintenance for the ETL (Extract, Transform and Load) processes using Pentaho.