1 / 22

Data Stream Processor

Data Stream Processor. Introduction. CONFIDENTIAL. Contents. What Is DSP. 1. DSP Transfer. 2. DSP Parser. 3. DSP Loader. 4. DSP Generator. 5. What is DSP.

ganya
Download Presentation

Data Stream Processor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Stream Processor Introduction CONFIDENTIAL

  2. Contents What Is DSP 1 DSPTransfer 2 DSPParser 3 DSPLoader 4 DSPGenerator 5

  3. What is DSP DSPstands for Data Stream Processor. It is an enterprise system for creating and managing enterprise data streams and for processing batch files. DSPcan transport files in and out of company network, parse them and load the parsed data into different database systems or generate new data files for downstream clients. DSPis designed to be run by an enterprise scheduler but any of its main processes can be run manually or in a daemon mode (as a background process with configurable sleep time between runs). DSPcan process thousands of files and gigabytes of transactions a day; it can run on any of the popular enterprise platforms such as Linux, Solaris, AIX and Windows. DSPis more than just a batch file processor. It enables users to create complex internal data streams and perform near-real time data updates on downstream client systems.

  4. Business Advantages Of DSP • Reduce development costs and speed-up time-to-production for new data Let’s say the IT department gets a request from a business to load new data from a vendor. Let’s assume it takes a developer a minimum one week to write a program to download the file, parse it and load it into a database including all the related tasks such as testing, move-to-production planning, implementation etc. At 8 hours per day and an average cost of $100 per hour the total cost this little project is $4000. While $4000 may not seem like a big number to a large IT department if one considers that there are usually dozens of these requests a year in an organization like that the savings balloon to tens of thousands of dollars a year, $48,000 just for the first dozen requests. With DSPall the developer has to do is setup several parameters in a few tables or a configuration file and test it. This shouldn’t take more than a couple of hours to an average developer. In fact a developer with some experience could probably do the whole setup on his or her coffee break.

  5. Business Advantages of DSP The other critical thing besides the cost reduction is the turn-around speed for the business. With DSPthey can have their data usually by the end of the day. • Improve Data Quality When you offload the tedious and mechanical work to DSPyour developers can concentrate on the data itself. They can spend most of their time analyzing the data And creating custom programs for data validation. This will improve the data quality which should direct positive impact on the company bottom line. • Simpler maintenance and support Large data processing plants often have hundreds of scripts spread out all over the company network. This makes it difficult to even keep track of them, to say nothing about maintenance and support. With DSPimplementation a lot of these can be decommissioned making the maintenance easier. Additionally a lot of the support is thus off-loaded to the vendor. • Lower development risk Since IT doesn’t need to do a lot of the development that can be performed by DSP the department automatically reduces its development risks which are often significant.

  6. DSP is Simple, Powerful, Flexible The System was designed with three main goals Flexibility DSP Power Simplicity

  7. General System Attributes • The system is easy to administer; easy to install, configure, operate and maintain. All the functionality is contained in a few binary files. There are no application servers to install and maintain, no multitude of jar files to keep track of. • There are multiple ways to configure the system and each run; in the database or in the Configuration file(s) or on the command line or any combination of these. Yet most of the options don’t need to be configured at all because they will default to the most logical setup. This can significantly speed up a new process configuration and make it less error-prone. • All the major components can be run in parallel processing mode allowing better performance and larger throughput of the data. • The system is customizable. It allows new internal variables to be defined and offers internal hooks for external custom code to be run within the system.

  8. File Transfer is the movement of files in and out of company network as well as within it Get File Parsingis extracting, validating and formatting data from the files to get it ready for loading Parse File Generating is creating data files from database data or other files File Loading is loading of the parsed data into various databases Make Load Batch Processing Tasks DSP supports four most common tasks in any data processing

  9. DSP File Transfer Vendors Customers HTTP/HTTPS FTP/SFTP DSP Transfer moves files between vendors, clients and users; in and out of the enterprise network Enterprise

  10. File Transfer Processing Tasks DSP doesn’t just transfer files, it can also do encryption, compression and archiving on them as required within the same transfer process Incoming File Outgoing File Download Upload File Transfer Decrypt Encrypt Decompress Compress Archive Archive Task Sequences within File Transfer

  11. Processing Control DSP Transfer allows process monitoring and flexible system date card flipping

  12. Other Transfer Functionality

  13. DSP File Parser DSP Parser matches file data to specific database tables and creates table data files ready for loading Data File Parser Database Table Files

  14. DSP Parser Functionality

  15. DSP File Loader Loader Table Files DSP Loader loads table files into the designated database tables Enterprise Databases

  16. DSP Loader Functionality

  17. DSP File Generator Input File Output File DSP Generator extracts data from a database or a file and generates a new data file File Generator Database Output File

  18. Generator Functionality

  19. DSP Platform & DB Support Linux DSP Supports most major platforms and database systems ORACLE SYBASE Solaris DSP MSSQL MySQL AIX DB2* PostgreSQL* Windows * Planned in the future releases

  20. Database Config File Command Line Flexible Process Setup DSP offers flexible processing setup through the use of System database tables which can be overridden by Config files which in turn can be overridden on the Command line

  21. Planned Functionality Future releases will add other significant functionality to create a comprehensive back end enterprise system LOADER PARSER TRANSFER GENERATOR

  22. Thank You ! CONFIDENTIAL

More Related