A Java File Transfer Framework (and More) Using Quartz

1. A Java File Transfer Framework (and More) Using Quartz Jared Lynem Amway Corporation jlynem@gmail.com

2. Hello • I’m Jared • We’ll talk about: • File transfers: requirements, reasons, concepts • A framework for file transfers • A way to schedule them • Other quick victories • Links to resources at the end of this document • Feel free to contact me • jlynem@gmail.com

3. Requirements • The first business data “integrations” • Moving paper files from one office to another • Ditto machines! Crazy. • Offices like on Mad Men • Fast forward to early digital integrations • Moving data files from one server to another • Bread and butter, reliable, easy

4. Requirements – What? • What – select and move files based on location, file name • When – based on a schedule or as a process step • Where – multiple targets, multiple sources, archiving, PCI/PII considerations • How – Protocols: FTP, SFTP, FTPS, CIFS, VPN; Encryption: PGP

5. Requirements – Why? • Why indeed. • Not our problem, just do as the customer requests. • Seriously though, fight for the best design.

6. Reasons • Why should we use file transfers? • To communicate securely with external entities • To interact with legacy applications • Because they’re simple and easy to support • When you don’t need data transformation • When someone need to review the data • Beware of PCI/PII requirements!

7. Reasons • Why develop a framework? • Oracle SOA Suite “FTP Adapter” doesn’t do what you might think it does. • Oracle Data Integrator FTP/SFTP setup is clunky, not easily extensible • Often times, many very similar integrations are required

8. Concepts - Java • Object oriented programming • Native language for Weblogic Application Server • Relatively easy, JDeveloper is a great IDE • Much, much faster than SOA tools • Widely used, plenty of knowledgeable developers • Oracle: “Resistance is futile”

9. Concepts – Transfer Protocols • FTP – Old ‘n reliable • Credential based login, or anonymous • Not encrypted • Easy to implement, widely supported • SFTP – “FTP” over SSH • Encrypted transfer using Secure Shell • Allows for additional authentication measures

10. Concepts – Transfer Protocols • FTPS – FTP with encryption over SSL • Support for multiple authentication methods • Support for public key certificates (SSL) • CIFS (aka SMB) – Used by Windows • Allows for Active Directory or Kerberos authentication • Handy when FTP isn’t set up on Windows servers

11. Concepts – Security • Transport Level Security (See above) • Data Level Security – PGP (Pretty Good Privacy) • Offers encryption in a wide variety of algorithms and strengths • Offers data compression, integrity checking, and message authentication • Unbreakable!

12. The Framework • Most file transfers follow a similar design: • Trigger condition (sometimes) • Source(s) for files • Encryption/decryption/renaming (sometimes) • Target(s) for files • Archiving (sometimes)

13. The Framework - Goal • To make these common tasks configurable via XML elements • To allow custom tasks to work seamlessly with common tasks to minimize development time

14. The Framework – JAXB • The XML part is easy: • Create an XML schema with elements for each step’s configuration • Use JAXB (built-in to JDeveloper) to create matching Java objects • Use these objects as “properties” for your step implementations • Create a mapping between XML element types and Java classes to execute

15. The Framework – Trigger Condition • A trigger condition decides whether the transfer should occur • Typically a “trigger file” is used, placed in a directory near where the actual file will be • Useful for preventing accidental transfer during file write • Configurations: FTP location, authentication, file name

16. The Framework – Source config • FTP location, authentication, remote directory, file names or file patterns • Java supports regular expressions, so use them • Not all FTP servers support glob expressions

17. The Framework – local work • Encryption config • Public key • Whether to delete unencrypted file afterwards (yes) • Decryption config • Private key • Passphrase

18. The Framework – Target Config • Multiple target elements, each with: • FTP location, authentication • Remote directory

19. The Framwork - Archiving • Considerations • Local vs. remote • Append timestamps, other clerical tasks • Archiving sensitive data

20. The Framework – Custom task • Create your own Java class to do some complicated work (i.e., building a file from a database query) • XML configuration: • Class name • Name/Value properties used by the class

21. Putting it All Together – Task Interface • A “Task” needs: • A definition: settings, configurations • A context: what happened before? • A result: something that can be interpreted and acted upon

22. Task Interface – Task Definition • Say, one of the XML elements you just defined • The Task knows what to do with it

23. Task Interface - Context • A way to keep track of what happened in previous steps • Most obvious example is, what files are we working with? • Doesn’t need to be too complex

24. Task Interface – Result • Three values • Success/Failure • Whether to continue • Message for grateful support person

25. One More Interface - Plan • Just a way to build a list of tasks from a given XML configuration • Execute them in order, checking the result after each task • Bonus points: have “Plan” implement “Task”

26. phew • Hard part is over • Lunch soon! I feel ya, bro

27. Quartz • Well known (among Java nerds) library for scheduling jobs • Free as in speech, beer • Well documented, tons of features • Runs in a clustered environment • Free web-app UIs exist • You can pay for support if you want

28. Quartz – Runs stuff • Use Quartz if: • You want to run something every five minutes • You want to run something every day • You want to run something on demand • You want to run something every 15 minutes from noon to 2pm on the fourth day of each month and every second Tuesday

29. An Alternative to Quartz • Weblogic Job Scheduler • Built-in to Weblogic =) • Not portable =( • Not widely used =( • More complicated setup =( • Documented by Oracle =(

30. Quartz – Concepts • Job – interface for Java classes, with only and “execute” method. Quartz can run any class that implements Job. • Trigger – Tells the schedule when to run • Groups – Group Jobs for better control • Listener – Your own optional class that gets notified when things run (for logging purposes) • Job Data Map – Configurations or data passed in when the Job is run.

31. Putting the FTP Framework in Quartz • Create a Java class that implements Job (so Quartz can run it) • Schedule the Job, put your XML file in the Job Data Map so it is accessible by your new class. • Have the class set up a Plan of Tasks based on the XML file, and execute it.

32. Bonus - Quartz/SOA Interaction • Quartz -> SOA • Use the javax.xml.soap libraries to create a simple web service Job • Now you can schedule your SOA composites! • SOA -> Quartz • Create a simple class in your scheduler project that triggers a job on-demand in Quartz based on an input job name. • In JDeveloper, right-click and select “Create Web Service” • Now you can kick-off scheduler jobs from SOA!

33. You’ve been a great audience • Takeaways: • Java is your best bet for file transfers if you’re using Oracle products for integration • Abstract Source/Target/Encryption tasks since they’re all pretty much the same • XML is a good format for configurations since it’s extensible and easy to import to Java

34. Thanks OWMUG • Takeaways: • Quartz is a powerful, free scheduling tool used by many large organizations • Quartz can interact with SOA via web services, too (or anything Java can interact with- ODI, AS400, SQL, etc.)

35. select * from QUESTIONS

36. Libraries and Links (all free!) • Quartz - http://quartz-scheduler.org/ • Apache Java FTP/FTPS client library - http://commons.apache.org/net/ • JSCH Java SFTP client library – http://www.jcraft.com/jsch/ • CIFS/SMB client library - http://www.samba.org/ • Bouncy Castle Java PGP library - http://www.bouncycastle.org/ • JWatch Quartz Manager UI - http://code.google.com/p/jwatch/