1 / 25

PTTP & IDL – External / Internal File Name Validation

PTTP & IDL – External / Internal File Name Validation. Doug Clough SYNERGETICS Engineered Systems. 7 December 2009 www.ptagis.org. IDL – Loader Invocation. PTTP. Files are pre-processed by idl_validateFile.pl one-by-one as IDL runs. IDL – Loader Invocation. PTTP.

cissy
Download Presentation

PTTP & IDL – External / Internal File Name Validation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. PTTP & IDL – External / Internal File Name Validation Doug Clough SYNERGETICS Engineered Systems 7 December 2009 www.ptagis.org

  2. IDL – Loader Invocation PTTP Files are pre-processed byidl_validateFile.pl one-by-one as IDL runs

  3. IDL – Loader Invocation PTTP Files are pre-processed byidl_validateFile.pl one-by-one as IDL runs PROBLEM: We must detect external-internal file name mismatch before file is presented to IDL

  4. Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have a look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

  5. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type Symbolic link in archive directory is created first …

  6. sub analyze {    my    (      $obj    ) = @_;   $TX_ID  = $obj->get("PTTP_Req.TX_ID");   $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;   ( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));   foreach $FileName ( @DataFiles ) {      # NOTE: Assume that all files of type 'INT' and 'MUL'      #       will be analyzed by TASS and SBCA. Leave it to      #       TASS_Monitor and SBCA_Monitor to decide which      #       files to analyze.      $FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE");      $ReqOper  = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");      if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) &&           ( $ReqOper  =~ /LOAD/i          )    ) {         # Capture $SiteCode for use below ...         $FileName =~ /^\s*(...).*$/;         $SiteCode = $1;         # Unless the file is already present in the final FDVL archive         # location, create a symbolic link there pointing to the file in         # $TX_Dir. FileType not important here; INT and MUL are equivalent.FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');         $HasFilesToAnalyze = 'True';      }   } PTTP_Req::analyze()

  7. sub makeArchiveLink {   my    (      $PhysicalPath    , $ArchivalPath    , $FullSpec    , $CmdString    );   my    (      $SourceDir    , $FileName    , $FileType    # INT, MUL, or TAG (INT and MUL give same result)    )        = @_;   $PhysicalPath = "${SourceDir}/${FileName}";   $ArchivalPath = &sayAbsolutePath($FileName,$FileType);   $FullSpec     = "${ArchivalPath}/${FileName}";unless ( -f $FullSpec ) {      # Create directories if they don't exist ...      &checkDirs($FileName,$FileType);$CmdString    = "ln -s $PhysicalPath $FullSpec";      system("$CmdString");   }   else {      print "makeArchiveLink ...\n"                            if ( $DEBUG );      print "   Already exists: $FullSpec\n"                   if ( $DEBUG );      $DUMMY = <STDIN>                                         if ( $DEBUG );   }} FDVL_ArchiveTree::makeArchiveLink()

  8. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type How are files presented for loading?

  9. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event(s) • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event One per file run_loader($evt){<copies $FileName to $INTERROGATION_DATA>}

  10. sub runLoader {   my    (      $TX_ID    , $F_NAME    , $F_TYPE    , $RetVal    , $IDL_Name    , $CmdString    , $DEBUG    );   $DEBUG = 0;   #$DEBUG = 1;   my    (      $r_evt    ) = @_;   $TX_ID  = $$r_evt->get("PTTP_Event.TX_ID");   $F_NAME = $$r_evt->get("PTTP_Event.EventData.F_NAME");   $F_TYPE = $$r_evt->get("PTTP_Event.EventData.F_TYPE");   # Assemble file name for use by Interrogation Data Loader (IDL) ...   $IDL_Name = $F_NAME . ":" . $TX_ID . ":" . $F_TYPE;# Make file available for processing by IDL ...   $CmdString = "cp " . "$PTTP_SERVER_PERSIST_DIR/$TX_ID/$F_NAME"              . " "   . "$PTTP_INT_DATA/$IDL_Name";   system("$CmdString");} PTTP_intIDL::runLoader()

  11. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type What we want: Only do this if external and internal file names match!

  12. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type In other words: Block event generation if names don’t match

  13. Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have a look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

  14. Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have another look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

  15. sub analyze {    my    (      $obj    ) = @_;   $TX_ID  = $obj->get("PTTP_Req.TX_ID");   $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;   ( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));   foreach $FileName ( @DataFiles ) {      # NOTE: Assume that all files of type 'INT' and 'MUL'      #       will be analyzed by TASS and SBCA. Leave it to      #       TASS_Monitor and SBCA_Monitor to decide which      #       files to analyze.      $FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE");      $ReqOper  = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");      if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) &&           ( $ReqOper  =~ /LOAD/i          )    ) {         # Capture $SiteCode for use below ...         $FileName =~ /^\s*(...).*$/;         $SiteCode = $1;         # Unless the file is already present in the final FDVL archive         # location, create a symbolic link there pointing to the file in         # $TX_Dir. FileType not important here; INT and MUL are equivalent.FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');         $HasFilesToAnalyze = 'True';      }   } PTTP_Req::analyze()

  16. Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have another look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

  17. PTTP_Ack DATE | 2009/03/26 10:57:06 FILES  TST00305.A   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.B   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.C   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.D   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.E   F_TYPE | INT   REQUEST    OPER | LOAD SENDER  ACCT | ptoc  ADDR | FrankenFish  NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979 Typical A_AST.txt file ...

  18. PTTP_Ack DATE | 2009/03/26 10:57:06 FILES  TST00305.A   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.B   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.C   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.D   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.E   F_TYPE | INT   REQUEST    OPER | LOAD SENDER  ACCT | ptoc  ADDR | FrankenFish  NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979 Typical A_AST.txt file ... Change LOAD to REFUSED

  19. #------------------------------------------------------------------------------------------#           I N I T I A L I Z A T I O N   A N D   D E C L A R A T I O N S#------------------------------------------------------------------------------------------# Includes ...use PTTP_Message;use PTTP_Event;use HashAssemblage;@ISA = ("PTTP_Message","HashAssemblage");# Initialize operation precedence ...@OpSeq =  (    'OPER'   # File LOAD or CORRECTION  , 'STAT'   # File record-type statistics  , 'HIST'   # History of OPERations on file  , 'TEST'   # Move files to "test archive" - do NOT load  );# Initialize list of status values for each operation that# indicate completion or failure (i.e. final state) ...%FiniStat =  (    'OPER' => 'LOADED|REFUSED|INVALID|NET_DUPS|NET_ANOMALIES|SYS_FAIL|AUTH_FAIL|AGG_FAIL'  , 'STAT' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL'  , 'HIST' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL'  , 'TEST' => 'ARCHIVED|FAILED|SYS_FAIL|AUTH_FAIL'  ); PTTP_Ack.pm

  20. # Generate an instance of PTTP_Event to initiate processing of first# request for each file named in the A_AST "To Do" list ...sub genEvents {   my    (      $FileList    , @Files    , $File    , $FileType    , $F_Name    , $NextOp    , $OpVal    , $TX_ID    , $evt    , $EventData    , $CanonicalDNA    , $DBG_File    );   my    (      $ack    # Reference to PTTP_Ack object instance    ) = @_;   # Obtain list of files named in the "To Do" list ...   $FileList = $ack->opFiles();   @Files = split(/\n/,$FileList);   # Generate an event for each file with un-fulfilled requests ...   # NOTE: At this point, all of the files should qualify.   foreach $File ( @Files ) {$ack->genNextEvent($File);   }} PTTP_Ack::genEvents()

  21. # Generate an instance of PTTP_Event to initiate processing of the# next request for the specified file ...sub genNextEvent {   my    (      $ack    # Reference to PTTP_Ack object instance    , $File    ) = @_;   # Protect embedded '.' ...   $F_Name = $File;   $F_Name =~ s/\./\.\./g;$NextOp   = $ack->nextOp($File);   $TX_ID    = $ack->get("PTTP_Ack.TX_ID");   $OpVal    = $ack->get("PTTP_Ack.FILES.$F_Name.REQUEST.$NextOp");   $FileType = $ack->get("PTTP_Ack.FILES.$F_Name.F_TYPE"); unless ( $NextOp =~ /None/i ) {      $evt = new PTTP_Event();      $evt->set("PTTP_Event.TX_ID='$TX_ID'");      $evt->set("PTTP_Event.Source='PTTP_Client'");      $evt->set("PTTP_Event.Target='PTTP_Server'");      $EventData =        "PTTP_Event\n"      . " EventData\n"      . "  F_NAME | " . $File     . "\n"      . "  F_TYPE | " . $FileType . "\n"      . "  " . $NextOp . " | " . $OpVal . "\n"      ;      $evt->growStructure($EventData);      $evt->persist();   } } PTTP_Ack::genNextEvent()

  22. PTTP_Ack DATE | 2009/03/26 10:57:06 FILES  TST00305.A   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.B   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.C   F_TYPE | INT   REQUEST    OPER | REFUSED  TST00305.D   F_TYPE | INT   REQUEST    OPER | LOAD  TST00305.E   F_TYPE | INT   REQUEST    OPER | LOAD SENDER  ACCT | ptoc  ADDR | FrankenFish  NAME | Testing MSM Client (John) STATUS | RECEIVED TX_ID | TX01DE6A28-98AC-4609-94E2-34983A311979 Typical A_AST.txt file ...

  23. # Returns the 'next operation' in sequence for a given FileName.# If all operations are complete, returns 'None'.sub nextOp  {   my    (      $F_Name    , $OpName    , $ReqString    , @Requests    , $Request    , $ReqStat    , $FinalStat    );   my    (      $obj    , $FileName    ) = @_;   $F_Name = $FileName;   $F_Name =~ s/\./\.\./g;  # Protect embedded '.'   $ReqString = $obj->listKeys("PTTP_Ack.FILES.$F_Name.REQUEST");   ( @Requests ) = split(/\n/,$ReqString);   foreach $OpName ( @OpSeq ) {      foreach $Request ( @Requests ) {         if ( $OpName =~ /$Request/i ) {            $ReqStat   = $obj->get("PTTP_Ack.FILES.$F_Name.REQUEST.$OpName");            $FinalStat = $FiniStat{$OpName};unless ( $ReqStat =~ /$FinalStat/ ) {               # Generate return value ...return $OpName;            }         }      }   }   # Generate return value ...return 'None';} PTTP_Ack::nextOp() REFUSED is a FinalState

  24. Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have another look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

  25. PTTP & IDL – External / Internal File Name Validation Doug Clough SYNERGETICS Engineered Systems That’s all, folks! 7 December 2009 www.ptagis.org

More Related