PTTP & IDL – External / Internal File Name Validation

PTTP & IDL – External / Internal File Name Validation Doug Clough SYNERGETICS Engineered Systems 7 December 2009 www.ptagis.org

IDL – Loader Invocation PTTP Files are pre-processed byidl_validateFile.pl one-by-one as IDL runs

IDL – Loader Invocation PTTP Files are pre-processed byidl_validateFile.pl one-by-one as IDL runs PROBLEM: We must detect external-internal file name mismatch before file is presented to IDL

Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have a look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type Symbolic link in archive directory is created first …

sub analyze { my ( $obj ) = @_; $TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID; ( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES")); foreach $FileName ( @DataFiles ) { # NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze. $FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER"); if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) { # Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1; # Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent.FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT'); $HasFilesToAnalyze = 'True'; } } PTTP_Req::analyze()

sub makeArchiveLink { my ( $PhysicalPath , $ArchivalPath , $FullSpec , $CmdString ); my ( $SourceDir , $FileName , $FileType # INT, MUL, or TAG (INT and MUL give same result) ) = @_; $PhysicalPath = "${SourceDir}/${FileName}"; $ArchivalPath = &sayAbsolutePath($FileName,$FileType); $FullSpec = "${ArchivalPath}/${FileName}";unless ( -f $FullSpec ) { # Create directories if they don't exist ... &checkDirs($FileName,$FileType);$CmdString = "ln -s $PhysicalPath $FullSpec"; system("$CmdString"); } else { print "makeArchiveLink ...\n" if ( $DEBUG ); print " Already exists: $FullSpec\n" if ( $DEBUG ); $DUMMY = <STDIN> if ( $DEBUG ); }} FDVL_ArchiveTree::makeArchiveLink()

PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type How are files presented for loading?

PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event(s) • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event One per file run_loader($evt){<copies $FileName to $INTERROGATION_DATA>}

sub runLoader { my ( $TX_ID , $F_NAME , $F_TYPE , $RetVal , $IDL_Name , $CmdString , $DEBUG ); $DEBUG = 0; #$DEBUG = 1; my ( $r_evt ) = @_; $TX_ID = $$r_evt->get("PTTP_Event.TX_ID"); $F_NAME = $$r_evt->get("PTTP_Event.EventData.F_NAME"); $F_TYPE = $$r_evt->get("PTTP_Event.EventData.F_TYPE"); # Assemble file name for use by Interrogation Data Loader (IDL) ... $IDL_Name = $F_NAME . ":" . $TX_ID . ":" . $F_TYPE;# Make file available for processing by IDL ... $CmdString = "cp " . "$PTTP_SERVER_PERSIST_DIR/$TX_ID/$F_NAME" . " " . "$PTTP_INT_DATA/$IDL_Name"; system("$CmdString");} PTTP_intIDL::runLoader()

PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type What we want: Only do this if external and internal file names match!

PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type In other words: Block event generation if names don’t match

Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have a look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

Not only do we want to suppress file loading We want to prevent generation of sym-link in archival directory We want to return diagnostic message to sender And what about other files in the PTTP transaction? Do we want to reject all of the files if one or more has name problem? Do we want to reject only the specific files that have problems? In either case, we need to take action upstream of PTTP_Ack::genEvents() Have another look at PTTP_Req::analyze() Loops through all files in the TX Calls FDVL_ArchiveTree::makeArchiveLink() Insert code before this call to compare internal and external names If the names match, create the link and continue as presently coded Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED This will suppress generation of an invocation event for the file If external and internal names don’t match …

sub analyze { my ( $obj ) = @_; $TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID; ( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES")); foreach $FileName ( @DataFiles ) { # NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze. $FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER"); if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) { # Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1; # Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent.FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT'); $HasFilesToAnalyze = 'True'; } } PTTP_Req::analyze()

#------------------------------------------------------------------------------------------# I N I T I A L I Z A T I O N A N D D E C L A R A T I O N S#------------------------------------------------------------------------------------------# Includes ...use PTTP_Message;use PTTP_Event;use HashAssemblage;@ISA = ("PTTP_Message","HashAssemblage");# Initialize operation precedence ...@OpSeq = ( 'OPER' # File LOAD or CORRECTION , 'STAT' # File record-type statistics , 'HIST' # History of OPERations on file , 'TEST' # Move files to "test archive" - do NOT load );# Initialize list of status values for each operation that# indicate completion or failure (i.e. final state) ...%FiniStat = ( 'OPER' => 'LOADED|REFUSED|INVALID|NET_DUPS|NET_ANOMALIES|SYS_FAIL|AUTH_FAIL|AGG_FAIL' , 'STAT' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'HIST' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'TEST' => 'ARCHIVED|FAILED|SYS_FAIL|AUTH_FAIL' ); PTTP_Ack.pm

# Generate an instance of PTTP_Event to initiate processing of first# request for each file named in the A_AST "To Do" list ...sub genEvents { my ( $FileList , @Files , $File , $FileType , $F_Name , $NextOp , $OpVal , $TX_ID , $evt , $EventData , $CanonicalDNA , $DBG_File ); my ( $ack # Reference to PTTP_Ack object instance ) = @_; # Obtain list of files named in the "To Do" list ... $FileList = $ack->opFiles(); @Files = split(/\n/,$FileList); # Generate an event for each file with un-fulfilled requests ... # NOTE: At this point, all of the files should qualify. foreach $File ( @Files ) {$ack->genNextEvent($File); }} PTTP_Ack::genEvents()

# Generate an instance of PTTP_Event to initiate processing of the# next request for the specified file ...sub genNextEvent { my ( $ack # Reference to PTTP_Ack object instance , $File ) = @_; # Protect embedded '.' ... $F_Name = $File; $F_Name =~ s/\./\.\./g;$NextOp = $ack->nextOp($File); $TX_ID = $ack->get("PTTP_Ack.TX_ID"); $OpVal = $ack->get("PTTP_Ack.FILES.$F_Name.REQUEST.$NextOp"); $FileType = $ack->get("PTTP_Ack.FILES.$F_Name.F_TYPE"); unless ( $NextOp =~ /None/i ) { $evt = new PTTP_Event(); $evt->set("PTTP_Event.TX_ID='$TX_ID'"); $evt->set("PTTP_Event.Source='PTTP_Client'"); $evt->set("PTTP_Event.Target='PTTP_Server'"); $EventData = "PTTP_Event\n" . " EventData\n" . " F_NAME | " . $File . "\n" . " F_TYPE | " . $FileType . "\n" . " " . $NextOp . " | " . $OpVal . "\n" ; $evt->growStructure($EventData); $evt->persist(); } } PTTP_Ack::genNextEvent()

# Returns the 'next operation' in sequence for a given FileName.# If all operations are complete, returns 'None'.sub nextOp { my ( $F_Name , $OpName , $ReqString , @Requests , $Request , $ReqStat , $FinalStat ); my ( $obj , $FileName ) = @_; $F_Name = $FileName; $F_Name =~ s/\./\.\./g; # Protect embedded '.' $ReqString = $obj->listKeys("PTTP_Ack.FILES.$F_Name.REQUEST"); ( @Requests ) = split(/\n/,$ReqString); foreach $OpName ( @OpSeq ) { foreach $Request ( @Requests ) { if ( $OpName =~ /$Request/i ) { $ReqStat = $obj->get("PTTP_Ack.FILES.$F_Name.REQUEST.$OpName"); $FinalStat = $FiniStat{$OpName};unless ( $ReqStat =~ /$FinalStat/ ) { # Generate return value ...return $OpName; } } } } # Generate return value ...return 'None';} PTTP_Ack::nextOp() REFUSED is a FinalState

PTTP & IDL – External / Internal File Name Validation Doug Clough SYNERGETICS Engineered Systems That’s all, folks! 7 December 2009 www.ptagis.org

PTTP & IDL – External / Internal File Name Validation